Reinforcement learning method, action generation system, computer device, and storage medium

Abstract

The application relates to a reinforcement learning method, an action generation system, a computer device and a storage medium. The method comprises the following steps: encoding historical state and action information in a preset time range to obtain encoded trajectory information; generating a target action based on the encoded trajectory information and a preset deep learning network; determining a predicted return of the target action based on the target action and a current state corresponding to the target action; determining an exploration ability adjustment value based on an action probability distribution in the current state; and adjusting parameters of the deep learning network based on the predicted return and the exploration ability adjustment value. The method can improve the robustness of policy generation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of reinforcement learning, and particularly relates to a reinforcement learning method, an action generation system, a computer device and a storage medium. BACKGROUND

[0002] Reinforcement learning is a machine learning method that learns how to perform a task by interacting with an environment, selects an action according to a current state, and accumulates rewards to learn an execution strategy.

[0003] However, the traditional value distribution reinforcement learning is often inputted with a combination of a current state and an action, and is dependent on the current state to determine a strategy, so some important historical information affecting decision-making is often missed, thereby causing the problem of low robustness of the strategy generation.

[0004] Therefore, the current reinforcement learning method still has the problem of low robustness of the strategy generation. SUMMARY

[0005] Therefore, it is necessary to provide a reinforcement learning method, an action generation system, a computer device and a storage medium capable of improving the robustness of the strategy generation.

[0006] In a first aspect, the present application provides a reinforcement learning method applied to an action generation system, and the reinforcement learning method comprises the following steps:

[0007] encoding historical state and action information in a preset time range to obtain encoded trajectory information;

[0008] generating a target action based on the encoded trajectory information and a preset deep learning network;

[0009] determining a predicted return of the target action based on the target action and a current state corresponding to the target action;

[0010] determining an exploration ability adjustment value based on an action probability distribution under the current state;

[0011] adjusting parameters of the deep learning network based on the predicted return and the exploration ability adjustment value.

[0012] In one embodiment, the step of encoding the historical state and action information in the preset time range to obtain the encoded trajectory information comprises the following steps:

[0013] obtaining historical state and action information in a preset time range based on a time sliding window;

[0014] The encoder encodes the historical state and action information to obtain encoded trajectory information.

[0015] In one of the embodiments, the obtaining of the target action based on the encoded trajectory information and a preset deep learning network comprises:

[0016] generating a probability distribution of a plurality of candidate actions based on the encoded trajectory information and the preset deep learning network;

[0017] determining the target action based on the probability distribution of the plurality of candidate actions.

[0018] In one of the embodiments, the determining of the predicted return of the target action based on the target action and a current state corresponding to the target action comprises:

[0019] obtaining the current state corresponding to the target action;

[0020] calculating a future return distribution of the target action based on the target action and the current state;

[0021] mapping at least one sample randomly sampled from a uniform distribution to the future return distribution to obtain at least one quantile value;

[0022] determining the predicted return of the target action based on the quantile value.

[0023] In one of the embodiments, the determining of the exploration ability adjustment value based on the action probability distribution under the current state comprises:

[0024] calculating a spatial distance between the action probability distribution under the current state and a uniform distribution;

[0025] taking the spatial distance as the exploration ability adjustment value.

[0026] In a second aspect, the present application provides an action generation system, comprising an encoder, an actor network, a critic network, an explorer and a policy optimization module, wherein:

[0027] the encoder is configured to encode historical state and action information in a preset time range to obtain encoded trajectory information;

[0028] the actor network is configured to generate a target action based on the encoded trajectory information;

[0029] the critic network is configured to determine a predicted return of the target action based on the target action and a current state corresponding to the target action;

[0030] The explorer is used to determine an exploration capability adjustment value based on the action probability distribution in the current state;

[0031] The strategy optimization module is used to adjust the parameters of the actor network based on the predicted return and exploration ability adjustment values.

[0032] In one embodiment, the encoder includes a buffer and an encoding component, wherein:

[0033] The buffer is used to acquire historical state and action information within a preset time range;

[0034] The encoding component is used to encode historical state and action information within a preset time range to obtain encoded trajectory information.

[0035] In one embodiment, the explorer is also used for:

[0036] Calculate the action probability distribution in the current state and the spatial distance between the uniform distributions;

[0037] The spatial distance is used as the adjustment value for the exploration capability.

[0038] Thirdly, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0039] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0040] The aforementioned reinforcement learning method, action generation system, computer device, and storage medium encode historical state and action information within a preset time range to obtain encoded trajectory information; based on the encoded trajectory information and a preset deep learning network, a target action is generated; based on the target action and the current state corresponding to the target action, the predicted reward of the target action is determined; based on the action probability distribution in the current state, an exploration capability adjustment value is determined; based on the predicted reward and the exploration capability adjustment value, the parameters of the deep learning network are adjusted. By encoding historical state and action information, context-based decision-making can be achieved, and by determining the exploration capability adjustment value based on the action probability distribution in the current state, the agent can be encouraged to fully explore the state space, avoiding deterministic policies that lead to valuable actions, thereby improving the robustness of policy generation. Attached Figure Description

[0041] Figure 1An application environment diagram of the reinforcement learning method in an embodiment;

[0042] Figure 2 A flowchart of the reinforcement learning method in an embodiment;

[0043] Figure 3 A structural block diagram of the action generation system in an embodiment;

[0044] Figure 4 A structural block diagram of the action generation system in another embodiment;

[0045] Figure 5 An internal structure diagram of the computer device in an embodiment. DETAILED DESCRIPTION

[0046] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0047] The reinforcement learning method provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 . The terminal 102 communicates with the server 104 through a network. The data storage system can store data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on a cloud or other network server. After the terminal 102 generates an action, a feedback can be obtained in the environment, which can be based on corresponding feedback information by the server 104 for the generated action, or can be the state collection of the surrounding environment by the terminal 102 through a sensor, etc. The terminal 102 encodes the historical state and action information in a preset time range to obtain encoded trajectory information; generates a target action based on the encoded trajectory information and a preset deep learning network; determines a predicted return of the target action based on the target action and a current state corresponding to the target action; determines an exploration ability adjustment value based on an action probability distribution under the current state; and adjusts parameters of the deep learning network based on the predicted return and the exploration ability adjustment value. The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0048] In an embodiment, as shown in Figure 2 , a reinforcement learning method is provided, and the method is applied inFigure 1 The terminal 102 in the system 100 is taken as an example for illustration, including the following steps:

[0049] In step S100, historical state and action information in a preset time range is encoded to obtain encoded trajectory information.

[0050] The historical state and action information can be historical state and action information collected in the preset time range.

[0051] The state can be a state of the terminal 102 itself, or a feature and information of an environment in which the terminal 102 is located. For example, the state can include a current time, a location, etc. When the reinforcement learning method is applied to a game environment, the state can include a position coordinate, a speed and an acceleration of a character itself, and weather conditions, light intensity, etc. in the game environment. When the reinforcement learning method is applied to a robot or automatic driving, the state can include data of a sensor, such as laser radar ranging, infrared sensor reading, etc. When the reinforcement learning method is applied to resource scheduling and management, the state can also include a state of a current available resource, such as a battery capacity, a battery health state, a storage capacity, etc. The present embodiment is not limited in this regard.

[0052] The action information can include a target action generated by the terminal 102. For example, when the reinforcement learning method is applied to a game environment, a robot or automatic driving, the target action can be adjusting to a specified direction, moving a specified number of steps in a specified direction, etc. When the reinforcement learning method is applied to resource scheduling and management, the target action can be an operation or a control instruction sent to a specific object. The present embodiment is not limited in this regard.

[0053] Further, the action information in the preset time range can include a plurality of target actions generated by the terminal 102 in the preset time range.

[0054] Encoding the historical state and action information to obtain the encoded trajectory information can be converting the historical state and action information into a compact representation, i.e. the encoded trajectory information, through an encoder or other encoding network. By encoding the historical state and action information to obtain the encoded trajectory information, long-term dependencies between the state and the action can be captured, and more abundant context information can be provided for subsequent action generation.

[0055] In step S200, a target action is generated based on the encoded trajectory information and a preset deep learning network.

[0056] The target action can be generated by calling the preset deep learning network to process the encoded trajectory information, so as to generate an action with the highest value corresponding to the encoded trajectory information as the target action.

[0057] Step S300, determining a predicted return of the target action based on the target action and the current state corresponding to the target action.

[0058] Wherein, after the target action is performed, the feedback of the current state obtained under the target action can be obtained, and according to the target action and the current state corresponding to the target action, the immediate reward that can be obtained after the target action is collected and the future expected cumulative reward can be judged.

[0059] Step S400, determining an exploration ability adjustment value based on the action probability distribution under the current state.

[0060] Wherein, the exploration ability is the ability to explore unknown areas, and the exploration ability adjustment value can be the difference value between the action probability distribution and the uniform distribution. It can be understood that the deep learning network can be according to the value of the selected action among the multiple candidate actions under the current state, that is, the high and low of the probability of selection, to filter the candidate action with the highest value as the target action. When the uniformity of the probability distribution of multiple candidate actions under the same state is large, it means that the deep learning network will follow a more certain strategy to give the target action, and the ability to explore unknown areas is low. By calculating the exploration ability adjustment value, the strategy of the deep learning network can be avoided to tend to be a deterministic strategy, so as to avoid missing valuable target actions.

[0061] Step S500, adjusting the parameters of the deep learning network based on the predicted return and the exploration ability adjustment value.

[0062] Wherein, based on the predicted return and the exploration ability adjustment value, the parameters of the deep learning network can be adjusted, which can be through gradient descent and other optimization algorithms to find the minimum difference between the predicted return and the actual return, and to balance the improvement of the exploration ability, so as to ensure the optimization strategy while improving the performance on a specific task, and to achieve the effect of improving the robustness.

[0063] The reinforcement learning method provided in the embodiment encodes historical state and action information in a preset time range to obtain encoded trajectory information; generates a target action based on the encoded trajectory information and a preset deep learning network; determines a predicted return of the target action based on the target action and a current state corresponding to the target action; determines an exploration ability adjustment value based on an action probability distribution in the current state; and adjusts parameters of the deep learning network based on the predicted return and the exploration ability adjustment value. Through encoding of historical state and action information, context-based decision making can be achieved, and through determining the exploration ability adjustment value based on the action probability distribution in the current state, the agent can be encouraged to fully explore the state space, and deterministic strategies can be avoided to determine valuable actions, thereby achieving the effect of improving the robustness of strategy generation.

[0064] In one of the embodiments, the encoding of the historical state and action information in the preset time range to obtain the encoded trajectory information includes:

[0065] The historical state and action information in the preset time range are obtained based on a time sliding window.

[0066] The historical state and action information are encoded by an encoder to obtain the encoded trajectory information.

[0067] The time sliding window can be a fixed-length time window, which continuously slides forward to maintain the latest historical state and action information as the agent interacts with the environment.

[0068] The encoding of the historical state and action information by the encoder can be a recurrent neural network such as RNN, LSTM, GRU, etc., which converts the historical state and action information into a vector or a matrix, i.e., the encoded trajectory information, thereby capturing patterns and trends in time series data and providing effective information for subsequent target actions.

[0069] In a specific embodiment, the encoder can be implemented using a Transformer encoder.

[0070] The reinforcement learning method provided in the embodiment can efficiently manage and utilize historical information by obtaining historical state and action information based on a time sliding window and obtaining encoded trajectory information based on an encoder, so that the agent has stronger adaptability and robustness when facing complex and variable environments, thereby achieving the effect of improving the robustness of strategy generation.

[0071] In one of the embodiments, the obtaining of the target action based on the encoded trajectory information and the preset deep learning network includes:

[0072] generate a probability distribution of a plurality of candidate actions based on the encoded trajectory information and a preset deep learning network;

[0073] determine a target action based on the probability distribution of the plurality of candidate actions.

[0074] In some embodiments, the preset deep learning network can be a neural network such as a policy network, which can output a probability distribution including a plurality of candidate actions. The probability distribution of the plurality of candidate actions includes the plurality of candidate actions and the likelihood of each candidate action being selected.

[0075] It can be understood that by generating a probability distribution, the relative advantages and disadvantages of different actions can be explicitly expressed, and it is also convenient for subsequent exploration ability adjustment and calculation of exploration ability adjustment values.

[0076] Based on the probability distribution of the plurality of candidate actions, the target action can be determined by selecting the action with the highest selection probability as the target action, thereby preserving the exploration ability while avoiding premature convergence to a suboptimal solution.

[0077] The reinforcement learning method provided in the embodiments can determine a target action by generating a probability distribution of a plurality of candidate actions, thereby improving the quality of decision-making while enhancing the exploration ability of the algorithm, and achieving the effect of improving the flexibility and robustness of the policy.

[0078] In one of the embodiments, the determining the predicted return of the target action based on the target action and a current state corresponding to the target action comprises:

[0079] obtaining a current state corresponding to the target action;

[0080] calculating a future return distribution of the target action based on the target action and the current state;

[0081] mapping at least one sample randomly sampled from a uniform distribution to the future return distribution to obtain at least one quantile value;

[0082] determining the predicted return of the target action based on the quantile value.

[0083] In some embodiments, the current state corresponding to the target action can be state information used in the generation stage of the target action. The current state is used to represent the environment state and the self-state of the agent when the target action is generated.

[0084] Based on the target action and the current state, a future return distribution of the target action is calculated. The future return distribution can be a distribution of future returns determined based on the target action and the current state. It can be understood that, by calculating the future return distribution, a comprehensive perspective of various returns that can be obtained after taking the target action can be obtained, rather than a single expected value, so that the uncertainty and risk can be better reflected, and a more robust decision can be made.

[0085] Further, in the embodiment, the future return distribution can be modeled based on an implicit quantile network. For example, a random return can be modeled by the implicit quantile network.

[0086] At least one sample is randomly sampled from a uniform distribution and mapped to the future return distribution to obtain at least one quantile value. The at least one quantile value can be obtained by sampling one or more samples from the uniform distribution, and pushing the samples forward to the corresponding quantile values sampled from the future return distribution.

[0087] The predicted return of the target action can be determined according to the quantile value. The predicted return can be determined by taking an average value, a median value, or other one or more quantile values selected based on the requirements of an application scenario.

[0088] The reinforcement learning method provided in the embodiment can accurately evaluate the return distribution that can be brought by the target action in the future by calculating the future return distribution of the target action and mapping the quantile value from the uniform distribution to determine the predicted return, so that the robustness of the policy generation is improved.

[0089] In one of the embodiments, determining the exploration ability adjustment value based on the action probability distribution in the current state comprises:

[0090] Calculating a spatial distance between the action probability distribution in the current state and the uniform distribution.

[0091] Taking the spatial distance as the exploration ability adjustment value.

[0092] The action probability distribution in the current state can be a probability distribution of a plurality of candidate actions obtained by a deep learning network based on an encoded trajectory distribution. The uniform distribution can be a uniform distribution in which the probabilities of the plurality of candidate actions are the same.

[0093] The spatial distance between the action probability distribution in the current state and the uniform distribution can be calculated by a KL divergence, a cross-entropy, a divergence regularization, a Wasserstein distance, or the like.

[0094] After the spatial distance between the action probability distribution and the uniform distribution is obtained, the spatial distance can be used as an exploration ability adjustment value to adjust the parameters of the deep learning network.

[0095] The reinforcement learning method provided in the embodiment calculates the spatial distance between the action probability distribution and the uniform distribution in the current state as an exploration ability adjustment value, so that the agent can automatically adjust the exploration behavior in different situations, which helps to more effectively find the optimal strategy and can improve the robustness and comprehensiveness of the strategy generation.

[0096] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0097] Based on the same inventive concept, the embodiments of the present application also provide an action generation system for implementing the above-mentioned reinforcement learning method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more action generation system embodiments provided below can refer to the limitations of the reinforcement learning method described above, which will not be repeated here.

[0098] In one embodiment, as shown in Figure 3 An action generation system is provided, including an encoder 100, an actor network 200, a critic network 300, an explorer 400, and a policy optimization module 500, wherein:

[0099] The encoder 100 is configured to encode historical state and action information in a preset time range to obtain encoded trajectory information;

[0100] The actor network 200 is configured to generate a target action based on the encoded trajectory information;

[0101] The critic network 300 is configured to determine a predicted return of the target action based on the target action and a current state corresponding to the target action;

[0102] The explorer 400 is configured to determine an exploration ability adjustment value based on the action probability distribution in the current state.

[0103] The policy optimization module 500 is configured to adjust the parameters of the actor network 200 based on the predicted return and the exploration ability adjustment value.

[0104] In one embodiment, the encoder 100 includes a buffer and an encoding component, wherein:

[0105] The buffer is configured to obtain historical state and action information within a preset time range.

[0106] The encoding component is configured to encode the historical state and action information within the preset time range to obtain encoded trajectory information.

[0107] Further, the encoding component can be a Transformer encoder.

[0108] In one embodiment, the explorer 400 is further configured to:

[0109] Calculate the spatial distance between the action probability distribution in the current state and the uniform distribution.

[0110] Take the spatial distance as the exploration ability adjustment value.

[0111] In order to more clearly set forth the technical solutions of the present application, a detailed embodiment is further provided.

[0112] In one embodiment, as shown in Figure 4 , an action generation system is provided, which is applied to an agent, and the action generation system includes an encoder, an actor network, a critic network, and an explorer, wherein:

[0113] The encoder is a Transformer-based historical information encoder (Encoder), and the design of the Transformer encoder aims to make full use of historical state and action information to improve the decision-making effect in reinforcement learning.

[0114] Specifically, the encoder includes a buffer and a Transformer encoder, wherein:

[0115] The buffer is a sliding window buffer, and the historical state and action information are stored in a sliding window buffer, which is constantly updated to ensure that the encoder can use the latest historical trajectory information. In this way, the model will add new historical state and action information to the buffer every time it interacts with the environment to keep track of the historical state and action information in real time.

[0116] The Transformer encoder is a history trajectory information-based encoder that processes the history trajectory information in the buffer and combines it with the current state state s obtained from the environment to generate the final state representation x. This combination ensures that the current decision takes into account past environmental dynamics and action selection.

[0117] To improve the stability of training and the convergence of the model, the Transformer encoder adopts a "pre-training + joint training fine-tuning" strategy. In the pre-training phase, the Transformer is trained to predict the current state state s by encoding trajectory information and compare the prediction with the real state, so as to gradually learn how to effectively convert historical information into an accurate representation of the current state. After pre-training, the encoder enters the joint training phase, where the Transformer encoder is fine-tuned together with other components of reinforcement learning, such as the actor and critic, to better adapt to the optimization goal of the overall system.

[0118] Through the pre-training phase, the Transformer has been able to generate a relatively accurate state representation. On this basis, joint training fine-tuning further optimizes the performance of the encoder, ensuring its stability in the reinforcement learning training process and accelerating the convergence of the overall system.

[0119] Through the above method, the Transformer encoder not only effectively utilizes historical information, but also ensures the stability of the model in complex environments and rapid convergence through pre-training and fine-tuning mechanisms.

[0120] The actor network is an actor based on noise sampling, which can include a deep neural network. The actor network samples noise from the base distribution, specifically, it converts the base distribution P(X) ∈ M(R d ) to the action space and generates actions in a sample-to-sample manner, i.e., the target action a = π(s, ξ; θ π ): S × X → A, where s is the current state, ξ is the noise sampled from the base distribution P(X), i.e., ξ ~ P(X), S is the state space, X is the noise space, and A is the action space. θ π is the parameter of the actor network, which can also be written as a ~ π(·|s; θ π ), where the strategy is a random strategy, and the action under the state s can be generated by a lightweight method, i.e., first sample ξ from P(X), and then use a = π(s, ξ; θ π) to obtain the target action action a under state s. Through the above pre-push distribution, higher expression ability and modeling ability can be exhibited.

[0121] By sampling noise in the base distribution, the model can introduce appropriate randomness in the action generation process, which not only enhances the exploration ability of the policy, but also better captures the diversity and non-linear relationship in the complex environment. This method can effectively deal with the limitations of the expression ability of the Gaussian distribution in traditional random policy, so that the policy performs better in multi-peak distribution and highly nonlinear environment.

[0122] The critic network is a critic with quantile representation. The critic uses an implicit quantile network IQN to push the sample from a uniform distribution U(0, 1) to the corresponding quantile value sampled from the return distribution. The approximation of the return distribution is maintained by a weighted mixture of N Dirac functions. Specifically, the implicit quantile distribution can be used to model the random return and update it. Where s and a represent the state and action respectively, represents the quantile, and θ z represents the parameters of the implicit quantile distribution network. The quantile samples are extracted independently and identically distributed and sorted by τ i <τ i+1 For the random return Z π (s, a), the implicit quantile distribution induced by the IQN with parameters θ z is defined as a weighted mixture of N Dirac distributions:

[0123]

[0124] Where, and Further,

[0125]

[0126] represents the inverse cumulative distribution function of the random return Z π (s, a), and its cumulative distribution function CDF is represented as:

[0127]

[0128] For the purpose of risk sensitivity or other specific tasks, a reward reshaping type of utility function ψ(·) can be used to reshape the random reward R(s, a) of the policy.

[0129] ​​​wherein the critic network utilizes a state-action utility function to evaluate the expected utility after taking action a in state s, which is defined as:

[0130]

[0131] wherein s is the current state, a is the target action, s t+1 is the next state, R is the immediate reward, γ is the discount factor, and P is the transition probability distribution of the environment, wherein s0=s, a0=a, and the state utility function is:

[0132]

[0133] The above formula satisfies the following utility Bellman equation:

[0134]

[0135] For a given policy π, the critic network can evaluate its performance by obtaining the predicted return wherein μ0 is the initial state distribution.

[0136] In some other embodiments, the critic network can also represent the quantile return by a quantile regression network QRN, a fully parameterized quantile function FQF.

[0137] The explorer is a distribution difference-based explorer Explorer. It can be understood that the exploration ability of the model can be enhanced by introducing diverse behaviors in the policy. The explorer compares the difference between the policy distribution and the uniform distribution to guide the exploration of the target action, so as to ensure that the policy does not concentrate too much in a certain area, thereby improving the exploration ability of the policy and increasing the opportunity to discover a better policy. When the difference between the policy distribution and the uniform distribution is small, it means that the policy has good exploration ability; otherwise, it may mean that the policy is too concentrated on certain actions and has insufficient exploration ability.

[0138] The difference between the two distributions can be realized by maximum mean difference MMD, KL divergence, Wasserstein distance, entropy regularization, divergence regularization, etc.

[0139] Exemplarily, in the case of using KL divergence, the explorer can use the KL divergence to quantify the degree of deviation of the policy distribution from the uniform distribution, and optimize the policy by minimizing the deviation. In the case of using Wasserstein distance, the explorer can use the Wasserstein distance to guide the policy to approach the uniform distribution, measure the "geographical" distance between two probability distributions, and enhance the exploration ability of the policy. In the case of using entropy regularization, by increasing the entropy of the policy, the explorer can encourage the policy distribution to be more uniform, thereby improving the explorability. In the case of using divergence regularization, by imposing certain divergence regularization constraints on the policy distribution, such as Jensen-Shannon divergence, the explorer can ensure that the policy is not too concentrated, and maintain good explorability.

[0140] This embodiment takes the maximum mean discrepancy MMD as an example. Let F be a unit ball in a reproducing kernel Hilbert space (RKHS) H defined on a compact metric space X. Then, the MMD between two distributions p and q can be represented as:

[0141] MMD(p, q) := sup f∈F (E x～p [f(x)]-E y～p [f(y)])

[0142] For the maximum mean discrepancy MMD, it is calculated in an approximate manner, only needing to sample from the distribution, without needing to explicitly know the density function. Specifically, given m samples (x1, … x m ) sampled from p and n samples (y1, … y n ) sampled from q, then the MMD between p and q can be approximately represented by the following formula:

[0143]

[0144] where k is a kernel function for measuring the similarity between two data points, and the reference distribution can be a uniform distribution u(·|s) selected in the action space A. In this embodiment, dm(θ π ) represents the regular term of the explorer based on the sample, that is, the expected maximum mean discrepancy MMD between the policy distribution π(·|s; θ π ) and the uniform distribution u(·|s), and the formula is as follows:

[0145]

[0146] where θ π is the parameter of the actor network, and d π u0denotes the state occupancy measure with respect to policy π, defined as

[0147] It can be understood that the exploration ability of the policy distribution π(·|s; θ π ) is inversely proportional to dm(θ π ). By reducing the difference, the explorer can give the policy a diverse action, thereby effectively encouraging exploration.

[0148] According to the exploration ability adjustment value of the explorer and the predicted return of the critic network, the objective function of the actor network is adjusted through the BP back propagation algorithm.

[0149] In the process of policy optimization, the objective function of the actor network is used to maximize the expected utility while maintaining good exploration ability, and the specific formula is as follows:

[0150]

[0151] Wherein, α represents the weight of the regularization term, and ψ is the utility function. By maximizing J ψ (θ π ), the policy not only pursues higher expected utility, but also maintains the ability to explore due to the presence of the maximum mean difference MMD regularization term. The optimization process can be divided into two steps, in which:

[0152] First, the parameters of the critic network are updated using distributed TD learning, and the loss function is defined as follows:

[0153]

[0154] Wherein, can be efficiently optimized by the stochastic gradient descent (SGD) method. The distributed TD learning process can be carried out by minimizing the following Huber quantile regression loss:

[0155]

[0156] Wherein, κ is a constant threshold, δ ij (s,a) is the paired TD error between the two consecutive step implicit quantile approximations, and the calculation formula is as follows:

[0157]

[0158] Wherein, a' ~ π(·|s′), and are calculated based on two randomly sampled quantiles and .

[0159] Second, according to Jψ (θ π The parameters of the strategy are iteratively optimized using the gradient ascent method.

[0160] Understandably, strategy θ π Randomness affects action selection, including the generation of the target action a ~ π(·|s;θ). π ), and functions Accessing the density function of a forward strategy with a complex deep neural network (DNN) is not feasible, therefore J ψ (θ π The first part of the optimization, namely The computation of this would be extremely difficult. Therefore, in this embodiment, a Stochastic Utility Value Policy Gradient (SUVPG) theorem is employed, and approximate computation is performed only based on samples of the policy, thereby enabling policy optimization.

[0161] This embodiment provides an action generation system that uses Transformer to encode historical state and action information, enabling the agent to utilize richer contextual information. This combination of historical information and current state allows the agent to make more accurate and forward-looking decisions in complex environments, thereby improving task completion. By sampling noise from the base distribution and mapping it to the action space, it overcomes the limitations of the traditional Gaussian distribution. The use of this forward distribution enhances the policy's expressive power in complex, multimodal environments, better capturing the diversity and nonlinear relationships in the environment, thus improving the policy's adaptability and performance. The introduction of a distribution-based explorer effectively solves the problem of insufficient exploration ability in traditional reinforcement learning methods. By comparing the policy distribution with a uniform distribution, the explorer can guide the agent to conduct a wider range of explorations, avoiding getting trapped in local optima, thereby improving the policy's ability to discover global optima. By pre-training the Transformer and fine-tuning it during joint training, the stability and convergence of online learning are effectively improved. This technique not only accelerates the model training process but also reduces potential instability, enabling the model to adapt quickly to new data and maintain efficient learning. Furthermore, the enhanced randomness and diversity in policy generation make the policies more robust to noise and environmental changes. These improvements allow agents to maintain efficient behavioral decisions under a wider range of environmental conditions.

[0162] Each module in the aforementioned action generation system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0163] In one embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram thereof can be as shown in the figure. Figure 5 The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved through WIFI, mobile cellular network, NFC (Near Field Communication) or other technologies. The computer program is executed by the processor to implement a reinforcement learning method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0164] Those skilled in the art can understand that Figure 5 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0165] In one embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the reinforcement learning method of any of the above embodiments.

[0166] Encode the historical state and action information in a preset time range to obtain encoded trajectory information;

[0167] Generate a target action based on the encoded trajectory information and a preset deep learning network;

[0168] Determine a predicted return of the target action based on the target action and a current state corresponding to the target action;

[0169] Determine an exploration ability adjustment value based on the action probability distribution under the current state;

[0170] Adjust the parameters of the deep learning network based on the predicted return and the exploration ability adjustment value.

[0171] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the reinforcement learning method of any one of the above embodiments:

[0172] The historical state and action information in a preset time range are encoded to obtain encoded trajectory information;

[0173] Based on the encoded trajectory information and a preset deep learning network, a target action is generated;

[0174] Based on the target action and a current state corresponding to the target action, a predicted return of the target action is determined;

[0175] Based on the action probability distribution under the current state, an exploration ability adjustment value is determined;

[0176] Based on the predicted return and the exploration ability adjustment value, the parameters of the deep learning network are adjusted.

[0177] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0178] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0179] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0180] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A reinforcement learning method, characterized by, The application relates to a motion generation system applied to a robot, and the reinforcement learning method comprises the following steps: history state and action information in a preset time range are encoded to obtain encoded trajectory information; the action information comprises adjusting to a specified direction and moving a specified number of steps in the specified direction; a target action is generated based on the encoded trajectory information and a preset deep learning network; a predicted return of the target action is determined based on the target action and a current state corresponding to the target action; the determination of the predicted return of the target action based on the target action and the current state corresponding to the target action comprises the following steps: the current state corresponding to the target action is acquired; a future return distribution of the target action is calculated based on the target action and the current state; at least one quantile value is obtained by mapping at least one sample randomly sampled from a uniform distribution to the future return distribution; the future return distribution is obtained based on an implicit quantile network; the uniform distribution is a uniform distribution in an action space; the action space is obtained by converting a base distribution by sampling noise from the base distribution; the predicted return of the target action is determined based on the quantile value; an exploration ability adjustment value is determined based on an action probability distribution in the current state; the action probability distribution in the current state is a probability distribution of a plurality of candidate actions obtained by a deep learning network based on encoded trajectory information; the determination of the exploration ability adjustment value based on the action probability distribution in the current state comprises the following steps: a spatial distance between the action probability distribution in the current state and a uniform distribution is calculated; and the spatial distance is taken as the exploration ability adjustment value; parameters of the deep learning network are adjusted based on the predicted return and the exploration ability adjustment value.

2. The reinforcement learning method of claim 1, wherein, The encoding of the history state and the action information in the preset time range to obtain the encoded trajectory information comprises the following steps: history state and action information in a preset time range are acquired based on a time sliding window; the history state and the action information are encoded by an encoder to obtain encoded trajectory information.

3. The reinforcement learning method of claim 1, wherein, The generation of the target action based on the encoded trajectory information and the preset deep learning network comprises the following steps: probability distributions of a plurality of candidate actions are generated based on the encoded trajectory information and the preset deep learning network; a target action is determined based on the probability distributions of the plurality of candidate actions.

4. An action generation system, characterized by, The motion generation system is applied to a robot, and comprises an encoder, an actor network, a critic network, an explorer and a policy optimization module, wherein: the encoder is used for encoding history state and action information in a preset time range to obtain encoded trajectory information; the action information comprises adjusting to a specified direction and moving a specified number of steps in the specified direction; the actor network is used for generating a target action based on the encoded trajectory information; The critic network is configured to determine a predicted return of the target action based on the target action and a current state corresponding to the target action; the determination of the predicted return of the target action based on the target action and the current state corresponding to the target action comprises: obtaining the current state corresponding to the target action; calculating a future return distribution of the target action based on the target action and the current state; mapping at least one sample randomly sampled from a uniform distribution to the future return distribution to obtain at least one quantile value; the future return distribution is obtained based on an implicit quantile network; the uniform distribution is a uniform distribution in an action space; the action space is obtained by converting a base distribution by sampling noise from the base distribution; and determining the predicted return of the target action based on the quantile value. The explorer is configured to determine an exploration ability adjustment value based on an action probability distribution in the current state; the action probability distribution in the current state is a probability distribution of a plurality of candidate actions obtained by a deep learning network based on encoded trajectory information; and the determination of the exploration ability adjustment value based on the action probability distribution in the current state comprises: calculating a spatial distance between the action probability distribution in the current state and a uniform distribution; and taking the spatial distance as the exploration ability adjustment value. The policy optimization module is configured to adjust parameters of the actor network based on the predicted return and the exploration ability adjustment value.

5. The action generation system of claim 4, wherein, The encoder comprises a buffer and an encoding component, wherein: The buffer is configured to obtain historical state and action information in a preset time range; The encoding component is configured to encode the historical state and action information in the preset time range to obtain encoded trajectory information. 6.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-5 when the computer program is executed by the processor. The processor executes the computer program to implement the method in any one of claims 1 to 3.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1 to 3.

Images