Reinforcement learning method, action generation system, computer equipment and storage medium

Abstract

The invention relates to a reinforcement learning method, an action generation system, computer equipment and a storage medium. The method comprises the following steps: encoding historical state and action information within a preset time range to obtain encoding track information; generating a target action based on the coding track information and a preset deep learning network; determining a prediction return of the target action based on the target action and a current state corresponding to the target action; determining an exploration capability adjustment value based on the action probability distribution in the current state; and adjusting parameters of the deep learning network based on the prediction return and the exploration capability adjustment value. By adopting the method, the effect of improving the robustness of strategy generation can be achieved.

Description

Technical Field

[0001] The present application relates to the technical field of reinforcement learning, and in particular to a reinforcement learning method, an action generation system, a computer device and a storage medium. Background Art

[0002] Reinforcement learning is a machine learning method that allows an agent to learn how to perform tasks by interacting with the environment, and learns execution strategies by selecting actions based on the current state and accumulating rewards.

[0003] However, traditional value distribution reinforcement learning often inputs a combination of current state and action, and relies on the current state to determine the strategy. Therefore, it often misses some important historical information that affects the decision, which leads to the problem of low robustness of strategy generation.

[0004] This shows that the current reinforcement learning methods still have the problem of low robustness of strategy generation. Summary of the invention

[0005] Based on this, it is necessary to provide a reinforcement learning method, action generation system, computer device and storage medium that can improve the robustness of strategy generation in response to the above technical problems.

[0006] In a first aspect, the present application provides a reinforcement learning method, which is applied to an action generation system, and the reinforcement learning method includes:

[0007] Encode the historical state and action information within a preset time range to obtain encoded trajectory information;

[0008] Generate a target action based on the encoded trajectory information and a preset deep learning network;

[0009] Determining a predicted reward for the target action based on the target action and a current state corresponding to the target action;

[0010] Determining an exploration capability adjustment value based on the action probability distribution in the current state;

[0011] Based on the predicted reward and the exploration capability adjustment value, the parameters of the deep learning network are adjusted.

[0012] In one embodiment, encoding the historical state and action information within a preset time range to obtain the encoded trajectory information includes:

[0013] Based on the time sliding window, obtain the historical status and action information within the preset time range;

[0014] The historical state and action information are encoded based on an encoder to obtain encoded trajectory information.

[0015] In one embodiment, obtaining the target action based on the encoded trajectory information and a preset deep learning network includes:

[0016] Based on the encoded trajectory information and a preset deep learning network, generating a probability distribution of multiple candidate actions;

[0017] Based on the probability distribution of the multiple actions to be selected, a target action is determined.

[0018] In one embodiment, determining the predicted reward of the target action based on the target action and the current state corresponding to the target action includes:

[0019] Obtaining the current state corresponding to the target action;

[0020] Based on the target action and the current state, calculate the future reward distribution of the target action;

[0021] Mapping at least one sample randomly sampled from a uniform distribution to the future return distribution to obtain at least one quantile value;

[0022] Based on the percentile value, a predicted reward for the target action is determined.

[0023] In one embodiment, determining the exploration capability adjustment value based on the action probability distribution in the current state includes:

[0024] Calculate the spatial distance between the action probability distribution and the uniform distribution in the current state;

[0025] The spatial distance is used as the exploration capability adjustment value.

[0026] In a second aspect, the present application provides an action generation system, the action generation system comprising an encoder, an actor network, a critic network, an explorer, and a strategy optimization module, wherein:

[0027] The encoder is used to encode historical state and action information within a preset time range to obtain encoded trajectory information;

[0028] The actor network is used to generate a target action based on the encoded trajectory information;

[0029] The critic network is configured to determine a predicted reward for the target action based on the target action and a current state corresponding to the target action;

[0030] The explorer is used to determine an exploration capability adjustment value based on the action probability distribution in the current state;

[0031] The strategy optimization module is used to adjust the parameters of the actor network based on the predicted reward and the exploration capability adjustment value.

[0032] In one embodiment, an encoder includes a buffer and an encoding component, wherein:

[0033] The buffer is used to obtain historical status and action information within a preset time range;

[0034] The encoding component is used to encode the historical state and action information within a preset time range to obtain encoded trajectory information.

[0035] In one embodiment, the explorer is further configured to:

[0036] Calculate the spatial distance between the action probability distribution and the uniform distribution in the current state;

[0037] The spatial distance is used as the exploration capability adjustment value.

[0038] In a third aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described above when executed by a processor.

[0040] The above-mentioned reinforcement learning method, action generation system, computer device and storage medium obtain encoded trajectory information by encoding historical states and action information within a preset time range; generate a target action based on the encoded trajectory information and a preset deep learning network; determine the predicted reward of the target action based on the target action and the current state corresponding to the target action; determine the exploration ability adjustment value based on the action probability distribution in the current state; adjust the parameters of the deep learning network based on the predicted reward and the exploration ability adjustment value. By encoding historical states and action information, context-based decision-making can be achieved, and by determining the exploration ability adjustment value according to the action probability distribution in the current state, the intelligent agent can be encouraged to fully explore the state space and avoid actions with valuable measures caused by deterministic strategies, thereby achieving the effect of improving the robustness of strategy generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1A diagram of an application environment of a reinforcement learning method in one embodiment;

[0042] Figure 2 is a schematic diagram of a process of a reinforcement learning method in an embodiment;

[0043] Figure 3 is a structural block diagram of an action generation system in one embodiment;

[0044] Figure 4 is a structural block diagram of an action generation system in another embodiment;

[0045] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] The reinforcement learning method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. After the terminal 102 generates an action, feedback can be obtained in the environment. It can be based on the corresponding feedback information for the generated action by the server 104, or it can be that the terminal 102 collects the state of the surrounding environment through sensors, etc. The terminal 102 encodes the historical state and action information within the preset time range to obtain the encoded trajectory information; based on the encoded trajectory information and the preset deep learning network, the target action is generated; based on the target action and the current state corresponding to the target action, the predicted return of the target action is determined; based on the probability distribution of the action in the current state, the exploration ability adjustment value is determined; based on the predicted return and the exploration ability adjustment value, the parameters of the deep learning network are adjusted. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0048] In one embodiment, Figure 2 As shown, a reinforcement learning method is provided, which is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:

[0049] Step S100, encoding the historical state and action information within a preset time range to obtain encoded trajectory information.

[0050] The historical status and action information may be historical status and action information collected within a preset time range.

[0051] The state may be the state of the terminal 102 itself, or the characteristics and information of the environment in which the terminal 102 is located. Exemplarily, the state may include the current time, location, etc. When the reinforcement learning method is applied to a game environment, the state may include the position coordinates, speed and acceleration of the character itself, as well as the weather conditions and light intensity in the game environment; when the reinforcement learning method is applied to robots and automatic driving, the state may include sensor data, such as laser radar ranging, infrared sensor readings, etc.; when the reinforcement learning method is applied to resource scheduling and management, the state may also include the state of currently available resources, such as battery power, battery health status, storage capacity, etc., which is not limited in this embodiment.

[0052] The action information may include the target action generated by the terminal 102. For example, when the reinforcement learning method is applied to a game environment, a robot, or autonomous driving, the target action may be adjusting to a specified direction, moving a specified number of steps in a specified direction, etc.; when the reinforcement learning method is applied to resource scheduling and management, the target action may be an operation or control instruction sent to a specific object, which is not limited in this embodiment.

[0053] Furthermore, the action information within the preset time range may include multiple target actions generated by the terminal 102 within the preset time range.

[0054] The historical state and action information is encoded to obtain the encoded trajectory information, which can be converted into a compact representation, namely, the encoded trajectory information, through an encoder or other encoding network. By encoding the historical state and action information to obtain the encoded trajectory information, the long-term dependency between the state and the action can be captured, providing richer context information for subsequent action generation.

[0055] Step S200: generating a target action based on the encoded trajectory information and a preset deep learning network.

[0056] Among them, based on the encoded trajectory information and the preset deep learning network, the preset deep learning network can be called to process the encoded trajectory information, so as to generate the action with the highest value corresponding to the encoded trajectory information as the target action.

[0057] Step S300, determining a predicted reward for the target action based on the target action and a current state corresponding to the target action.

[0058] Among them, after executing the target action, you can get feedback on the current state obtained under the target action. According to the target action and the current state corresponding to the target action, you can judge the immediate reward that may be obtained after collecting the target action and the expected cumulative reward in the future.

[0059] Step S400: determining an exploration capability adjustment value based on the action probability distribution in the current state.

[0060] Among them, exploration ability is the ability to explore unknown areas, and the exploration ability adjustment value can be the difference between the probability distribution of actions and the uniform distribution. It can be understood that the deep learning network can screen the multiple candidate actions in the current state according to their value, that is, the probability of selection, and determine the candidate action with the highest value as the target action. When the probability distribution of multiple candidate actions is uneven in the same state, it means that the deep learning network will follow a more certain strategy to give the target action, and the ability to explore unknown areas is low. By calculating the exploration ability adjustment value, it is possible to avoid the strategy of the deep learning network tending to a deterministic strategy, thereby avoiding missing valuable target actions.

[0061] Step S500: adjusting the parameters of the deep learning network based on the predicted reward and the exploration capability adjustment value.

[0062] Among them, based on the predicted return and exploration ability adjustment value, the parameters of the deep learning network are adjusted. This can be done through optimization algorithms such as gradient descent to find a balance between minimizing the difference between the predicted return and the actual return and improving the exploration ability, thereby improving the performance on specific tasks while ensuring the optimization strategy, thereby achieving the effect of improving robustness.

[0063] A reinforcement learning method provided in this embodiment encodes historical state and action information within a preset time range to obtain encoded trajectory information; generates a target action based on the encoded trajectory information and a preset deep learning network; determines the predicted reward of the target action based on the target action and the current state corresponding to the target action; determines an exploration capability adjustment value based on the action probability distribution in the current state; and adjusts the parameters of the deep learning network based on the predicted reward and the exploration capability adjustment value. By encoding historical state and action information, context-based decision-making can be achieved, and by determining the exploration capability adjustment value according to the action probability distribution in the current state, the intelligent agent can be encouraged to fully explore the state space, avoiding actions with valuable measures caused by deterministic strategies, thereby achieving the effect of improving the robustness of strategy generation.

[0064] In one embodiment, encoding the historical state and action information within a preset time range to obtain the encoded trajectory information includes:

[0065] Based on the time sliding window, obtain the historical status and action information within the preset time range;

[0066] The historical state and action information are encoded based on an encoder to obtain encoded trajectory information.

[0067] Among them, the time sliding window can be a time window of fixed length. As the agent interacts with the environment, the time sliding window continues to slide forward to keep the latest historical state and action information.

[0068] The historical state and action information is encoded based on an encoder, which can be a recurrent neural network such as RNN, LSTM, GRU, etc., to convert the historical state and action information into a vector or matrix, that is, to encode the trajectory information, thereby capturing the patterns and trends in the time series data and providing effective information for subsequent target actions.

[0069] In a specific embodiment, the encoder may be implemented using a Transformer encoder.

[0070] A reinforcement learning method provided in this embodiment can efficiently manage and utilize historical information by acquiring historical state and action information based on a time sliding window and obtaining encoded trajectory information based on an encoder, so that the intelligent agent has stronger adaptability and robustness when facing a complex and changing environment, thereby achieving the effect of improving the robustness of strategy generation.

[0071] In one embodiment, obtaining the target action based on the encoded trajectory information and a preset deep learning network includes:

[0072] Based on the encoded trajectory information and a preset deep learning network, generating a probability distribution of multiple candidate actions;

[0073] Based on the probability distribution of the multiple actions to be selected, a target action is determined.

[0074] Among them, a preset deep learning network may be called to process the encoded trajectory information, and a probability distribution of multiple actions to be selected may be generated through the deep learning network. It is understandable that the preset deep learning network may be a neural network such as a policy network, which may output a probability distribution including multiple actions to be selected. The probability distribution of multiple actions to be selected includes multiple actions to be selected and the possibility of each action to be selected being selected.

[0075] It is understandable that by generating a probability distribution, the relative advantages and disadvantages of different actions can be clearly expressed, and it is also convenient for calculating the exploration ability adjustment value when the exploration ability is adjusted later.

[0076] Based on the probability distribution of the multiple candidate actions, the target action is determined, and the action with the highest selection probability is selected as the target action, so as to avoid premature convergence to a suboptimal solution while retaining the exploration capability.

[0077] A reinforcement learning method provided in this embodiment determines a target action by generating a probability distribution of multiple candidate actions. This can improve the quality of decision-making while enhancing the exploration ability of the algorithm, thereby achieving the effect of improving the flexibility and robustness of the strategy.

[0078] In one embodiment, determining the predicted reward of the target action based on the target action and the current state corresponding to the target action includes:

[0079] Obtaining the current state corresponding to the target action;

[0080] Based on the target action and the current state, calculate the future reward distribution of the target action;

[0081] Mapping at least one sample randomly sampled from a uniform distribution to the future return distribution to obtain at least one quantile value;

[0082] Based on the percentile value, a predicted reward for the target action is determined.

[0083] The current state corresponding to the target action may be the state information used in the generation phase of the target action. The current state is used to represent the environment state and the state of the agent when generating the target action.

[0084] Based on the target action and the current state, the future reward distribution of the target action is calculated. The future reward distribution can be the distribution of future rewards determined based on the target action and the current state. It can be understood that by calculating the future reward distribution, a comprehensive perspective of various rewards that may be obtained after taking the target action can be obtained, rather than a single expected value, so that uncertainty and risk can be better reflected, which helps to make more robust decisions.

[0085] Furthermore, in this embodiment, the future return distribution can be modeled based on an implicit quantile network. For example, random returns can be modeled by an implicit quantile network.

[0086] At least one sample randomly sampled from a uniform distribution is mapped to the future reward distribution to obtain at least one quantile value. This can be achieved by sampling one or more samples from a uniform distribution and pushing these samples forward to corresponding quantile values ​​sampled from the future reward distribution to obtain at least one quantile value.

[0087] The predicted return of the target action can be determined based on the percentile value. The predicted return can be determined by taking the average value, median value, etc., or it can be obtained by one or more percentile values ​​selected based on the needs of the application scenario.

[0088] A reinforcement learning method provided in this embodiment calculates the future reward distribution of the target action and maps the quantile value from the uniform distribution to determine the predicted reward. This can accurately evaluate the possible reward distribution of the target action in the future, thereby improving the robustness of strategy generation.

[0089] In one embodiment, determining the exploration capability adjustment value based on the action probability distribution in the current state includes:

[0090] Calculate the spatial distance between the action probability distribution and the uniform distribution in the current state;

[0091] The spatial distance is used as the exploration capability adjustment value.

[0092] The action probability distribution in the current state may be a probability distribution of multiple candidate actions obtained by the deep learning network based on the encoded trajectory distribution. The uniform distribution may be a uniform distribution with the same probability for multiple candidate actions.

[0093] The spatial distance between the action probability distribution and the uniform distribution in the current state is calculated, which can be obtained by calculating in a manner such as KL divergence, cross entropy, divergence regularization, Wasserstein distance, etc.

[0094] After obtaining the spatial distance between the action probability distribution and the uniform distribution, the spatial distance can be used as an exploration capability adjustment value to adjust the parameters of the deep learning network.

[0095] A reinforcement learning method provided in this embodiment calculates the spatial distance between the action probability distribution and the uniform distribution in the current state as the exploration ability adjustment value, so that the intelligent agent can automatically adjust the exploration behavior in different situations, which helps to more effectively discover the optimal strategy and can achieve the effect of improving the robustness and comprehensiveness of strategy generation.

[0096] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0097] Based on the same inventive concept, the embodiment of the present application also provides an action generation system for implementing the above-mentioned reinforcement learning method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more action generation system embodiments provided below can refer to the limitations of the reinforcement learning method above, and will not be repeated here.

[0098] In one embodiment, Figure 3 As shown, an action generation system is provided, including an encoder 100, an actor network 200, a critic network 300, an explorer 400 and a policy optimization module 500, wherein:

[0099] The encoder 100 is used to encode historical state and action information within a preset time range to obtain encoded trajectory information;

[0100] The actor network 200 is used to generate a target action based on the encoded trajectory information;

[0101] The critic network 300 is used to determine a predicted reward of the target action based on the target action and a current state corresponding to the target action;

[0102] The explorer 400 is used to determine the exploration capability adjustment value based on the action probability distribution in the current state;

[0103] The strategy optimization module 500 is used to adjust the parameters of the actor network 200 based on the predicted reward and the exploration capability adjustment value.

[0104] In one embodiment, the encoder 100 includes a buffer and an encoding component, wherein:

[0105] The buffer is used to obtain historical status and action information within a preset time range;

[0106] The encoding component is used to encode the historical state and action information within a preset time range to obtain encoded trajectory information.

[0107] Furthermore, the encoding component may be a Transformer encoder.

[0108] In one embodiment, the explorer 400 is further configured to:

[0109] Calculate the spatial distance between the action probability distribution and the uniform distribution in the current state;

[0110] The spatial distance is used as the exploration capability adjustment value.

[0111] In order to more clearly illustrate the technical solution of the present application, the present application also provides a detailed embodiment.

[0112] In one embodiment, Figure 4 As shown, an action generation system is provided, which is applied to an intelligent agent. The action generation system includes an encoder, an actor network, a critic network and an explorer, wherein:

[0113] The encoder is a historical information encoder based on Transformer. The design of Transformer encoder aims to make full use of historical state and action information to improve the decision-making effect in reinforcement learning.

[0114] Specifically, the encoder includes a buffer and a Transformer encoder, where:

[0115] The buffer is a sliding window buffer. The historical state and action information are stored in a sliding window buffer. This buffer is constantly updated to ensure that the encoder can use the latest historical trajectory information. In this way, the model will add new historical state and action information to the buffer every time it interacts with the environment to keep real-time tracking of historical state and action information.

[0116] The Transformer encoder is an encoder based on historical trajectory information, which is used to process the historical trajectory information in the buffer and combine it with the current state state s obtained in the environment to generate the final state representation x. This combination ensures that the current decision can fully take into account the past environmental dynamics and action selection.

[0117] In order to improve the stability of training and the convergence of the model, the Transformer encoder adopts the strategy of "pre-training + joint training and fine-tuning". In the pre-training stage, the Transformer is trained to predict the current state state s by encoding trajectory information, and compares the prediction result with the actual state, so as to gradually learn how to effectively convert historical information into an accurate representation of the current state. After pre-training, the encoder enters the joint training stage, in which the Transformer encoder fine-tunes itself together with other components of reinforcement learning, such as actors and critics, to better adapt to the optimization goals of the overall system.

[0118] Through the pre-training phase, Transformer has been able to generate relatively accurate state representations. On this basis, joint training and fine-tuning further optimize the performance of the encoder, making it stable during the reinforcement learning training process and accelerating the convergence of the overall system.

[0119] Through the above method, the Transformer encoder can not only effectively utilize historical information, but also ensure the stable performance and rapid convergence of the model in complex environments through pre-training and fine-tuning mechanisms.

[0120] The actor network is an actor based on noise sampling, where the actor Actor can include a deep neural network. The actor network samples noise from the base distribution. Specifically, it converts the base distribution P(X)∈M(R d ) is converted into action space and generates actions in a sample-to-sample manner, that is, the target action a = π(s, ξ; θ π ): S×X→A, where s is the current state, ξ is the noise collected from the basic distribution P(X), that is, ξ~P(X), S is the state space, X is the noise space, and A is the action space. θ π is the parameter of the actor network, which can also be written as a~π(·|s;θ π ), where the strategy is a random strategy, and the action in state s can be generated by a lightweight method, that is, first sampling ξ from P(X) and then using a = π(s, ξ; θ π) is transformed to obtain the target action action a under state s. Through the above push-forward distribution, higher expression and modeling capabilities can be demonstrated.

[0121] By sampling noise in the underlying distribution, the model can introduce appropriate randomness in the action generation process, which not only enhances the exploration ability of the strategy, but also better captures the diversity and nonlinear relationships in complex environments. This method can effectively address the limitations of the Gaussian distribution in traditional random strategies, making the strategy perform better in multi-peak distribution and highly nonlinear environments.

[0122] The critic network is a critic with quantile representation. The critic uses an implicit quantile network IQN to push samples sampled from the uniform distribution U(0, 1) to the corresponding quantile values ​​sampled from the reward distribution. The approximation of the reward distribution is maintained by a weighted mixture of N Dirac functions. Specifically, the implicit quantile distribution can be used To model the random return and update it. Where s and a represent the state and action respectively, represents the quantile, θ z Represents the parameters of the implicit quantile distribution network. i <τ i+1 Sorted Quantile Samples For a random return Z π (s, a), with parameters θ z The implicit quantile distribution induced by IQN is defined as a weighted mixture of N Dirac distributions:

[0123]

[0124] in, and Furthermore, for:

[0125]

[0126] Denotes the random return Z π The inverse cumulative distribution function of (s, a) is expressed as:

[0127]

[0128] For risk-sensitive or other task-specific purposes, a reward-reshaping type utility function ψ(·) can be adopted to reshape the policy’s random reward R(s, a).

[0129] Among them, the critic network uses the state-action utility function to calculate the expected utility after taking action a in state s. The formula of the state-action utility function is:

[0130]

[0131] Among them, s is the current state, a is the target action, and s t+1 is the state at the next moment, R is the immediate reward r, γ is the discount coefficient, and P is the transition probability distribution of the environment, where s0 = s, a0 = a, and the corresponding state utility function is:

[0132]

[0133] The above formula satisfies the following utility Bellman equation:

[0134]

[0135] For a given policy π, the critic network can be Get predicted returns to evaluate its performance, where μ0 is the initial state distribution.

[0136] In some other embodiments, the critic network can also represent the quantile return through a quantile regression network QRN and a fully parameterized quantile function FQF.

[0137] The explorer is an explorer based on distribution differences. It is understandable that by introducing diverse behaviors in the strategy, the exploration ability of the model can be enhanced. The explorer guides the exploration of the target action by comparing the difference between the strategy distribution and the uniform distribution, thereby ensuring that the strategy is not too concentrated in a specific area, thereby improving the exploration ability of the strategy and increasing the chance of discovering a better strategy. When the difference between the strategy distribution and the uniform distribution is small, it means that the strategy has a good exploration ability; otherwise, it may mean that the strategy is too concentrated on certain actions and lacks exploration ability.

[0138] Comparing the differences between distributions can be achieved through maximum mean difference (MMD), KL divergence, Wasserstein distance, entropy regularization, divergence regularization, etc.

[0139] For example, when using KL divergence, the explorer can use KL divergence to quantify the degree of deviation of the policy distribution from the uniform distribution, and optimize the policy by minimizing the deviation. When using Wasserstein distance, the explorer can use Wasserstein distance to guide the policy to move closer to the uniform distribution, measure the "geographical" distance between two probability distributions, and enhance the exploration ability of the policy. When using entropy regularization, by increasing the entropy of the policy, the explorer can make the policy distribution more uniform, thereby improving the exploratory nature. When using divergence regularization, by imposing some divergence regularization constraints on the policy distribution, such as Jensen-Shannon divergence, the explorer can ensure that the policy is not too concentrated and maintain good exploratory nature.

[0140] This embodiment takes the maximum mean difference MMD as an example. Let F be a unit ball in a reproducing kernel Hilbert space (RKHS) H, defined on a compact metric space X. Then, the MMD between two distributions p and q can be expressed as:

[0141] MMD(p,q):=sup f∈F (E x～p [f(x)]-E y～p [f(y)])

[0142] For the maximum mean difference MMD, it uses an approximate calculation method that only needs to be sampled from the distribution without explicitly knowing its density function. Specifically, given m samples (x1, ...x m ) and n samples (y1, ...y n ), then the MMD between p and q can be approximated by the following formula:

[0143]

[0144] Wherein, k is a kernel function, which is used to measure the similarity between two data points, and the reference distribution can be a uniform distribution u(·|s) in the action space A. In this embodiment, dm(θ π ) represents the sample-based regularization term of the explorer, i.e., the strategy distribution π(·|s;θ π ) and the uniform distribution u(·|s) is the expected maximum mean difference (MMD), which is given by the following formula:

[0145]

[0146] Among them, θ π are the parameters of the actor network, d π u0represents the state occupancy measure relative to policy π, defined as

[0147] It can be understood that the policy distribution π(·|s;θ π )’s exploration ability and dm(θ π ). By reducing the variance, the explorer can give diverse actions to the policy, thus effectively incentivizing exploration.

[0148] According to the exploration ability adjustment value of the explorer and the predicted return of the critic network, the objective function of the actor network is adjusted through the BP back-propagation algorithm.

[0149] In the process of policy optimization, the objective function of the actor network is used to maximize the expected utility while maintaining good exploration ability. The specific formula is as follows:

[0150]

[0151] Among them, α represents the weight of the regularization term, and ψ is the utility function. By maximizing J ψ (θ π ), the strategy not only pursues a higher expected utility, but also maintains the ability to explore due to the existence of the maximum mean difference MMD regularization term. The optimization process can be divided into two steps, where:

[0152] First, distributed TD learning is used to update the parameters of the critic network, and its loss function is defined as follows:

[0153]

[0154] in, It can be efficiently optimized by stochastic gradient descent (SGD). The distributed TD learning process can be performed by minimizing the following Huber quantile regression loss:

[0155]

[0156] Among them, κ is a constant threshold, δ ij (s,a) is the pairwise TD error between the implicit quantile approximations of two consecutive steps, calculated as:

[0157]

[0158] Among them, a'~π(·|s′), and is the quantile based on two random samples and Calculated.

[0159] Secondly, according to Jψ (θ π ) Use the gradient ascent method to iteratively optimize the parameters of the strategy.

[0160] It is understandable that the strategy θ π The randomness of affects the action selection, including the generation of the target action a~π(·|s;θ π ), and functions However, it is not feasible to access the density function of the push-forward strategy with complex deep neural networks (DNNs), so J ψ (θ π ), namely The calculation of will be very difficult. Therefore, in this embodiment, a random utility value policy gradient (SUVPG) theorem is adopted, and approximate calculation is performed only based on the sample of the strategy, so that the strategy can be optimized.

[0161] The action generation system provided in this embodiment uses Transformer to encode historical state and action information, so that the intelligent agent can use richer contextual information. This way of combining historical information with current state enables the intelligent agent to make more accurate and forward-looking decisions in complex environments, thereby improving the completion of tasks; by sampling noise from the basic distribution and mapping it to the action space, it breaks through the limitations of the traditional Gaussian distribution. The use of this forward distribution enhances the expressiveness of the strategy in complex and multimodal environments, and can better capture the diversity and nonlinear relationships in the environment, thereby improving the adaptability and performance of the strategy; the introduction of an explorer based on distribution differences effectively solves the problem of insufficient exploration ability in traditional reinforcement learning methods. By comparing the difference between the strategy distribution and the uniform distribution, the explorer can guide the intelligent agent to conduct more extensive exploration and avoid falling into the local optimal solution, thereby improving the strategy's ability to discover the global optimal solution; by pre-training the Transformer and fine-tuning it during the joint training process, the stability and convergence of online learning are effectively improved. This technical approach not only speeds up the model training process, but also reduces the instability that may occur during the training process, allowing the model to quickly adapt to new data and maintain efficient learning; because the randomness and diversity of the strategy generation process are enhanced, the strategy is more robust in the face of noise and environmental changes. Such improvements enable the intelligent agent to maintain efficient behavioral decisions under a wider range of environmental conditions.

[0162] Each module in the above action generation system can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0163] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a reinforcement learning method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0164] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0165] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the reinforcement learning method of any of the above embodiments is implemented:

[0166] Encode the historical state and action information within a preset time range to obtain encoded trajectory information;

[0167] Generate a target action based on the encoded trajectory information and a preset deep learning network;

[0168] Determining a predicted reward for the target action based on the target action and a current state corresponding to the target action;

[0169] Determining an exploration capability adjustment value based on the action probability distribution in the current state;

[0170] Based on the predicted reward and the exploration capability adjustment value, the parameters of the deep learning network are adjusted.

[0171] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the reinforcement learning method of any of the above embodiments is implemented:

[0172] Encode the historical state and action information within a preset time range to obtain encoded trajectory information;

[0173] Generate a target action based on the encoded trajectory information and a preset deep learning network;

[0174] Determining a predicted reward for the target action based on the target action and a current state corresponding to the target action;

[0175] Determining an exploration capability adjustment value based on the action probability distribution in the current state;

[0176] Based on the predicted reward and the exploration capability adjustment value, the parameters of the deep learning network are adjusted.

[0177] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0178] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0179] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0180] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A reinforcement learning method, characterized in that: Applied to the action generation system, the reinforcement learning method includes: Encode the historical state and action information within a preset time range to obtain encoded trajectory information; Generate a target action based on the encoded trajectory information and a preset deep learning network; Determining a predicted reward for the target action based on the target action and a current state corresponding to the target action; Determining an exploration capability adjustment value based on the action probability distribution in the current state; Based on the predicted reward and the exploration capability adjustment value, the parameters of the deep learning network are adjusted.

2. The reinforcement learning method according to claim 1, characterized in that: The encoding of the historical state and action information within a preset time range to obtain the encoded trajectory information includes: Based on the time sliding window, obtain the historical status and action information within the preset time range; The historical state and action information are encoded based on an encoder to obtain encoded trajectory information.

3. The reinforcement learning method according to claim 1, characterized in that: The obtaining of the target action based on the encoded trajectory information and the preset deep learning network includes: Based on the encoded trajectory information and a preset deep learning network, generating a probability distribution of multiple candidate actions; Based on the probability distribution of the multiple actions to be selected, a target action is determined.

4. The reinforcement learning method according to claim 1, characterized in that: The determining, based on the target action and the current state corresponding to the target action, the predicted reward of the target action includes: Obtaining the current state corresponding to the target action; Based on the target action and the current state, calculate the future reward distribution of the target action; Mapping at least one sample randomly sampled from a uniform distribution to the future return distribution to obtain at least one quantile value; Based on the percentile value, a predicted reward for the target action is determined.

5. The reinforcement learning method according to claim 1, characterized in that: The determining of the exploration capability adjustment value based on the action probability distribution in the current state includes: Calculate the spatial distance between the action probability distribution and the uniform distribution in the current state; The spatial distance is used as the exploration capability adjustment value.

6. An action generation system, characterized in that: The action generation system includes an encoder, an actor network, a critic network, an explorer, and a policy optimization module, wherein: The encoder is used to encode historical state and action information within a preset time range to obtain encoded trajectory information; The actor network is used to generate a target action based on the encoded trajectory information; The critic network is configured to determine a predicted reward for the target action based on the target action and a current state corresponding to the target action; The explorer is used to determine an exploration capability adjustment value based on the action probability distribution in the current state; The strategy optimization module is used to adjust the parameters of the actor network based on the predicted reward and the exploration capability adjustment value.

7. The action generation system according to claim 6, characterized in that: The encoder includes buffer and encoding components, where: The buffer is used to obtain historical status and action information within a preset time range; The encoding component is used to encode the historical state and action information within a preset time range to obtain encoded trajectory information.

8. The action generation system according to claim 6, characterized in that: The explorer is also used to: Calculate the spatial distance between the action probability distribution and the uniform distribution in the current state; The spatial distance is used as the exploration capability adjustment value.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images