Deep reinforcement learning training optimization method and device for adaptive experience replay
By constructing a summation tree in deep reinforcement learning and using predictive models to adaptively adjust sample priorities, the problem of low DRL training efficiency in large-scale power systems is solved, and a more efficient sample utilization and training process is achieved.
Patent Information
- Application Number
- CN202411451164.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-17
- Publication Date
- 2025-02-07
AI Technical Summary
In large-scale power systems, deep reinforcement learning (DRL) is difficult to effectively utilize massive operating data, resulting in low sample utilization and poor training efficiency. Existing empirical playback techniques, such as random empirical playback and priority empirical playback, have problems with low sample utilization efficiency and high computational burden.
By building a summation tree, we can efficiently calculate and update the priority of samples, and use the prediction model to predict the priority of samples, and integrate time difference error and reward priority to achieve adaptive correction of the priority of the experience pool sample to reduce sampling deviation and Calculate the burden.
The sampling and training efficiency is improved, ensuring that the model can converge to the optimal strategy faster and adapt to changes in complex power system environments.
Smart Images
Figure CN119398241A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a method and device for optimizing deep reinforcement learning training with adaptive experience replay. Background Art
[0002] At present, with the intelligent and large-scale development of power systems, deep reinforcement learning (DRL) has been widely used in many key fields such as intelligent scheduling, power load forecasting and energy management. DRL has shown strong adaptability and decision-making optimization capabilities, effectively coping with the complexity and uncertainty in power systems. However, with the continuous expansion of the scale of power systems, DRL also faces some new challenges in practical applications, among which the problems of low sample utilization and poor training efficiency are particularly prominent. In large-scale power systems, there are massive operating data with long time series spans and complex dynamics with environmental changes. In the face of such a large amount of data, DRL is often difficult to achieve effective utilization, resulting in a large number of redundant samples, which greatly affects the training efficiency. Experience Replay is a commonly used technology in reinforcement learning to improve the efficiency and stability of learning algorithms.
[0003] Existing experience replay technologies mainly include random experience replay and priority experience replay methods. In random experience, samples are randomly drawn from the experience pool, and all samples are given the same weight, which may cause some important samples that have a greater impact on the learning process to be rarely used, while unimportant or redundant samples are repeatedly used, resulting in low sample utilization efficiency; in the face of high-dimensional or complex tasks (such as large-scale power system scheduling), random experience replay may not be able to effectively capture the key features of the environment. Due to the high dimensionality and complexity of the data, it is difficult to extract the most effective samples for policy improvement by simple random sampling, resulting in unsatisfactory learning results. In addition, the sampling mechanism of random experience replay causes the latest experience data to not be used immediately, especially when the experience pool is large, new samples are likely not to be sampled for a long time, and the model continues to train with outdated experience, and cannot adapt to changes in the current environment in a timely manner. For environments with fast dynamic changes such as power systems, failure to use new experience in a timely manner will affect the adaptability of the strategy and cannot cope with rapidly changing loads or equipment states.
[0004] Prioritized experience replay requires calculating and updating the priority for each experience sample. In a large-scale experience pool, the process of calculating the priority of each sample and sampling according to the priority may be more complicated, and the large computational burden requires more computing resources and time. On the other hand, since prioritized experience replay tends to select samples with large errors, it may cause the agent to focus too much on certain specific samples, thereby introducing sampling bias. This bias may cause the model to overfit in certain extreme conditions, while ignoring the diversity of the overall environment. Summary of the invention
[0005] The present invention provides a deep reinforcement learning training optimization method and device with adaptive experience replay to achieve adaptive correction of experience pool sample priority, reduce sampling deviation and computational burden, and effectively improve sampling and training efficiency.
[0006] In order to solve the above technical problems, an embodiment of the present invention provides a deep reinforcement learning training optimization method with adaptive experience replay, comprising:
[0007] Building a sum tree based on the storage priority information of the experience pool, and calculating the sum of the first priority and the first sampling probability based on the sum tree;
[0008] Establishing a prediction model based on the sum of the first priorities and a regression algorithm, and obtaining a second priority of the prediction sample based on the prediction model;
[0009] Calculating a mean square error of the prediction samples based on the second priority level, and updating the prediction model based on the mean square error;
[0010] Calculating a second sampling probability and a reward priority of the sample based on the prediction model, and updating the priority information of the sample based on the reward priority;
[0011] The loss function is updated based on the importance weight, the gradient direction is calculated based on the updated loss function, the priority information of the sample is updated based on the gradient direction, and the sample is replayed back into the experience pool.
[0012] By constructing a sum tree, the present invention can efficiently calculate and update the priority of samples, reducing the computational burden of directly calculating the priority of each sample in traditional priority experience replay, and based on the prediction model, it can predict the priority of samples without directly calculating the priorities of all samples, thereby reducing the computational burden; by fusing the temporal difference error and the reward priority, the adaptive correction of the sample priority in the experience pool is achieved to reduce the sampling deviation, thereby effectively improving the sampling and training efficiency.
[0013] Furthermore, the construction of the sum tree based on the storage priority information of the experience pool includes:
[0014] Calculating the time difference error of the samples in the experience pool, and calculating the storage priority information of the samples based on the time difference error;
[0015] A sum tree is constructed based on the storage priority information, wherein the leaf nodes of the sum tree represent the storage priority, the middle branch node is the sum of the priorities of the left and right nodes, and the root node is the sum of the first priorities of the samples in the entire experience pool.
[0016] Further, the calculating the sum of the first priority and the first sampling probability based on the sum tree includes:
[0017] Obtaining the sum of the first priority based on the root node of the summation book;
[0018] The first sampling probability of the sample calculated based on the storage priority information is p(x)
[0019]
[0020] Among them, p i =|δ i |+∈ is the storage priority of sample i, δ i is the time difference error of sample i, ∈>0, is a small constant to ensure that the priority of the sample is positive, and a is the degree of using the priority.
[0021] Furthermore, the establishing of a prediction model based on the sum of the first priorities and a regression algorithm, and obtaining a second priority of the prediction sample based on the prediction model, includes:
[0022] A prediction model is constructed based on the regression algorithm, the sum of the first priority and the sum of the sample timestamps, and the sum of the second priority is obtained based on the prediction model; the prediction model includes:
[0023] x new =A1x1+A2x2+A3
[0024]
[0025] Among them, x1 is the sum of the first priority, x2 is the sum of the sample timestamps, τ i is the sample timestamp, A i Represents the parameters of the prediction model.
[0026] Further, the calculating a mean square error of the prediction samples based on the second priority and updating the prediction model based on the mean square error includes:
[0027] Minimize the mean square error between the second priority of the predicted sample and the true value of the time difference error based on the prediction model; the second priority is the predicted value of the time difference error output by the prediction model;
[0028] The prediction model is updated based on the exponential moving average method, specifically:
[0029] x new.t =ηx real.t-1 +(1-η)x new.t-1
[0030] Among them, x new.t is the time difference error prediction value at time t, x real.t-1 is the true value of the time difference error at the previous moment, x new.t-1 is the time difference error prediction value of the previous moment, and η is the smoothing coefficient.
[0031] Further, the calculating the second sampling probability and reward priority of the sample based on the prediction model, and updating the priority information of the sample based on the reward priority, includes:
[0032] Calculating the true sampling probability of the sample based on the prediction model, and calculating the reward priority of the sample;
[0033] A reward sampling probability of the sample is calculated based on the reward priority, and the second sampling probability is calculated based on the true sampling probability and the reward sampling probability.
[0034] Further, the updating of the loss function based on the importance weight, calculating the gradient direction based on the updated loss function, updating the priority information of the sample based on the gradient direction, and replaying the sample back to the experience pool includes:
[0035] Calculating an importance weight of a loss function based on the second sampling probability, and updating the loss function based on the importance weight;
[0036] The gradient direction is calculated based on the updated loss function, and the priority information of the sample is updated based on the gradient direction, and the sample is replayed back into the experience pool.
[0037] In a second aspect, the present invention provides a deep reinforcement learning training optimization device with adaptive experience replay, comprising: a sum tree construction module, a prediction model construction module, a model update module, a priority update module and a replay module;
[0038] The sum tree construction module is used to construct a sum tree based on the storage priority information of the experience pool, and calculate the sum of the first priority and the first sampling probability based on the sum tree;
[0039] The prediction model building module is used to establish a prediction model based on the sum of the first priorities and a regression algorithm, and obtain a second priority of the prediction sample based on the prediction model;
[0040] The model updating module is used to calculate a mean square error of the prediction samples based on the second priority level, and update the prediction model based on the mean square error;
[0041] The priority updating module is used to calculate the second sampling probability and reward priority of the sample based on the prediction model, and update the priority information of the sample based on the reward priority;
[0042] The replay module is used to update the loss function based on the importance weight, calculate the gradient direction based on the updated loss function, update the priority information of the sample based on the gradient direction, and replay the sample back to the experience pool.
[0043] In a third aspect, the present application also provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the deep reinforcement learning training optimization method with adaptive experience replay as described above.
[0044] In a fourth aspect, the present application also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the deep reinforcement learning training optimization method with adaptive experience replay as described above. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 A schematic diagram of a flow chart of a deep reinforcement learning training optimization method with adaptive experience replay provided in an embodiment of the present invention;
[0046] Figure 2 A schematic diagram of the structure of a deep reinforcement learning training optimization device with adaptive experience replay provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0047] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0048] The terms "first" and "second" and the like in the specification, claims and drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.
[0049] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0050] First, some terms in this application are explained to facilitate understanding by those skilled in the art.
[0051] (1) Temporal-Difference error (TD-error) is a key concept in reinforcement learning. In particular, in reinforcement learning algorithms based on value functions, TD error is used to measure the difference between the predicted value (predicted future return) and the actual value. This difference is used to update the value function, so that the agent can learn better strategies.
[0052] Example 1
[0053] See also Figure 1 , Figure 1 The present invention provides a method for optimizing deep reinforcement learning training with adaptive experience replay, which includes steps 101 to 105, as follows:
[0054] Step 101: constructing a summation tree based on the storage priority information of the experience pool, and calculating the sum of the first priority and the first sampling probability based on the summation tree;
[0055] In this embodiment, the step of constructing a sum tree based on the storage priority information of the experience pool includes:
[0056] Calculating the time difference error of the samples in the experience pool, and calculating the storage priority information of the samples based on the time difference error;
[0057] A sum tree is constructed based on the storage priority information, wherein the leaf nodes of the sum tree represent the storage priority, the middle branch node is the sum of the priorities of the left and right nodes, and the root node is the sum of the first priorities of the samples in the entire experience pool.
[0058] In this embodiment, the time difference error includes:
[0059]
[0060] Among them, r i is the reward at the i-th moment, Q +represents the target Q value, Q is the current Q value, S i is the state space, It is the state s in which action u is executed i The next state. ξ<1, is the discount coefficient.
[0061] In this embodiment, the time difference error is an important indicator for evaluating the performance of the current strategy. It reflects the error between the value function prediction and the actual observation. The priority of the sample can be dynamically adjusted according to the size of this error. The time difference error of each sample in the experience pool is calculated and converted into the storage priority information of the corresponding sample.
[0062] In this embodiment, the time difference error may be converted into storage priority information by using an absolute value or a weighted method.
[0063] In this embodiment, the storage priority information of sample i is p i =|δ i |+∈,∈>0, is a small constant that ensures that the priority of the sample is positive.
[0064] In this embodiment, the calculating the sum of the first priority and the first sampling probability based on the sum tree includes:
[0065] Obtaining the sum of the first priority based on the root node of the summation book;
[0066] The first sampling probability of the sample calculated based on the storage priority information is p(x)
[0067]
[0068] Among them, p i =|δ i |+∈ is the storage priority of sample i, δ i is the time difference error of sample i, ∈>0, is a small constant to ensure that the priority of the sample is positive, and a is the degree of using the priority.
[0069] In this embodiment, when a=0, the priority does not take effect, and when a=1, sampling is performed according to the current priority.
[0070] In this embodiment, by storing priority information and calculating the first sampling probability, the samples in the experience pool can be effectively introduced into the training process. The storage priority information is based on the TD error to dynamically adjust the importance of each sample in learning; and the first sampling probability further maps this importance to the actual sampling process, ensuring that in each training, those samples that are most helpful for improving the current strategy can be selected in a targeted manner. This adaptive sampling mechanism significantly improves the training efficiency and effect of deep reinforcement learning, allowing the model to converge to the optimal strategy faster in complex environments.
[0071] Step 102: establishing a prediction model based on a regression algorithm, and obtaining the sum of the second priority of the prediction samples based on the prediction model;
[0072] In this embodiment, establishing a prediction model based on a regression algorithm, and obtaining the sum of the second priorities of the prediction samples based on the prediction model, includes:
[0073] A prediction model is constructed based on the regression algorithm, the sum of the first priority and the sum of the sample timestamps, and the sum of the second priority is obtained based on the prediction model, wherein the prediction model includes:
[0074] x new =A1x1+A2x2+A3 (3)
[0075]
[0076] Among them, x1 is the sum of the first priority, x2 is the sum of the sample timestamps, τ i is the sample timestamp, A i Represents the parameters of the prediction model.
[0077] In this embodiment, a prediction model is constructed based on the regression algorithm, the sum of the first priority and the sum of the sample timestamps. The prediction model includes two input features, one of which is the sum of the first priority of the experience pool x old , obtained by reading the root node of the sum tree, and the other is the sum of the sample timestamps in the experience pool x t .
[0078] In this embodiment, when an experience enters the pool, the generation timestamp of the sample is recorded. For example, assuming that sample i has a sample timestamp τ i .
[0079] In this embodiment, each sample in the experience pool has a corresponding timestamp, which indicates the time when the sample was recorded or generated. The sum of the sample timestamps is used to measure the time distribution of all samples in the experience pool, reflecting the generation of samples within a certain time range. At the same time, the sample timestamp can indicate the "freshness" of the experience, helping to estimate the relative importance of the sample when selecting the sample.
[0080] In this embodiment, a prediction model is established based on a linear regression algorithm, and this model is used to predict the sum of the true priorities of the samples, where the sum of the true priorities includes the sum of the true time differential errors.
[0081] In this embodiment, the state, action, reward and other characteristics of the sample and the first priority corresponding to the sample are obtained, that is, the storage priority information. The first sampling probability is calculated based on the storage priority information, sampling is performed based on the first sampling probability, and the sampled prediction samples are input into the prediction model for training. The prediction samples are input into the prediction model to output a predicted TD-error sum, that is, the sum of the second priority. The training goal of the prediction model is to minimize the mean square error between the sum of the predicted time difference error and the sum of the real time difference error. Specifically:
[0082]
[0083] Where m represents the size of training data, x new.i represents the sum of the prediction time difference errors, that is, the sum of the first priority; x real Indicates the true value of the time difference error, which is the second priority.
[0084] Step 103: Calculate a mean square error of the prediction samples based on the second priority level, and update the prediction model based on the mean square error;
[0085] In this embodiment, calculating the mean square error of the prediction samples based on the second priority, and updating the prediction model based on the mean square error includes:
[0086] Minimize the mean square error between the second priority of the predicted sample and the true value of the time difference error based on the prediction model; the second priority is the predicted value of the time difference error output by the prediction model;
[0087] The prediction model is updated based on the exponential moving average method, specifically:
[0088] x new.t =ηx real.t-1 +(1-η)x new.t-1 (7)
[0089] Among them, x new.t is the time difference error prediction value at time t, x real.t-1 is the true value of the time difference error at the previous moment, x new.t-1 is the time difference error prediction value of the previous moment, and η is the smoothing coefficient.
[0090] In this embodiment, during the training iteration process, the prediction equation is updated based on the Exponential Moving Average (EMA) method. In this way, the prediction ability of the model can be gradually adjusted to optimize the sum of the true TD-error.
[0091] Step 104: Calculate a second sampling probability and a reward priority of the sample based on the prediction model, and update the priority information of the sample based on the reward priority;
[0092] In this embodiment, the calculating the second sampling probability and reward priority of the sample based on the prediction model, and updating the priority information of the sample based on the reward priority, includes:
[0093] Calculating the true sampling probability of the sample based on the prediction model, and calculating the reward priority of the sample;
[0094] A reward sampling probability of the sample is calculated based on the reward priority, and the second sampling probability is calculated based on the true sampling probability and the reward sampling probability.
[0095] In this embodiment, the constructed prediction model is used to predict each sample in the experience pool to obtain the time difference error of each sample, and the true sampling probability of the sample is calculated based on the time difference error.
[0096] In this embodiment, the reward priorities include:
[0097]
[0098] in, is the reward priority of the i-th sample, β>0, is a constant, r i is the reward at the ith moment.
[0099] When the reward value has upper and lower limits, the reward priority is:
[0100]
[0101] in, r max is the maximum reward value, r min is the minimum reward value.
[0102] In this embodiment, the reward sampling probability of the sample is calculated based on the reward priority, specifically:
[0103]
[0104] In this embodiment, the second sampling probability is calculated based on the real sampling probability and the reward sampling probability, specifically:
[0105] P j =λ*P T +(1-λ)*P R (11)
[0106] Among them, λ represents the importance ratio of the two factors, P T and P R Respectively represent the priority of the experience pool over TD-error and reward.
[0107] In this embodiment, the priority of samples can be adjusted dynamically through reward priority information, so that samples with greater impact on the learning process receive more attention. The introduction of reward priority helps reduce sampling bias and ensures that the model can better capture key features in complex tasks.
[0108] Step 105: Update the loss function based on the importance weight, calculate the gradient direction based on the updated loss function, update the priority information of the sample based on the gradient direction, and replay the sample back to the experience pool.
[0109] In this embodiment, the updating of the loss function based on the importance weight, calculating the gradient direction based on the updated loss function, updating the priority information of the sample based on the gradient direction, and replaying the sample back to the experience pool includes:
[0110] Calculating an importance weight of a loss function based on the second sampling probability, and updating the loss function based on the importance weight;
[0111] The gradient direction is calculated based on the updated loss function, and the priority information of the sample is updated based on the gradient direction, and the sample is replayed back into the experience pool.
[0112] In this embodiment, the second sampling probability p is evaluated. T The DDQN loss function under , the importance weight evaluation form, is as follows:
[0113]
[0114] Among them, q(x) is the first sampling probability of sample x, P(x) is the second sampling probability, and E is the expectation.
[0115] The importance weight of the loss function is:
[0116]
[0117] Then, the importance-weighted loss function is defined as:
[0118]
[0119] Among them, τ is the measurement factor.
[0120] In this embodiment, sample sampling training is performed after the weighted loss function is defined, and the priority weights of samples that can make the loss function decrease along the positive gradient direction are increased, and all the corrected priorities are input back into the experience pool.
[0121] In this embodiment, steps 101 to 105 are iteratively executed based on a preset replay batch, thereby achieving deep reinforcement learning training optimization with adaptive experience replay.
[0122] In this embodiment, by constructing a sum tree, the priority of the sample can be efficiently calculated and updated, reducing the computational burden of directly calculating the priority of each sample in the traditional priority experience replay, and based on the prediction model, the priority of the sample can be predicted without directly calculating the priority of all samples, thereby reducing the computational burden; by fusing the time difference error and the reward priority, the adaptive correction of the sample priority in the experience pool is achieved to reduce the sampling deviation, thereby effectively improving the sampling and training efficiency.
[0123] Please refer to Figure 2 , Figure 2 A schematic diagram of the structure of a deep reinforcement learning training optimization device with adaptive experience replay provided by an embodiment of the present invention, comprising: a sum tree construction module 201, a prediction model construction module 202, a model update module 203, a priority update module 204 and a replay module 205;
[0124] The sum tree construction module 201 is used to construct a sum tree based on the storage priority information of the experience pool, and calculate the sum of the first priority and the first sampling probability based on the sum tree;
[0125] The prediction model building module 202 is used to build a prediction model based on the sum of the first priorities and a regression algorithm, and obtain a second priority of the prediction sample based on the prediction model;
[0126] The model updating module 203 is used to calculate the mean square error of the prediction samples based on the second priority level, and update the prediction model based on the mean square error;
[0127] The priority updating module 204 is used to calculate the second sampling probability and reward priority of the sample based on the prediction model, and update the priority information of the sample based on the reward priority;
[0128] The replay module 205 is used to update the loss function based on the importance weight, calculate the gradient direction based on the updated loss function, update the priority information of the sample based on the gradient direction, and replay the sample back to the experience pool.
[0129] In this embodiment, the sum tree construction module includes:
[0130] Calculating the time difference error of the samples in the experience pool, and calculating the storage priority information of the samples based on the time difference error;
[0131] A sum tree is constructed based on the storage priority information, wherein the leaf nodes of the sum tree represent the storage priority, the middle branch node is the sum of the priorities of the left and right nodes, and the root node is the sum of the first priorities of the samples in the entire experience pool.
[0132] In this embodiment, the sum tree construction module includes:
[0133] Obtaining the sum of the first priority based on the root node of the summation book;
[0134] The first sampling probability of the sample calculated based on the storage priority information is p(x)
[0135]
[0136] Among them, p i =|δ i |+∈ is the storage priority of sample i, δ i is the time difference error of sample i, ∈>0, is a small constant to ensure that the priority of the sample is positive, and a is the degree of using the priority.
[0137] In this embodiment, the prediction model building module includes:
[0138] A prediction model is constructed based on the regression algorithm, the sum of the first priority and the sum of the sample timestamps, and the sum of the second priority is obtained based on the prediction model; the prediction model includes:
[0139] x new =A1x1+A2x2+A3
[0140]
[0141] Among them, x1 is the sum of the first priority, x2 is the sum of the sample timestamps, τ i is the sample timestamp, A i Represents the parameters of the prediction model.
[0142] In this embodiment, the model updating module includes:
[0143] Minimize the mean square error between the second priority of the predicted sample and the true value of the time difference error based on the prediction model; the second priority is the predicted value of the time difference error output by the prediction model;
[0144] The prediction model is updated based on the exponential moving average method, specifically:
[0145] x new.t =ηx real.t-1 +(1-η)x new.t-1
[0146] Among them, x new.t is the time difference error prediction value at time t, x real.t-1 is the true value of the time difference error at the previous moment, x new.t-1 is the time difference error prediction value of the previous moment, and η is the smoothing coefficient.
[0147] In this embodiment, the priority updating module includes:
[0148] Calculating the true sampling probability of the sample based on the prediction model, and calculating the reward priority of the sample;
[0149] A reward sampling probability of the sample is calculated based on the reward priority, and the second sampling probability is calculated based on the true sampling probability and the reward sampling probability.
[0150] In this embodiment, the playback module includes:
[0151] Calculating an importance weight of a loss function based on the second sampling probability, and updating the loss function based on the importance weight;
[0152] The gradient direction is calculated based on the updated loss function, and the priority information of the sample is updated based on the gradient direction, and the sample is replayed back into the experience pool.
[0153] The present invention realizes adaptive correction of sample priority in the experience pool by fusing the time difference error with adaptive experience replay of reward priority, reduces sampling deviation, and does not need to update the entire experience pool every time, thus reducing the computational burden and effectively improving sampling and training efficiency.
[0154] In an embodiment of the present invention, a terminal device is also provided, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the deep reinforcement learning training optimization method with adaptive experience replay as described above.
[0155] In an embodiment of the present invention, a computer-readable storage medium is also provided, and the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the deep reinforcement learning training optimization method with adaptive experience replay as described above.
[0156] Exemplarily, the computer program may be divided into one or more modules, one or more modules are stored in a memory and executed by a processor to implement the present invention. One or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in a terminal device.
[0157] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The terminal device may include, but is not limited to, a processor, a memory, and a display. Those skilled in the art will appreciate that the above components are merely examples of the terminal device and do not constitute a limitation on the terminal device. The terminal device may include more or fewer components than the components, or may combine certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.
[0158] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.
[0159] The memory can be used to store computer programs and / or modules. The processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function (such as a sound playback function, a text conversion function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, text message data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0160] Among them, if the module based on the deep reinforcement learning training optimization of an adaptive experience replay is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can be executed by the processor. The steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0161] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.
Claims
1. A deep reinforcement learning training optimization method with adaptive experience replay, characterized in that: include: Building a sum tree based on the storage priority information of the experience pool, and calculating the sum of the first priority and the first sampling probability based on the sum tree; Establishing a prediction model based on the sum of the first priorities and a regression algorithm, and obtaining a second priority of the prediction sample based on the prediction model; Calculating a mean square error of the prediction samples based on the second priority level, and updating the prediction model based on the mean square error; Calculating a second sampling probability and a reward priority of the sample based on the prediction model, and updating the priority information of the sample based on the reward priority; The loss function is updated based on the importance weight, the gradient direction is calculated based on the updated loss function, the priority information of the sample is updated based on the gradient direction, and the sample is replayed back into the experience pool.
2. The method for optimizing deep reinforcement learning training with adaptive experience replay according to claim 1, characterized in that: The step of constructing a summation tree based on the storage priority information of the experience pool includes: Calculating the time difference error of the samples in the experience pool, and calculating the storage priority information of the samples based on the time difference error; A summation tree is constructed based on the storage priority information, wherein the leaf nodes of the summation tree represent the storage priority, the middle branch node is the sum of the priorities of the left node and the right node, and the root node is the sum of the first priorities of the samples in the entire experience pool.
3. The method for optimizing deep reinforcement learning training with adaptive experience replay according to claim 2, characterized in that: The calculating the sum of the first priority and the first sampling probability based on the sum tree includes: Obtaining the sum of the first priority based on the root node of the summation book; The first sampling probability of the sample calculated based on the storage priority information is p(x) Among them, p i =|δ i |+∈ is the storage priority of sample i, δ i is the time difference error of sample i, ∈>0, is a small constant that ensures that the priority of the sample is positive; a is the degree of use of the priority.
4. The method for optimizing deep reinforcement learning training with adaptive experience replay according to claim 3, characterized in that: The step of establishing a prediction model based on the sum of the first priorities and a regression algorithm, and obtaining a second priority of the prediction sample based on the prediction model, includes: A prediction model is constructed based on the regression algorithm, the sum of the first priority and the sum of the sample timestamps, and the sum of the second priority is obtained based on the prediction model; the prediction model includes: <h2 style=";text-align:left;direction:ltr">x<h2 style=";text-align:left;direction:ltr"> new <h2 style=";text-align:left;direction:ltr"> =A1x1+A2x2+A3 Among them, x1 is the sum of the first priority, x2 is the sum of the sample timestamps, τ i is the sample timestamp, A i Represents the parameters of the prediction model.
5. The method for optimizing deep reinforcement learning training with adaptive experience replay according to claim 4, characterized in that: The calculating a mean square error of the prediction samples based on the second priority, and updating the prediction model based on the mean square error, comprises: Minimize the mean square error between the second priority of the predicted sample and the true value of the time difference error based on the prediction model; the second priority is the predicted value of the time difference error output by the prediction model; The prediction model is updated based on the exponential moving average method, specifically: x new.t =ηx real.t-1 +(1-n)x new.t-1 Among them, x new.t is the time difference error prediction value at time t, x real.t-1 is the true value of the time difference error at the previous moment, x new.t-1 is the time difference error prediction value of the previous moment, and η is the smoothing coefficient.
6. The method for optimizing deep reinforcement learning training with adaptive experience replay according to claim 5, characterized in that: The calculating the second sampling probability and reward priority of the sample based on the prediction model, and updating the priority information of the sample based on the reward priority, includes: Calculating the true sampling probability of the sample based on the prediction model, and calculating the reward priority of the sample; A reward sampling probability of the sample is calculated based on the reward priority, and the second sampling probability is calculated based on the true sampling probability and the reward sampling probability.
7. The method for optimizing deep reinforcement learning training with adaptive experience replay according to claim 6, characterized in that: The updating of the loss function based on the importance weight, calculating the gradient direction based on the updated loss function, updating the priority information of the sample based on the gradient direction, and replaying the sample back to the experience pool includes: Calculating an importance weight of a loss function based on the second sampling probability, and updating the loss function based on the importance weight; The gradient direction is calculated based on the updated loss function, and the priority information of the sample is updated based on the gradient direction, and the sample is replayed back into the experience pool.
8. A deep reinforcement learning training optimization device with adaptive experience replay, characterized in that: include: A sum tree construction module, a prediction model construction module, a model update module, a priority update module and a replay module; The sum tree construction module is used to construct a sum tree based on the storage priority information of the experience pool, and calculate the sum of the first priority and the first sampling probability based on the sum tree; The prediction model building module is used to establish a prediction model based on the sum of the first priorities and a regression algorithm, and obtain a second priority of the prediction sample based on the prediction model; The model updating module is used to calculate a mean square error of the prediction samples based on the second priority level, and update the prediction model based on the mean square error; The priority updating module is used to calculate the second sampling probability and reward priority of the sample based on the prediction model, and update the priority information of the sample based on the reward priority; The replay module is used to update the loss function based on the importance weight, calculate the gradient direction based on the updated loss function, update the priority information of the sample based on the gradient direction, and replay the sample back to the experience pool.
9. A terminal device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a deep reinforcement learning training optimization method with adaptive experience replay as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a deep reinforcement learning training optimization method with adaptive experience replay as described in any one of claims 1 to 7.