Replay ratio adjusting method applied to reinforcement learning decision model

By dynamically adjusting the playback ratio in the reinforcement learning decision model, and optimizing the sample ratio based on performance indicators and strategies, the learning instability and inefficiency caused by fixed playback ratios are solved, and the learning efficiency and final performance of the model are improved.

CN119005291APending Publication Date: 2024-11-22UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411120215.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the existing reinforcement learning decision-making model, fixed playback ratios lead to over-learning of early samples, ignoring fresh samples, or being unable to effectively utilize early samples, resulting in unstable learning training and unable to effectively improve learning efficiency and algorithm performance.

Method used

By dynamically adjusting the playback ratio of empirical playback, based on different stages of algorithm learning and current performance indicators, a playback ratio adjustment method based on performance feedback parameters and balance strategy is adopted to balance the writing speed of fresh experience and optimize the playback buffer. sample ratio.

Benefits of technology

The learning efficiency and final strategy performance of the reinforcement learning decision model are improved, ensuring that the model can effectively utilize various samples at different training stages, and the stability and efficiency are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005291A_ABST
    Figure CN119005291A_ABST
Patent Text Reader

Abstract

The invention discloses a replay ratio adjustment method applied to a reinforcement learning decision model, and the method comprises the steps: carrying out the random sampling of sample data in a current replay buffer region, determining a current training sample, the current replay buffer region corresponding to a current replay ratio, and the current replay ratio being the number ratio of new samples to old samples; training the reinforcement learning decision model based on the current training sample to determine a current performance feedback parameter; and if the current training round meets a preset replay ratio adjustment condition, determining a target replay ratio corresponding to the replay buffer area based on the current performance feedback parameter, the current replay ratio and at least one preset replay ratio adjustment strategy, so as to perform sample updating on the current replay buffer area based on the target replay ratio, the speed of writing fresh experience into the replay buffer area is balanced by dynamically adjusting the replay ratio, and the learning efficiency and the final strategy performance of the reinforcement learning decision model algorithm are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer processing technology, and in particular to a replay ratio adjustment method applied to a reinforcement learning decision model. Background Art

[0002] Reinforcement learning is a machine learning paradigm that learns decision-making strategies based on trial and error. In the reinforcement learning process, the agent interacts with the environment, selects an action based on the current state, and obtains corresponding feedback rewards or penalties from the environment. The goal of the agent is to learn a decision-making strategy that can obtain the maximum cumulative reward after experiencing a series of states. Experience replay can break the time correlation of data and improve learning efficiency by saving the interaction data between the agent and the environment in the replay buffer and sampling from it in subsequent learning. The replay ratio is the ratio between new samples and old samples in the replay buffer during the reinforcement learning training of the decision model.

[0003] Currently, the replay ratio in the training process of reinforcement learning decision models is fixed. However, a replay ratio that is too high may lead to over-learning of early samples and neglect of current fresh samples; while a replay ratio that is too low may lead to ineffective use of early samples, resulting in unstable learning and training. In summary, this method of fixed replay ratio cannot effectively improve the learning efficiency and algorithm performance of reinforcement learning decision models. Summary of the invention

[0004] The present invention provides a replay ratio adjustment method applied to a reinforcement learning decision model, so as to dynamically adjust the replay ratio of experience replay according to different stages of algorithm learning and current performance indicators, rather than adopting a fixed ratio. By dynamically adjusting the replay ratio to balance the speed of writing fresh experience into the replay buffer, the learning efficiency and final strategy performance of the reinforcement learning decision model algorithm are improved.

[0005] The present invention provides a replay ratio adjustment method applied to a reinforcement learning decision model, the method comprising:

[0006] Randomly sampling the sample data in the current replay buffer to determine the current training sample of the reinforcement learning decision model in the current training round; wherein the current replay buffer is used to store new samples and old samples required by the reinforcement learning decision model in the current training round, and the current replay buffer corresponds to a current replay ratio, which is the ratio of the number of the new samples to the number of the old samples;

[0007] Training the reinforcement learning decision model based on the current training sample to determine at least one current performance feedback parameter of the reinforcement learning decision model in the current training round;

[0008] If the current training round meets the preset replay ratio adjustment conditions, the target replay ratio corresponding to the replay buffer is determined based on the current performance feedback parameters, the current replay ratio and at least one pre-set replay ratio adjustment strategy, so as to perform sample update on the current replay buffer based on the target replay ratio; wherein the replay ratio adjustment strategy includes a replay ratio adjustment strategy based on performance feedback parameters and / or a replay ratio adjustment strategy based on a balanced replay ratio.

[0009] Optionally, the replay ratio adjustment strategy is a replay ratio adjustment strategy based on a performance feedback parameter, and determining a target replay ratio corresponding to the replay buffer based on the current performance feedback parameter, the current replay ratio, and at least one preset replay ratio adjustment strategy includes:

[0010] The current performance feedback parameter is processed according to a first preset update function to determine a target playback ratio.

[0011] Optionally, the current performance feedback parameter includes a current average cumulative reward parameter and a current state value parameter, and the first preset update function is:

[0012]

[0013] Wherein, A1 is the target replay ratio, A0 is the current replay ratio, Amax is the maximum replay ratio threshold, Amin is the minimum replay ratio threshold, Va is the current state value parameter, Va' is the state value parameter of the previous cycle of the current training round, R is the current average cumulative reward parameter, R' is the average cumulative reward parameter of the previous cycle of the current training round, and α1 and α2 are preset weight values.

[0014] Optionally, the replay ratio adjustment strategy is a replay ratio adjustment strategy based on a balanced replay ratio, and determining a target replay ratio corresponding to the replay buffer based on the current performance feedback parameter, the current replay ratio, and at least one preset replay ratio adjustment strategy includes:

[0015] Determining a model evaluation value of the reinforcement learning decision model after a current training round based on the at least one current performance feedback parameter;

[0016] Based on the model evaluation value and a preset model evaluation threshold, the current replay ratio is adjusted to determine a target replay ratio corresponding to the replay buffer.

[0017] Optionally, the at least one current performance feedback parameter includes a current cumulative reward parameter, a preset task completion rate parameter, and a current environment interaction efficiency parameter, and determining the model evaluation value of the reinforcement learning decision model after the current training round based on the at least one current performance feedback parameter includes:

[0018] Based on the weighted average of the current cumulative reward parameter, the preset task completion rate parameter and the current environment interaction efficiency parameter, the model evaluation value of the reinforcement learning decision model after the current training round is determined.

[0019] Optionally, adjusting the current replay ratio based on the model evaluation value and a preset model evaluation threshold to determine a target replay ratio corresponding to the replay buffer includes:

[0020] If the model evaluation value is greater than a preset model evaluation threshold, the current playback ratio is reduced and adjusted based on a preset adjustment step to obtain a target playback ratio;

[0021] If the model evaluation value is less than or equal to a preset model evaluation threshold, the current replay ratio is increased and adjusted based on a preset adjustment step to obtain a target replay ratio.

[0022] Optionally, the replay ratio adjustment strategy also includes a replay ratio adjustment strategy based on a decision model learning rate.

[0023] Optionally, the replay ratio adjustment strategy based on the decision model learning rate includes: when the current learning rate of the reinforcement learning decision model in the current training round changes, based on the linkage mapping relationship between the change rate and the replay ratio, determining the target replay ratio corresponding to the replay buffer.

[0024] Optionally, the performing sample updating on the current replay buffer based on the target replay ratio includes:

[0025] Based on the target replay ratio, the current replay buffer, and the reinforcement learning decision model determined in the previous training round, the current replay buffer is sampled and updated.

[0026] Optionally, the performing sample updating on the current replay buffer based on the target replay ratio and the reinforcement learning decision model determined in the previous training round includes:

[0027] Determining a second number of new samples to be explored based on a first number of sample data already in the current replay buffer and the target replay ratio;

[0028] According to the second quantity of the existing sample data and the reinforcement learning decision model determined in the previous training round, a second quantity of target exploration new samples is determined, and the target exploration new samples are added to the current replay buffer.

[0029] The technical solution of the embodiment of the present invention determines the current training sample of the reinforcement learning decision model in the current training round by randomly sampling the sample data in the current replay buffer, wherein the current replay buffer corresponds to the current replay ratio, and the current replay ratio is the ratio of the number of new samples to the number of old samples, so that the reinforcement learning decision model is trained based on the current training sample to determine at least one current performance feedback parameter of the reinforcement learning decision model under the current training round, and further, if the current training round meets the preset replay ratio adjustment condition, then based on the current performance feedback parameter, the current replay ratio and at least one pre-set replay ratio adjustment strategy, the target replay ratio corresponding to the replay buffer is determined to update the sample of the current replay buffer based on the target replay ratio. The technical solution provided by the embodiment of the present invention improves the learning efficiency and final strategy performance of the reinforcement learning decision model algorithm by dynamically adjusting the replay ratio to balance the speed of writing fresh experience into the replay buffer.

[0030] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 is a flow chart of a replay ratio adjustment method applied to a reinforcement learning decision model provided according to Embodiment 1 of the present invention;

[0033] Figure 2 is a schematic diagram of reinforcement learning according to an embodiment of the present invention;

[0034] Figure 3 is a flow chart of a replay ratio adjustment method applied to a reinforcement learning decision model provided according to Embodiment 2 of the present invention;

[0035] Figure 4 It is an overall flow chart of a replay ratio adjustment method applied to a reinforcement learning decision model provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] Embodiment 1

[0039] Figure 1 This is a flowchart of a replay ratio adjustment method applied to a reinforcement learning decision model provided in Example 1 of the present invention. This embodiment can be applied to any situation where the replay ratio needs to be adaptively adjusted during the training process of the reinforcement learning decision model.

[0040] Before introducing this solution, we can first briefly introduce the technical background of this solution. Reinforcement learning is a paradigm and methodology of machine learning, which is used to describe and solve the problem of how an intelligent agent can maximize its rewards or achieve specific goals by learning strategies during its interaction with the environment. The core concepts of reinforcement learning include intelligent agents and environments. Figure 2 ,like Figure 2 As shown in the figure, it consists of two parts: the agent and the environment. In the process of reinforcement learning, the agent and the environment are constantly interacting. After the agent obtains a certain state in the environment, it will use the state to output an action (acion), which is also called a decision. Then this action will be executed in the environment. The environment will output the next state and the reward brought by the current action based on the action taken by the agent. The purpose of the agent is to obtain as many rewards as possible from the environment.

[0041] Experience replay is a technique used in reinforcement learning. Its main purpose is to improve learning efficiency and stability by reusing or replaying past experience. Experience replay trains by randomly selecting a batch of samples from the replay buffer instead of directly using the latest experience data. This method helps alleviate the instability and inefficiency caused by the correlation and non-static distribution problems between samples during training. In addition, experience replay also helps solve the "catastrophic forgetting" problem that may occur when neural networks learn new tasks, that is, the previously learned information may be overwritten or destroyed in the process of adapting to new tasks. By reusing past experience, experience replay helps the model maintain knowledge of old tasks while continuously learning new tasks, thereby improving the generalization ability and adaptability of the model. The replay ratio can represent the ratio between the old training data in the replay buffer and the fresh data collected during the learning process.

[0042] For different task scenarios, the optimal replay ratio often varies. A replay ratio that is too high may lead to over-learning of early samples and neglect of current fresh samples; while a replay ratio that is too low may lead to the inability to effectively utilize early samples, resulting in unstable learning and training. Therefore, how to dynamically adapt the replay ratio according to the task scenario is crucial to improving the performance of the reinforcement learning algorithm. Based on this, the purpose of an embodiment of the present invention is to dynamically adjust the replay ratio of experience replay, rather than adopting a fixed replay ratio, to improve the learning efficiency and final performance of the algorithm in complex environments through a more refined data sampling strategy.

[0043] like Figure 1 As shown, the replay ratio adjustment method applied to the reinforcement learning decision model includes:

[0044] S110: Randomly sample the sample data in the current replay buffer to determine the current training sample of the reinforcement learning decision model in the current training round.

[0045] Among them, the reinforcement learning decision model is a method of learning how to make the best decision through the interaction between the agent and the environment. The replay buffer is related to the agent in the reinforcement learning system. The agent is responsible for interacting with the environment to collect data and collect historical data (including current state, action, next state, etc.) to provide reserves for the training samples of the reinforcement learning decision model. In this process, the replay buffer plays the role of storing and replaying these historical data, so that the reinforcement learning decision model can use these data for training and optimization. Since the training process of the reinforcement learning decision model is carried out in a cyclic iterative manner, the so-called current replay buffer refers to the replay buffer corresponding to the current training round. The current training sample refers to the training sample to be used in the current training round.

[0046] The current replay buffer is used to store new samples and old samples required by the reinforcement learning decision model of the current training round. The current replay buffer corresponds to the current replay ratio. The current replay ratio is the ratio of the number of new samples to the number of old samples in the current replay buffer.

[0047] Before training the reinforcement learning decision model, it is necessary to pre-initialize the reinforcement learning algorithm (i.e., the reinforcement learning decision model) and the task environment, initialize the reinforcement learning hyperparameters and the initial replay ratio, and initialize the capacity of the replay buffer.

[0048] In this embodiment, the reinforcement learning algorithm should use an off-policy algorithm, which can learn from data generated by other behavioral strategies, which means that data generated by past strategies can be used, or data generated by multiple strategies can be used for learning at the same time, which provides great flexibility and gives the algorithm the advantage of being naturally suitable for reusing data, thereby improving sample efficiency. For example, a Q-learning algorithm based on deep learning (Deep Q-Network, DQN) can be used, which introduces a replay buffer and a target network to improve the stability and performance of the algorithm.

[0049] In the initial stage, the replay buffer is empty. At this time, the intelligent agent can interact with the task environment to collect training sample data and store these training sample data in the replay buffer. When the number of training samples in the replay buffer reaches the first preset threshold, the replay buffer can be considered to have completed initialization.

[0050] When the first training round is the current training round, the strategy model at this time is the initial strategy model, and the sample data in the current replay buffer (i.e., the replay buffer that has completed initialization) are all new samples relative to the current training round; the current replay buffer can be randomly sampled to determine the current training sample of a preset data volume; thus, the initial strategy model obtains new interaction data through training based on the current training samples, and adds these new interaction data to the replay buffer. Based on the new interaction data obtained by the initial strategy model, the model parameters of the initial strategy model can be optimized to obtain the first updated decision model.

[0051] In the case where the second training round is the current training round, the policy model at this time is the first updated decision model, and the sample data in the current replay buffer includes old samples (i.e., those samples when the replay buffer was initialized) and new samples (i.e., those new interaction data newly added to the replay buffer during the first training round). The ratio of new samples to old samples is the current replay ratio; the new samples and old samples in the current replay buffer can be randomly sampled to determine the current training samples of the reinforcement learning decision model in the current training round. Based on the same method, by analogy, the current training samples corresponding to the third training round, ..., and the current training samples corresponding to the Nth training round can be determined.

[0052] S120: Train the reinforcement learning decision model based on the current training sample to determine at least one current performance feedback parameter of the reinforcement learning decision model in the current training round.

[0053] Among them, the performance feedback parameters are used to characterize the model performance of the reinforcement learning decision model after the current training round. That is, the performance feedback parameters are the performance evaluation parameters of the decision model. For example, the performance feedback parameters may include cumulative rewards, average cumulative rewards, completion rate of specific tasks, state value parameters, and efficiency of interaction with the environment. The current performance feedback parameters are the performance feedback parameters determined by the reinforcement learning decision model after training based on the current training samples in the current training round.

[0054] Specifically, in the current training round, after the reinforcement learning decision model is trained according to the current training sample, the current performance feedback parameters can be obtained, and the model parameters of the decision model can be optimized according to the loss function. After the optimization of the model parameters of the decision model is completed, the current training round ends. The next training round can be started, and the next training round is used as the current training round, and the steps of randomly sampling the sample data in the current replay buffer, determining the current training sample of the reinforcement learning decision model in the current training round, and training the reinforcement learning decision model based on the current training sample to determine at least one current performance feedback parameter of the reinforcement learning decision model in the current training round can be repeated.

[0055] S130. If the current training round meets the preset replay ratio adjustment conditions, determine the target replay ratio corresponding to the replay buffer based on the current performance feedback parameters, the current replay ratio and at least one pre-set replay ratio adjustment strategy, so as to perform sample update on the current replay buffer based on the target replay ratio.

[0056] The replay ratio adjustment strategy is a method for adjusting the size of the replay ratio. The replay ratio adjustment strategy includes a replay ratio adjustment strategy based on performance feedback parameters and / or a replay ratio adjustment strategy based on a balanced replay ratio.

[0057] The preset replay ratio adjustment condition is a preset rule for determining when the target replay ratio can be determined. In this embodiment, the preset replay ratio adjustment condition is: the current training round reaches the preset training round. For example, it is preset that the replay ratio is updated once after every N training rounds, that is, the preset training rounds are N, 2N, 3N, ..., etc.

[0058] The target replay ratio is a replay ratio determined according to the replay ratio adjustment strategy. After the target replay ratio is determined, the number of new samples required can be determined, and then the number of new samples can be generated, and the new samples are added to the replay buffer to achieve the purpose of updating the samples in the current replay buffer.

[0059] In this embodiment, during the iterative training of the reinforcement learning decision model, each time a new training round is entered, the current cumulative training round can be counted. If the current cumulative training round does not meet the preset replay ratio adjustment condition, there is no need to update the replay ratio, that is, there is no need to determine the target replay ratio; if the current cumulative training round meets the preset replay ratio adjustment condition, the replay ratio needs to be updated, that is, the target replay ratio needs to be determined. This step is mainly for situations where the replay ratio needs to be updated, and the target replay ratio is described in detail below.

[0060] In this embodiment, the replay ratio adjustment strategy based on the performance feedback parameter and the replay ratio adjustment strategy based on the balanced replay ratio may be used alone or in combination, which is not specifically limited herein.

[0061] Specifically, for the replay ratio adjustment strategy based on performance feedback parameters, the specific implementation method of determining the target replay ratio corresponding to the replay buffer based on the current performance feedback parameters, the current replay ratio and the replay ratio adjustment strategy based on performance feedback parameters may include: processing the current performance feedback parameters according to a first preset update function to determine the target replay ratio.

[0062] The first preset update function is a function pre-constructed according to at least one performance feedback parameter, and is used to determine the target playback ratio according to the performance feedback parameter.

[0063] In this embodiment, a first preset update function may be pre-constructed, and based on obtaining at least one current performance feedback parameter, the at least one current performance feedback parameter may be introduced into the first preset update function to obtain a target playback ratio.

[0064] Optionally, as a specific implementation, the current performance feedback parameter includes a current average cumulative reward parameter and a current state value parameter, and the first preset update function is:

[0065]

[0066] Where A1 is the target replay ratio, A0 is the current replay ratio, Amax is the maximum replay ratio threshold, Amin is the minimum replay ratio threshold, Va is the current state value parameter, Va' is the state value parameter of the previous cycle of the current training round, R is the current average cumulative reward parameter, R' is the average cumulative reward parameter of the previous cycle of the current training round, and α1 and α2 are preset weight values.

[0067] In this embodiment, the determined current state value parameter, current average cumulative reward parameter, and the previous state value parameter and previous average cumulative reward parameter determined in the previous training round can be substituted into the above formula. After calculating the formula, the target replay ratio can be obtained.

[0068] For the replay ratio adjustment strategy based on a balanced replay ratio, the specific implementation method of determining the target replay ratio corresponding to the replay buffer based on the current performance feedback parameter, the current replay ratio, and the replay ratio adjustment strategy based on the balanced replay ratio may include: determining the model evaluation value of the reinforcement learning decision model after the current training round based on at least one current performance feedback parameter; adjusting the current replay ratio based on the model evaluation value and the preset model evaluation threshold to determine the target replay ratio corresponding to the replay buffer.

[0069] The model evaluation value is used to characterize the performance of the decision model after the current training round. The preset model evaluation threshold is a pre-set fixed threshold used to judge the performance of the decision model after the current training round.

[0070] Specifically, the current performance feedback parameter may include: at least one of: a current cumulative reward parameter, a preset task completion rate parameter, and a current environment interaction efficiency parameter. Based on this, based on at least one current performance feedback parameter, a specific implementation method for determining the model evaluation value of the reinforcement learning decision model after the current training round may include: determining the model evaluation value of the reinforcement learning decision model after the current training round based on the weighted average sum of the current cumulative reward parameter, the preset task completion rate parameter, and the current environment interaction efficiency parameter.

[0071] In this embodiment, after obtaining at least one current performance feedback parameter, a weighted sum operation may be performed on each current performance feedback parameter based on a preset weight coefficient to obtain a model evaluation value.

[0072] Furthermore, a temporary qualitative judgment can be made on the model of the reinforcement learning decision model after the current training round based on the model evaluation value and the preset model evaluation threshold, and then different replay ratio adjustment methods can be adopted according to different judgment results to obtain the target replay ratio corresponding to the replay buffer.

[0073] Optionally, as a specific implementation method, the current replay ratio is adjusted based on the model evaluation value and the preset model evaluation threshold to determine the target replay ratio corresponding to the replay buffer, including: if the model evaluation value is greater than the preset model evaluation threshold, the current replay ratio is decreased based on the preset adjustment step to obtain the target replay ratio; if the model evaluation value is less than or equal to the preset model evaluation threshold, the current replay ratio is increased based on the preset adjustment step to obtain the target replay ratio.

[0074] The preset adjustment step size is used to represent the magnitude of adjusting the playback ratio.

[0075] In this embodiment, if the model evaluation value is greater than the preset model evaluation threshold, it indicates that the model after the current training round is temporarily characterized as having good model performance, indicating that the sample training results in the replay buffer are good. In order to stabilize the training effect, the number of new samples in the current replay buffer can be reduced. Based on this, the current replay ratio can be reduced and adjusted according to the preset adjustment step size to obtain the target replay ratio. If the model evaluation value is less than or equal to the preset model evaluation threshold, it indicates that the model after the current training round is temporarily characterized as having poor model performance, indicating that the sample training results in the replay buffer are not good and more new data exploration is needed. The number of new samples in the current replay buffer can be increased. Based on this, the current replay ratio can be increased and adjusted according to the preset adjustment step size to obtain the target replay ratio.

[0076] It should be noted that before the sample data in the current replay buffer is updated, all sample data in the current replay buffer is used as old data for the current training round. Therefore, the amount of old data is fixed. Therefore, the replay ratio needs to be adjusted by adjusting the number of new samples added to the current replay buffer.

[0077] In this embodiment, after obtaining the target replay ratio, based on the target replay ratio and the current replay ratio, it can be determined how many new samples need to be added to the current replay buffer so that the replay ratio of the replay buffer can be changed from the current replay ratio to the target replay ratio. On this basis, a certain number of new samples can be collected and added to the replay buffer, thereby completing the operation of updating the samples of the current replay buffer based on the target replay ratio.

[0078] After step S130, steps S110-S130 are repeatedly executed in a loop until a training end condition of the reinforcement learning decision model is reached. For example, the training end condition may include that the current training round reaches a preset maximum training round or the learning performance reaches a preset threshold.

[0079] The technical solution of the embodiment of the present invention determines the current training sample of the reinforcement learning decision model in the current training round by randomly sampling the sample data in the current replay buffer, wherein the current replay buffer corresponds to the current replay ratio, and the current replay ratio is the ratio of the number of new samples to the number of old samples, so that the reinforcement learning decision model is trained based on the current training sample to determine at least one current performance feedback parameter of the reinforcement learning decision model under the current training round, and further, if the current training round meets the preset replay ratio adjustment condition, then based on the current performance feedback parameter, the current replay ratio and at least one pre-set replay ratio adjustment strategy, the target replay ratio corresponding to the replay buffer is determined to update the sample of the current replay buffer based on the target replay ratio. The technical solution provided by the embodiment of the present invention improves the learning efficiency and final strategy performance of the reinforcement learning decision model algorithm by dynamically adjusting the replay ratio to balance the speed of writing fresh experience into the replay buffer.

[0080] Embodiment 2

[0081] Figure 3 It is a flowchart of a replay ratio adjustment method applied to a reinforcement learning decision model provided in the second embodiment of the present invention. On the basis of the above embodiment, S130 is further refined, wherein the technical terms that are the same as or corresponding to the above embodiment are not repeated here.

[0082] like Figure 3 As shown, the method includes:

[0083] S210: Randomly sample the sample data in the current replay buffer to determine the current training sample of the reinforcement learning decision model in the current training round.

[0084] S220: Train the reinforcement learning decision model based on the current training sample to determine at least one current performance feedback parameter of the reinforcement learning decision model in the current training round.

[0085] S230: If the current training round meets the preset replay ratio adjustment condition, determine a target replay ratio corresponding to the replay buffer based on the current performance feedback parameter, the current replay ratio and at least one preset replay ratio adjustment strategy.

[0086] In this embodiment, the replay ratio adjustment strategy also includes a replay ratio adjustment strategy based on a decision model learning rate.

[0087] Optionally, the replay ratio adjustment strategy based on the decision model learning rate includes: when the current learning rate of the reinforcement learning decision model in the current training round changes, based on the linkage mapping relationship between the change rate and the replay ratio, determining the target replay ratio corresponding to the replay buffer.

[0088] Among them, the learning rate refers to a key hyperparameter when training a neural network, which controls the learning speed of the model during the training process. The learning rate determines the step size of the model parameter update, that is, how the model adjusts its weights according to the gradient of the loss function in each iteration. This parameter has a crucial impact on the efficiency of the training process and the performance of the final model. The linkage mapping relationship is used to characterize the degree of change of the replay ratio when the rate of change changes. The specific representation form of the linkage mapping relationship can be a function.

[0089] The learning rate changes during model training. During the entire model training process, the learning rate is not fixed, but can be adjusted according to the needs of training. This adjustment can occur at any stage of training, especially in the later stages of training, when the model is close to the ideal effect, it is particularly important to adjust the learning rate to prevent the model from stagnating at the local optimal solution. The adjustment of the learning rate can be achieved through a variety of strategies. For example, when the performance of the model on the validation set stops improving, the learning rate can be appropriately reduced; when the performance of the model on the validation set improves too quickly, the learning rate can be appropriately increased. This strategy of dynamically adjusting the learning rate helps the model better converge to the optimal solution, improves the generalization ability of the model, and enhances the stability of the model.

[0090] In this embodiment, a preset function expression that characterizes the linkage mapping relationship between the change rate and the replay ratio can be determined in advance. Whenever the learning change rate is adjusted, that is, when the current learning rate of the reinforcement learning decision model in the current training round changes, the target replay ratio corresponding to the replay buffer can be determined based on the change in the learning rate and the preset function expression.

[0091] S240: Update samples of the current replay buffer based on the target replay ratio, the current replay buffer, and the reinforcement learning decision model determined in the previous training round.

[0092] In this embodiment, based on the target replay ratio and the current replay ratio of the current replay buffer, it can be determined how many new samples need to be added to the current replay buffer in order to change the replay ratio of the replay buffer from the current replay ratio to the target replay ratio. On this basis, a certain number of old samples can be randomly determined from the current replay buffer, and these old samples are processed by the reinforcement learning decision model determined by the previous training round to obtain a certain number of new samples, which are added to the replay buffer, thereby completing the operation of updating the samples of the current replay buffer based on the target replay ratio.

[0093] Optionally, as a specific implementation method, based on the target replay ratio, the current replay buffer, and the reinforcement learning decision model determined by the previous training round, the specific implementation method of updating the samples in the current replay buffer may include: determining the second number of new samples to be explored based on the first number of existing sample data in the current replay buffer and the target replay ratio; determining the second number of target exploration new samples based on the second number of existing sample data and the reinforcement learning decision model determined by the previous training round, and adding the target exploration new samples to the current replay buffer.

[0094] In this embodiment, based on the first number of existing sample data in the current replay buffer and the target replay ratio, it can be determined how many new samples are needed, that is, the second number of new samples to be explored can be determined; then, the second number of existing sample data is input into the reinforcement learning decision model determined by the previous training round, and the model can output the second number of target exploration new samples; thus, the second number of target exploration new samples are added to the current replay buffer, thereby completing the operation of updating samples in the current replay buffer based on the target replay ratio.

[0095] The technical solution of the embodiment of the present invention, the replay ratio adjustment strategy also includes a replay ratio adjustment strategy based on the decision model learning rate, and the replay ratio adjustment strategy based on the decision model learning rate includes: when the current learning rate of the reinforcement learning decision model in the current training round changes, based on the linkage mapping relationship between the change rate and the replay ratio, determine the target replay ratio corresponding to the replay buffer. In addition, the current replay buffer can be sampled and updated based on the target replay ratio, the current replay buffer, and the reinforcement learning decision model determined by the previous training round. The technical solution provided by the embodiment of the present invention improves the learning efficiency and final strategy performance of the reinforcement learning decision model algorithm by dynamically adjusting the replay ratio to balance the speed of writing fresh experience into the replay buffer.

[0096] Embodiment 3

[0097] Next, a specific example is used to illustrate the specific implementation process of the replay ratio adjustment method applied to the reinforcement learning decision model.

[0098] For ease of explanation, the following is an example of the multi-agent reinforcement learning QMIX algorithm in the 3s_vs_5z environment in the StarCraft environment. The QMIX algorithm is both a different strategy (the target strategy and the behavior strategy can be different) algorithm and a multi-agent algorithm. It is based on the DQN algorithm. The core is that it can decompose the decision-making problems of multiple agents into local decisions and ensure global consistency through a hybrid network. It is suitable for solving complex multi-agent collaborative task environments such as StarCraft. The 3s_vs_5z environment can be understood as: the three hunter units that need to be controlled face a disadvantage of five fanatics in terms of quantity. At the same time, the hunters and fanatics have different capabilities and attack ranges. The goal is to win the battle. This requires the agent to learn how to combine the characteristics of different units to overcome the disadvantage in quantity, which requires more complex coordination and tactical strategies, and is difficult. In this scenario, the specific implementation process of the present invention is as follows:

[0099] Figure 4 This is a general flow chart of a replay ratio adjustment method applied to a reinforcement learning decision model provided according to the third embodiment of the present invention, which mainly includes four steps: reinforcement learning initialization, reinforcement learning process, updating replay ratio, and iterative loop termination judgment. Among them, the initialization process step is to prepare for the subsequent steps; the reinforcement learning process step mainly performs standard reinforcement learning to collect data and model training; the replay ratio update process dynamically and adaptively updates the replay ratio according to a variety of update strategies; the iterative loop termination judgment step is used to promote the iterative loop of the entire process, that is, the termination judgment.

[0100] The first step is to initialize reinforcement learning

[0101] (1) Initialize a shared sub-agent Q network with a value network model Q(o, a, θ) (the Q network is a reinforcement learning decision model), where o is the local observation state of the sub-agent, a is the action of the sub-agent, and θ is the sub-agent network model parameter;

[0102] (2) Initialize a mixed strategy network model (The hybrid strategy network model is also a reinforcement learning decision model) where q is the output of the sub-agent network Q(o,a,θ) corresponding to the sub-action a, s is the global state of the task environment, are the hybrid network model parameters;

[0103] (3) Initialize a replay buffer D (replaybuffer);

[0104] (4) Initialize and set the maximum replay ratio threshold and the minimum replay ratio threshold;

[0105] (5) Initialize the starting state s0 of the 3s_vs_5z environment and other reinforcement learning hyperparameters, including the number of agents, experience pool capacity, batch size, discount factor γ, soft update target network coefficient τ, learning rate (Q network), learning rate (hybrid network), ε-greedy exploration initial value, ε-greedy exploration decay rate, and optimizer;

[0106] (6) Setting an adaptive replay ratio update function. This embodiment takes the replay ratio adjustment strategy based on performance feedback parameters as an example for explanation.

[0107] Step 2: Strengthen the learning process

[0108] In this step, the reinforcement learning agent interacts with the environment, collects new experience data, stores it in the replay buffer, and samples data from the replay buffer for learning based on the current replay ratio. Specifically, the following steps may be included:

[0109] (1) Interact with the environment according to the current strategy Q(o, a, θ), and each sub-agent performs the corresponding sub-agent action i is used to identify the sub-agent, obtain the global reward r0, and the new local state observation of all sub-agents and new global state s1;

[0110] (2) Store the data tuple (o0, s0, a0, r0, o1, s1) into the replay buffer D;

[0111] (3) Repeat steps (1) and (2) until the data requirements for training are met, and randomly sample data from the experience replay buffer D to obtain the current training sample;

[0112] (4) Use the current training samples for training. The training process is described in detail below.

[0113] S1. Calculate the sub-agent Q network loss function

[0114] Using the mean square error of TD-error, the calculation formula for the sub-agent Q network loss function is as follows:

[0115]

[0116] in, is the target Q value, calculated as follows:

[0117] Assume that the current state is s and the currently selected joint action is a=(a 1 ,…,an ), and reaches the next state s', then:

[0118] in, is the Q value network corresponding to the target network.

[0119] S2. The formula for calculating the hybrid network loss function is:

[0120] L=(Q tot (s,a;θ))-y tot ) 2

[0121] Among them, y tot is the maximum global Q value obtained by optimal behavior selection, which is calculated as:

[0122]

[0123] Among them, r tot is the total reward obtained by the entire environment for performing the joint action a in this state, is the hybrid network corresponding to the target hybrid network.

[0124] S3, optimize the network in two stages

[0125] The first stage is to fix the hybrid network parameters, update the network parameters of each sub-agent Q, and minimize the above Li loss function; the second stage is to fix the sub-agent Q network parameters, update the hybrid network parameters, and minimize the L loss function. Use a stochastic gradient descent optimization algorithm (such as RMSProp) for multi-step updates.

[0126] In addition, at regular training step lengths, the parameters of the sub-agent Q network and the hybrid network are soft-updated to the corresponding target network at a ratio of τ to improve training stability.

[0127] The third step is to update the replay ratio, which specifically includes:

[0128] (1) Sample a batch of states from the replay buffer D and calculate the overall action value Va of the current environment based on the current sub-agent strategy Q and the hybrid network;

[0129] (2) Calculate performance indicators, including the average cumulative reward R', global action state value Va' of the previous iteration training, and the average cumulative reward R and state value Va of this iteration training, where R, R', Va, and Va' are all normalized. If the performance indicators are significantly improved compared to the previous iteration, the current learning strategy can be considered effective. If the performance improves, increase the replay ratio, that is, reduce the amount of fresh data written to the replay buffer to stabilize learning; if the performance decreases or stagnates, reduce the replay ratio to introduce more fresh data from the environment to try to solve possible overfitting or insufficient exploration problems.

[0130] (3) Determine the target playback ratio according to the first preset update function and the performance index calculated in (2), where the first preset update function is:

[0131]

[0132] Where A1 is the target replay ratio, A0 is the current replay ratio, Amax is the maximum replay ratio threshold, Amin is the minimum replay ratio threshold, Va is the current state value parameter, Va' is the state value parameter of the previous cycle of the current training round, R is the current average cumulative reward parameter, R' is the average cumulative reward parameter of the previous cycle of the current training round, α1 and α2 are preset weight values ​​used to control the sensitivity to the state value and average cumulative reward during the adaptive update process.

[0133] Step 4: Iteration loop termination

[0134] (1) Repeat the second and third steps;

[0135] (2) When the average cumulative reward reaches the termination condition, the training is stopped and the decision model is obtained.

[0136] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0137] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A replay ratio adjustment method applied to a reinforcement learning decision model, characterized in that: include: Randomly sampling the sample data in the current replay buffer to determine the current training sample of the reinforcement learning decision model in the current training round; wherein the current replay buffer is used to store new samples and / or old samples required by the reinforcement learning decision model in the current training round, and the current replay buffer corresponds to a current replay ratio, which is the ratio of the number of the new samples to the number of the old samples; Training the reinforcement learning decision model based on the current training sample to determine at least one current performance feedback parameter of the reinforcement learning decision model in the current training round; If the current training round meets the preset replay ratio adjustment conditions, the target replay ratio corresponding to the replay buffer is determined based on the current performance feedback parameters, the current replay ratio and at least one pre-set replay ratio adjustment strategy, so as to perform sample update on the current replay buffer based on the target replay ratio; wherein the replay ratio adjustment strategy includes a replay ratio adjustment strategy based on performance feedback parameters and / or a replay ratio adjustment strategy based on a balanced replay ratio.

2. The method according to claim 1, characterized in that The replay ratio adjustment strategy is a replay ratio adjustment strategy based on a performance feedback parameter, and determining a target replay ratio corresponding to the replay buffer based on the current performance feedback parameter, the current replay ratio, and at least one preset replay ratio adjustment strategy includes: The current performance feedback parameter is processed according to a first preset update function to determine a target playback ratio.

3. The method according to claim 2, characterized in that The current performance feedback parameters include the current average cumulative reward parameter and the current state value parameter, and the first preset update function is: Wherein, A1 is the target replay ratio, A0 is the current replay ratio, Amax is the maximum replay ratio threshold, Amin is the minimum replay ratio threshold, Va is the current state value parameter, Va′ is the state value parameter of the previous cycle of the current training round, R is the current average cumulative reward parameter, R′ is the average cumulative reward parameter of the previous cycle of the current training round, and α1 and α2 are preset weight values.

4. The method according to claim 1, characterized in that: The replay ratio adjustment strategy is a replay ratio adjustment strategy based on a balanced replay ratio, and determining a target replay ratio corresponding to the replay buffer based on the current performance feedback parameter, the current replay ratio, and at least one preset replay ratio adjustment strategy includes: Determining a model evaluation value of the reinforcement learning decision model after a current training round based on the at least one current performance feedback parameter; Based on the model evaluation value and a preset model evaluation threshold, the current replay ratio is adjusted to determine a target replay ratio corresponding to the replay buffer.

5. The method according to claim 4, characterized in that The at least one current performance feedback parameter includes a current cumulative reward parameter, a preset task completion rate parameter, and a current environment interaction efficiency parameter. The determining, based on the at least one current performance feedback parameter, a model evaluation value of the reinforcement learning decision model after the current training round includes: Based on the weighted average of the current cumulative reward parameter, the preset task completion rate parameter and the current environment interaction efficiency parameter, the model evaluation value of the reinforcement learning decision model after the current training round is determined.

6. The method according to claim 4, characterized in that The adjusting the current replay ratio based on the model evaluation value and the preset model evaluation threshold to determine the target replay ratio corresponding to the replay buffer includes: If the model evaluation value is greater than a preset model evaluation threshold, the current playback ratio is reduced and adjusted based on a preset adjustment step to obtain a target playback ratio; If the model evaluation value is less than or equal to a preset model evaluation threshold, the current replay ratio is increased and adjusted based on a preset adjustment step to obtain a target replay ratio.

7. The method according to claim 1, characterized in that The replay ratio adjustment strategy also includes a replay ratio adjustment strategy based on a decision model learning rate.

8. The method according to claim 7, characterized in that The replay ratio adjustment strategy based on the decision model learning rate includes: when the current learning rate of the reinforcement learning decision model in the current training round changes, based on the linkage mapping relationship between the change rate and the replay ratio, determining the target replay ratio corresponding to the replay buffer.

9. The method according to claim 1, characterized in that: The updating of samples of the current replay buffer based on the target replay ratio includes: Based on the target replay ratio, the current replay buffer, and the reinforcement learning decision model determined in the previous training round, the current replay buffer is sampled and updated.

10. The method according to claim 9, characterized in that The step of updating samples of the current replay buffer based on the target replay ratio and the reinforcement learning decision model determined in the previous training round includes: Determining a second number of new samples to be explored based on a first number of sample data already in the current replay buffer and the target replay ratio; According to the second quantity of the existing sample data and the reinforcement learning decision model determined in the previous training round, a second quantity of target exploration new samples is determined, and the target exploration new samples are added to the current replay buffer.

Citation Information

Cited By

  • Fan speed regulation method and device, electronic equipment and storage medium

    CN119616911A

  • Fan speed regulation method, device, electronic equipment and storage medium

    CN119616911B