Asynchronous reinforcement learning training method and device, equipment and medium
By introducing a decoupled policy gradient loss function and a two-layer importance weighting process in asynchronous reinforcement learning training, the non-policy bias problem caused by the inconsistency between sample data and model training process is solved, thereby improving the stability and convergence accuracy of training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BAIYANG TIMES (BEIJING) TECH CO LTD
- Filing Date
- 2026-04-13
- Publication Date
- 2026-05-12
AI Technical Summary
In asynchronous reinforcement learning training, the generation of sample data is executed in parallel with the training of the model. This results in the use of sample data that is generated based on historical behavior policy models rather than current behavior policy models, which can easily lead to non-policy bias and affect the stability and convergence of model training.
By introducing a decoupled policy gradient loss function, a two-layer importance weight is constructed using the sample behavior policy probability, the near-end behavior policy probability, and the current behavior policy probability. This includes behavior importance weight and policy gradient importance weight. Upper limit constraints and pruning are applied, and backpropagation is performed in conjunction with the decoupled policy gradient loss function to update the model parameters.
It effectively handles non-policy bias in asynchronous reinforcement learning, improves training stability and convergence accuracy, and ensures the stability and effectiveness of model training.
Smart Images
Figure CN122021784A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to an asynchronous reinforcement learning training method, apparatus, device, and medium. Background Technology
[0002] With the development of artificial intelligence technology, reinforcement learning has been widely applied to tasks such as mathematical reasoning, code generation, dialogue decision-making, and automatic control. Currently, the reinforcement learning training process typically adopts an asynchronous training paradigm, that is, the inference engine continuously generates sample data, and the training engine continuously and asynchronously acquires sample data for model training.
[0003] However, in the asynchronous training paradigm described above, the generation of sample data and the training of the model are executed in parallel. This means that the sample data used by the training engine during model training is often generated by the inference engine based on historical behavior policy models, rather than based on the current behavior policy model. This can easily lead to non-policy bias, thereby affecting the stability of model training and reducing model convergence performance. Summary of the Invention
[0004] To address the aforementioned issues, this application provides an asynchronous reinforcement learning training method, apparatus, device, and medium that can effectively handle non-policy biases in asynchronous reinforcement learning and improve training stability and convergence accuracy.
[0005] The embodiments of this application disclose the following technical solutions: In a first aspect, this application discloses an asynchronous reinforcement learning training method applied to a training engine, the method comprising: Sample data is extracted from the sample data area; the sample data is generated by an inference engine that runs asynchronously with the training engine and written into the sample data area; the sample data includes sample behavior policy probabilities; the sample behavior policy probabilities are the output probabilities of the sample behavior policy model for the corresponding actions in the sample data when the inference engine generates the sample data; Based on the proximal behavior policy probability and the sample behavior policy probability, the behavior importance weight is determined, and the behavior importance weight is subjected to an upper limit constraint to obtain the processed behavior importance weight; the proximal behavior policy probability is the output probability of the proximal behavior policy model in the training engine for the corresponding action in the sample data; Based on the current behavior policy probability and the proximal behavior policy probability, the policy gradient importance weight is determined, and the policy gradient importance weight is pruned to obtain the processed policy gradient importance weight; the current behavior policy probability is the output probability of the current behavior policy model in the training engine for the corresponding action in the sample data; The loss value is determined by decoupling the policy gradient loss function based on the processed behavior importance weight, the policy gradient importance weight, the advantage function corresponding to the sample data; Backpropagation is performed based on the loss value to update the model parameters of the current behavior strategy model.
[0006] Optionally, determining the behavior importance weight based on the proximal behavior policy probability and the sample behavior policy probability includes: The importance weight of a behavior is determined based on the ratio of the probabilities of near-end behavior strategies to the probabilities of the sample behavior strategies.
[0007] Optionally, the step of applying an upper limit constraint to the importance weights of the behaviors to obtain the processed importance weights of the behaviors includes: The smaller value between the behavior importance weight and the preset upper limit threshold is determined as the processed behavior importance weight.
[0008] Optionally, determining the policy gradient importance weight based on the current behavior policy probability and the near-end behavior policy probability includes: The importance weight of the policy gradient is determined based on the ratio of the current policy probability to the probabilities of the near-end policy.
[0009] Optionally, the step of pruning the policy gradient importance weights to obtain processed policy gradient importance weights includes: The processed policy gradient importance weights are determined by limiting the policy gradient importance weights to a preset clipping range.
[0010] Optionally, the sample data further includes version information of the sample behavior policy model; determining the behavior importance weight based on the proximal behavior policy probability and the sample behavior policy probability includes: The expiration level of the sample data is determined based on the version information of the sample behavior strategy model and the version information of the current behavior strategy model. If the expiration degree is less than or equal to the preset expiration degree threshold, then the importance weight of the behavior is determined based on the probabilities of the near-end behavior strategy and the probabilities of the sample behavior strategy.
[0011] Optionally, the sample data includes multiple sub-data generated from different versions of the behavior policy model; the step of determining the loss value by decoupling the policy gradient loss function includes: The loss values of the multiple sub-data are determined by the gradient loss function of the decoupling strategy, and the loss values of the multiple sub-data are aggregated to obtain the loss value of the sample data.
[0012] Secondly, this application provides an asynchronous reinforcement learning training device for use in a training engine. The device includes: a data extraction module, a first determination module, a second determination module, a third determination module, and a model update module. The data extraction module is used to extract sample data from the sample data area; the sample data is generated by an inference engine that runs asynchronously with the training engine and written into the sample data area; the sample data includes sample behavior policy probabilities; the sample behavior policy probabilities are the output probabilities of the sample behavior policy model for the corresponding actions in the sample data when the inference engine generates the sample data; The first determining module is used to determine the behavior importance weight based on the proximal behavior policy probability and the sample behavior policy probability, and to perform upper limit constraint processing on the behavior importance weight to obtain the processed behavior importance weight; the proximal behavior policy probability is the output probability of the proximal behavior policy model in the training engine for the corresponding action in the sample data; The second determining module is used to determine the policy gradient importance weight based on the current behavior policy probability and the proximal behavior policy probability, and to perform a pruning process on the policy gradient importance weight to obtain the processed policy gradient importance weight; the current behavior policy probability is the output probability of the current behavior policy model in the training engine for the corresponding action in the sample data; The third determining module is used to determine the loss value by decoupling the policy gradient loss function based on the processed behavior importance weight, the policy gradient importance weight, the processed policy gradient importance weight, and the advantage function corresponding to the sample data. The model update module is used to perform backpropagation based on the loss value and update the model parameters of the current behavior policy model.
[0013] Thirdly, this application discloses an asynchronous reinforcement learning training device, the device comprising: a memory and a processor; The memory is used to store programs; The processor is configured to execute the program to implement the various steps of the asynchronous reinforcement learning training method as described in the first aspect.
[0014] Fourthly, this application discloses a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the various steps of the asynchronous reinforcement learning training method as described in the first aspect.
[0015] Compared with the prior art, this application has the following beneficial effects: This application provides an asynchronous reinforcement learning training method, apparatus, device, and medium. The method is applied to a training engine and includes: extracting sample data from a sample data area; the sample data is generated by an inference engine running asynchronously with the training engine and written into the sample data area; the sample data includes sample behavior policy probabilities; the sample behavior policy probabilities are the output probabilities of the sample behavior policy model for corresponding actions in the sample data when the inference engine generates the sample data; determining behavior importance weights based on the proximal behavior policy probabilities and the sample behavior policy probabilities, and applying an upper limit constraint to the behavior importance weights to obtain the processed behavior importance weights; the proximal behavior policy probabilities are the output probabilities of the training engine... The near-end behavior policy model outputs the probability of the corresponding action in the sample data. Based on the current behavior policy probability and the near-end behavior policy probability, policy gradient importance weights are determined and pruned to obtain processed policy gradient importance weights. The current behavior policy probability is the output probability of the current behavior policy model in the training engine for the corresponding action in the sample data. Based on the processed behavior importance weights, policy gradient importance weights, and the advantage function corresponding to the sample data, a loss value is determined by decoupling the policy gradient loss function. Backpropagation is performed based on the loss value to update the model parameters of the current behavior policy model. Therefore, this application introduces a decoupled policy gradient loss function during training and constructs a two-layer importance weight using the sample behavior policy probability, the near-end behavior policy probability, and the current behavior policy probability: on the one hand, the behavior importance weight corrects the distribution deviation between asynchronous samples and the current policy; on the other hand, the policy gradient importance weight and its pruning mechanism limit the magnitude of a single update, thereby effectively handling non-policy bias in asynchronous reinforcement learning and improving training stability and convergence accuracy. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 A flowchart of an asynchronous reinforcement learning training method provided in this application embodiment; Figure 2 A flowchart illustrating another asynchronous reinforcement learning training method provided in this application embodiment; Figure 3 A flowchart for generating sample data is provided as an embodiment of this application; Figure 4This is a flowchart illustrating how to write sample data to an embodiment of this application. Figure 5 A flowchart for calculating a loss value is provided as an embodiment of this application; Figure 6 Another flowchart for calculating the loss value provided in this application embodiment; Figure 7 A flowchart of a model adjustment provided for an embodiment of this application; Figure 8 A schematic diagram of an asynchronous reinforcement learning training framework provided in an embodiment of this application; Figure 9 This is a schematic diagram of an asynchronous reinforcement learning training device provided in an embodiment of this application. Detailed Implementation
[0018] As described earlier, in the asynchronous training paradigm, the generation of sample data and the training of the model are executed in parallel. This means that the sample data used by the training engine during model training is often generated by the inference engine based on historical behavior policy models, rather than based on the current behavior policy model. This can easily lead to non-policy bias, thereby affecting the stability of model training and reducing model convergence performance.
[0019] Through research, the inventors have proposed an asynchronous reinforcement learning training method, device, equipment, and medium. This application introduces a decoupled policy gradient loss function during the training process and constructs a two-layer importance weight using sample behavior policy probability, near-end behavior policy probability, and current behavior policy probability. On the one hand, the behavior importance weight corrects the distribution deviation between asynchronous samples and the current policy. On the other hand, the policy gradient importance weight and its pruning mechanism limit the magnitude of a single update, thereby effectively handling non-policy bias in asynchronous reinforcement learning and improving training stability and convergence accuracy.
[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.
[0021] See Figure 1 This figure is a flowchart of an asynchronous reinforcement learning training method provided in an embodiment of this application. The method is applied to a training engine and includes: S101: Extract sample data from the sample data area; the sample data is generated by the inference engine running asynchronously with the training engine and written into the sample data area; the sample data includes sample behavior policy probabilities; the sample behavior policy probabilities are the output probabilities of the sample behavior policy model for the corresponding actions in the sample data when the inference engine generates the sample data.
[0022] In the asynchronous reinforcement learning training method provided in this application embodiment, the inference engine and the training engine operate independently and asynchronously. The inference engine generates sample data and writes it into a sample data area. The training engine extracts sample data from the sample data area for model training. For example, the inference engine and the training engine can be deployed on different computing resources, such as deploying the inference engine on an inference card and the training engine on a training card. This improves the parallelism between the sample data generation process and the model training process.
[0023] The sample data includes states, actions, rewards, and sample behavior policy probabilities π_behave. The sample behavior policy probability π_behave is the output probability of the sample behavior policy model for the corresponding action (or generated lexical unit) in the sample data when the inference engine generates the sample data.
[0024] S102: Determine the importance weight of the behavior based on the proximal behavior policy probability and the sample behavior policy probability, and apply an upper limit constraint to the behavior importance weight to obtain the processed behavior importance weight; the proximal behavior policy probability is the output probability of the proximal behavior policy model in the training engine for the corresponding action in the sample data.
[0025] First, the training engine obtains the sample behavior policy probability π_behave and the proximal behavior policy probability π_proximal. The proximal behavior policy probability π_proximal is the output probability of the proximal behavior policy model in the training engine for the corresponding action in the sample data. Then, the training engine determines the behavior importance weight w_behave based on the ratio of the proximal behavior policy probability π_proximal to the sample behavior policy probability π_behave. Next, to avoid training instability caused by an excessively large behavior importance weight w_behave, the training engine applies an upper limit constraint to the behavior importance weight w_behave. Specifically, the smaller value between the behavior importance weight w_behave and a preset upper limit threshold is determined as the processed behavior importance weight w_behave'. For example, the preset upper limit threshold can be 5, and the processed behavior importance weight w_behave' can be represented by the following formula (1): w_behave'=min(w_behave,5)(1) Therefore, step S102 can correct the distribution difference between sample behavior strategies and proximal behavior strategies, and prevent weight explosion through upper limit constraints, thereby reducing the impact of extreme weights on training stability.
[0026] S103: Determine the policy gradient importance weights based on the current behavior policy probability and the proximal behavior policy probability, and prune the policy gradient importance weights to obtain the processed policy gradient importance weights; the current behavior policy probability is the output probability of the current behavior policy model in the training engine for the corresponding action in the sample data.
[0027] First, the training engine obtains the probabilities of the near-end action policy π_proximal and the current action policy π_θ. The current action policy probability π_θ is the output probability of the current action policy model in the training engine for the corresponding action in the sample data. Then, the training engine determines the policy gradient importance weight w_ppo based on the ratio of the current action policy probability π_θ to the near-end action policy probability π_proximal. Next, to limit the update magnitude of the current action policy relative to the near-end action policy, the training engine performs a pruning process on the policy gradient importance weight w_ppo. Specifically, by limiting the policy gradient importance weight w_ppo to a preset pruning interval, the processed policy gradient importance weight w_ppo' is determined. For example, the preset pruning interval can be 0.95-1.05, and the processed policy gradient importance weight w_ppo' can be represented by the following formula (2): w_ppo'=clip(w_ppo,0.95,1.05)(2) Therefore, step S103 can limit the magnitude of a single update of the current behavior policy relative to the proximal behavior policy, preventing the policy from collapsing due to excessively large single updates, thereby improving the stability of the training process.
[0028] S104: Determine the loss value by decoupling the policy gradient loss function based on the processed behavior importance weights, policy gradient importance weights, and the advantage function corresponding to the sample data.
[0029] The advantage function measures the quality of an action relative to the average level of the current state. For example, the advantage function can be represented by the following formula (3): A=RV s (3) Where A is the advantage function value, R is the reward value, and V is the reward value. sThis serves as the baseline value for the state. Specifically, the reward value is determined through reward evaluation of sample data. For example, in a mathematical reasoning task, the reward value can be determined based on whether the answer is correct; for instance, a correct answer yields a reward of 1, while an incorrect answer yields a reward of 0. In a code generation task, the reward value can be determined based on whether the generated code passes test cases after execution; for instance, passing test cases yields a reward of 1, while failing test cases yields a reward of 0. In a customer service dialogue task, the reward value can be determined based on user satisfaction ratings or preset evaluation rules.
[0030] After obtaining the advantage function, the training engine determines the loss value by decoupling the policy gradient loss function based on the processed behavior importance weights, policy gradient importance weights, and the advantage function corresponding to the sample data. For example, the decoupling policy gradient loss function L can be represented by the following formula (4): L=-w_behave'·min(w_ppo·A,w_ppo'·A)(4) Here, w_ppo·A represents the update direction and intensity of the current behavior policy relative to the proximal behavior policy, while w_ppo'·A is used to restrict the change in the current behavior policy when the magnitude of change is too large. Furthermore, by introducing processed behavior importance weights, the distribution deviation between the asynchronous proximal behavior policy and the sample behavior policy can be further corrected.
[0031] Therefore, step S104, by decoupling the policy gradient loss function, can simultaneously take into account sample distribution correction and policy update stability in asynchronous training scenarios.
[0032] S105: Perform backpropagation based on the loss value and update the model parameters of the current behavior policy model.
[0033] The training engine calculates gradients based on the loss value and performs backpropagation to update the model parameters of the current behavior policy model, thereby obtaining the updated behavior policy model. For example, the model parameters can be updated based on gradient descent.
[0034] In summary, the embodiments of this application provide an asynchronous reinforcement learning training method. This application introduces a decoupled policy gradient loss function during the training process and constructs a two-layer importance weight using sample behavior policy probability, near-end behavior policy probability, and current behavior policy probability. On the one hand, the behavior importance weight corrects the distribution deviation between asynchronous samples and the current policy. On the other hand, the policy gradient importance weight and its pruning mechanism limit the magnitude of a single update. This effectively handles non-policy bias in asynchronous reinforcement learning and improves training stability and convergence accuracy.
[0035] See Figure 2The figure is a flowchart of another asynchronous reinforcement learning training method provided in an embodiment of this application. The method includes: S201: The inference engine generates sample data; the sample data includes sample behavior policy probabilities; the sample behavior policy probabilities are the output probabilities of the sample behavior policy model for the corresponding actions in the sample data when the inference engine generates the sample data.
[0036] In the asynchronous reinforcement learning training method provided in this application embodiment, the inference engine and the training engine operate independently and asynchronously. The inference engine is used to generate sample data, which includes states, actions, rewards, and sample behavior policy probabilities π_behave, etc. The sample behavior policy probability π_behave is the output probability of the sample behavior policy model for the corresponding action (or generated lexical unit) in the sample data when the inference engine generates the sample data.
[0037] It's important to note that the inference engine does not continuously generate sample data indefinitely. Instead, it generates sample data only within the capacity constraints of the sample data area (i.e., it is constrained by the maximum number of samples to be processed, which is the upper limit of the number of sample data items allowed to be stored in the sample data area). See also... Figure 3 This figure is a flowchart illustrating the generation of sample data according to an embodiment of this application. If the number of sample data stored in the sample data area is greater than or equal to the maximum number of samples to be processed, the inference engine pauses the generation of sample data. If the number of sample data stored in the sample data area is less than the maximum number of samples to be processed, the inference engine continues to generate sample data and continues to execute the subsequent S202 step. The maximum number of samples to be processed can be determined based on the maximum expiration rate and the training batch size. The maximum expiration rate refers to the maximum number of version steps that sample data is allowed to lag behind the current behavior strategy, and the training batch size refers to the number of sample data used in each training iteration. For example, the maximum number of samples to be processed = (maximum expiration rate + 1) × training batch size. For instance, if the maximum expiration rate = 4 and the training batch size = 32, then the maximum number of samples to be processed = (4 + 1) × 32 = 160. This ensures that the number of sample data backlogged in the sample data area matches the time range of sample data allowed to participate in training, thereby avoiding excessive backlog of sample data.
[0038] It should also be noted that during the sample data generation process, if the version of the behavior policy model is updated, the inference engine does not need to forcibly interrupt the sample data generation process. Instead, it can continue to generate the sample data and record the version information corresponding to different sub-data in the sample data. For example, when the first part of the sample data is generated by the behavior policy model of version v1 and the second part is generated by the behavior policy model of version v2, it can record which sub-data in the sample data belongs to the behavior policy model of version v1 and which belongs to the behavior policy model of version v2. This allows for the calculation and aggregation of loss values for the sub-data corresponding to different versions during the training phase. This avoids the direct discarding of the entire long trajectory sample data due to version switching during generation, thereby improving the utilization rate of the sample data.
[0039] S202: The inference engine writes the sample data into the sample data area.
[0040] The inference engine writes the generated sample data into the sample data area for the training engine to extract and perform training later.
[0041] See Figure 4 This figure is a flowchart illustrating the writing of sample data to the sample data area according to an embodiment of this application. Before or during the writing process, the inference engine can also perform integrity statistics on the generated sample data, i.e., determine the truncation ratio no_eos_ratio of the sample data. The truncation ratio refers to the proportion of sample data that is truncated (without an end marker output) due to reaching the maximum generation length, and it is calculated as the ratio of the number of truncated sample data to the total number of generated sample data.
[0042] If the truncation ratio no_eos_ratio exceeds the preset ratio threshold no_eos_ratio_threshold (e.g., 5%), an alarm is triggered, and the maximum allowed generation length of sample data max_new_tokens is automatically increased. For example, the maximum allowed generation length max_new_tokens can be increased from 2048 to 3072 to reduce truncation caused by length limitations.
[0043] If the truncation ratio no_eos_ratio does not exceed the preset ratio threshold no_eos_ratio_threshold (e.g., 5%), then the dynamic filtering mechanism is enabled, and the filtered sample data is written to the sample data area.
[0044] It should be noted that the dynamic filtering mechanism can be used to proactively exclude abnormal sample data. Abnormal sample data can be at least one of the following: excessively long sample data, excessively short sample data, low-quality sample data, and incomplete sample data. For example, sample data longer than 3000 characters can be identified as excessively long sample data (possibly generated by the behavioral policy model getting stuck in a repetitive loop or diverging, belonging to an abnormal trajectory); sample data shorter than 100 characters can be identified as excessively short sample data (possibly generated by the behavioral policy model stopping prematurely or failing to reason effectively, lacking learning value); and sample data with reward values lower than a preset reward threshold can be identified as low-quality sample data (e.g., sample data with incorrect answers in mathematical reasoning, or sample data that failed to execute in code generation). This improves the quality of the sample data written to the sample data area, thereby reducing training noise.
[0045] S203: The training engine extracts sample data from the sample data area; the sample data also includes version information of the sample behavior policy model.
[0046] The training engine extracts sample data from the sample data region for subsequent loss calculation and model updates. In addition to sample behavior policy probabilities, the sample data may also include version information of the inference engine's sample behavior policy model.
[0047] It should be noted that when the sample data includes multiple sub-data generated by different versions of the behavioral strategy model, the version information can be refined to the sub-data level.
[0048] S204: The training engine determines the expiration level of the sample data based on the version information of the sample behavior policy model and the version information of the current behavior policy model.
[0049] The version information of the current behavior policy model refers to the version information of the current behavior policy model that the training engine is optimizing. The expiration degree of the sample data is used to characterize the degree of difference between the sample behavior policy model and the current behavior policy model. For example, the expiration degree can be calculated as the absolute difference between the version number of the sample behavior policy model and the version number of the current behavior policy model. For instance, if the version number of the sample behavior policy model is V1 and the version number of the current behavior policy model is V3, then the expiration degree is 3-1=2.
[0050] S205: If the expiration degree is less than or equal to the preset expiration degree threshold, the training engine determines the behavior importance weight based on the probabilities of the near-end behavior policy and the probabilities of the sample behavior policy, and applies an upper limit constraint to the behavior importance weight to obtain the processed behavior importance weight; the probabilities of the near-end behavior policy are the output probabilities of the near-end behavior policy model in the training engine for the corresponding actions in the sample data.
[0051] When the expiration rate exceeds the preset expiration rate threshold, the sample data is considered to be outside the acceptable time frame. In such cases, the sample data may be subject to restrictions on its use, reduced sampling priority, filtering, or discarding to reduce the impact of outdated sample data on training stability.
[0052] When the expiration rate is less than or equal to a preset expiration threshold, the sample data is considered to still be within an acceptable timeframe and can continue to participate in subsequent model training. For example, the preset expiration threshold can be 4, meaning that the sample behavior policy model is allowed to lag behind the current behavior policy model by up to 4 versions.
[0053] See Figure 5 This figure is a flowchart illustrating the calculation of a loss value according to an embodiment of this application. For sample data that meets the training conditions, the training engine performs subsequent steps to calculate the loss value: First, the training engine obtains the sample behavior policy probability π_behave and the proximal behavior policy probability π_proximal. Then, based on the ratio of the proximal behavior policy probability π_proximal to the sample behavior policy probability π_behave, the training engine determines the behavior importance weight w_behave. Next, to avoid excessively large behavior importance weight w_behave causing training instability, the training engine applies an upper limit constraint to w_behave. Specifically, the smaller of the behavior importance weight w_behave and a preset upper limit threshold is determined as the processed behavior importance weight w_behave'. This step corrects for the distribution difference between the sample behavior policy and the proximal behavior policy and prevents weight explosion through the upper limit constraint, thereby reducing the impact of extreme weights on training stability.
[0054] S206: The training engine determines the policy gradient importance weights based on the current behavior policy probability and the proximal behavior policy probability, and then prunes the policy gradient importance weights to obtain the processed policy gradient importance weights; the current behavior policy probability is the output probability of the current behavior policy model in the training engine for the corresponding action in the sample data.
[0055] First, the training engine obtains the probabilities of the near-end behavior policy (π_proximal) and the current behavior policy (π_θ). Then, based on the ratio of the current behavior policy probability (π_θ) to the near-end behavior policy probability (π_proximal), the training engine determines the policy gradient importance weight (w_ppo). Next, to limit the update magnitude of the current behavior policy relative to the near-end behavior policy, the training engine prunes the policy gradient importance weight (w_ppo). Specifically, by restricting the policy gradient importance weight (w_ppo) within a preset pruning interval, the processed policy gradient importance weight (w_ppo') is determined. This step limits the single update magnitude of the current behavior policy relative to the near-end behavior policy, preventing excessively large single updates from causing policy collapse, thereby improving the stability of the training process.
[0056] S207: The training engine determines the loss value by decoupling the policy gradient loss function based on the processed behavior importance weights, policy gradient importance weights, and the advantage function corresponding to the sample data.
[0057] The advantage function measures how well an action performs relative to the average level of the current state. After obtaining the advantage function, the training engine determines the loss value by decoupling the policy gradient loss function, based on the processed action importance weights, the processed policy gradient importance weights, and the advantage function corresponding to the sample data.
[0058] See Figure 6 This figure is a flowchart illustrating another method for calculating the loss value provided in an embodiment of this application. When the sample data includes multiple sub-data generated by different versions of the behavior policy model, the loss value of each sub-data can be determined by decoupling the policy gradient loss function, and the loss values of the multiple sub-data can be aggregated (e.g., summed, averaged, or weighted) to obtain the loss value of the sample data.
[0059] For example, when the first half of a long trajectory sample data is generated by the behavior policy model of version v1 and the second half by the behavior policy model of version v2, the loss value can first be calculated for the first half of the sub-data using the sample behavior policy probability corresponding to version v1, and the loss value can be calculated for the second half of the sub-data using the sample behavior policy probability corresponding to version v2. Then, the two loss values are aggregated to obtain the loss value of the long trajectory sample data. This allows for the continued use of long trajectory sample data across versions without interrupting or discarding the sample data during version switching.
[0060] S208: The training engine performs backpropagation based on the loss value and updates the model parameters of the current behavior policy model.
[0061] S209: The training engine monitors model statistics, which include at least one of the following: mean importance weights, mean behavior weights, and weight variance.
[0062] The training engine monitors model statistics related to training stability during or after model parameter updates. These statistics can include at least one of the following: mean importance weights, mean behavior weights, and weight variance. The mean importance weights refer to the arithmetic mean of the policy gradient importance weights within the current monitoring window; ideally, the mean importance weights should be close to 1. The mean behavior weights refer to the arithmetic mean of the behavior importance weights within the current monitoring window; ideally, the mean behavior weights should also be close to 1. Weight variance refers to the dispersion of the policy gradient importance weights or the behavior importance weights.
[0063] S210: If the difference between the model statistical index value and the corresponding reference value exceeds the preset difference threshold, the training engine will execute the adjustment strategy corresponding to the model statistical index.
[0064] See Figure 7 The figure is a flowchart of a model adjustment provided in an embodiment of this application. Ideally, the mean of importance weights and the mean of behavior weights should be close to 1, that is, the reference value should be 1.
[0065] When the difference between the model's statistical metric value and the corresponding reference value exceeds a preset difference threshold, it may affect training stability, requiring the implementation of an adjustment strategy corresponding to the model's statistical metric. When the difference between the model's statistical metric value and the corresponding reference value does not exceed the preset difference threshold, no adjustment is required.
[0066] Specifically, when the difference between the mean of importance weights and the corresponding reference value (e.g., 1) exceeds a preset difference threshold (e.g., 0.2), the current training update amplitude can be considered too large. In this case, the gradient accumulation steps can be reduced, for example, from 4 to 2, thereby reducing the intensity of a single parameter update and making the update smoother. When the difference between the mean of behavior weights and the corresponding reference value (e.g., 1) exceeds a preset difference threshold (e.g., 0.2), the difference between the sample data and the current behavior policy can be considered large, indicating a high degree of staleness in the sample data. In this case, the maximum expiration can be reduced, for example, from 4 to 3, thereby tightening the effective sample window and making the sample data participating in subsequent training closer to the current behavior policy. When the difference between the weight variance and the corresponding reference value exceeds a preset difference threshold, the difference between sample data can be considered large, indicating strong training fluctuations. In this case, the training batch size can be increased, for example, by increasing the training batch size by a preset factor (e.g., 1.5 times), thereby reducing the variance of gradient estimation and making training more stable.
[0067] S211: The training engine updates the version information of the behavior policy model and sends the updated version information to the inference engine so that the inference engine can generate sample data based on the updated version information.
[0068] The training engine updates the version information of the behavior policy model and sends the updated version information to the inference engine. When the inference engine receives the updated version information from the training engine, it can generate subsequent sample data based on the updated version information.
[0069] Furthermore, after the version update of the behavior policy model, a re-evaluation of the sample data area capacity can be triggered (i.e., redetermine the maximum number of samples to be processed so that the cache size of the sample data matches the current training policy).
[0070] See Figure 8 The figure is a schematic diagram of an asynchronous reinforcement learning training framework provided in an embodiment of this application. The asynchronous reinforcement learning training framework may include an inference engine, a sample data area, and a training engine. The inference engine runs asynchronously with the training engine, and the sample data area is used to cache sample data generated by the inference engine and for the training engine to retrieve.
[0071] In summary, this application provides an asynchronous reinforcement learning training method. By introducing a decoupled policy gradient loss function during training, this application constructs a two-layer importance weight using sample behavior policy probabilities, near-end behavior policy probabilities, and current behavior policy probabilities. On one hand, the behavior importance weight corrects the distribution deviation between asynchronous samples and the current policy; on the other hand, the policy gradient importance weight and its pruning mechanism limit the magnitude of a single update. This effectively handles non-policy bias in asynchronous reinforcement learning, improving training stability and convergence accuracy. Furthermore, this application also improves sample utilization and training performance in long-chain inference tasks through mechanisms such as sample expiration control, sample data area capacity management, cross-version sub-data segmented training, dynamic adjustment of long chain generation length, and abnormal sample filtering.
[0072] See Figure 9 The figure is a schematic diagram of an asynchronous reinforcement learning training device provided in an embodiment of this application. The asynchronous reinforcement learning training device 900 includes: a data extraction module 901, a first determination module 902, a second determination module 903, a third determination module 904, and a model update module 905.
[0073] The data extraction module 901 is used to extract sample data from the sample data area. The sample data is generated by the inference engine that runs asynchronously with the training engine and written into the sample data area. The sample data includes sample behavior policy probabilities. The sample behavior policy probabilities are the output probabilities of the sample behavior policy model for the corresponding actions in the sample data when the inference engine generates the sample data. The first determining module 902 is used to determine the importance weight of the behavior based on the proximal behavior policy probability and the sample behavior policy probability, and to perform upper limit constraint processing on the behavior importance weight to obtain the processed behavior importance weight; the proximal behavior policy probability is the output probability of the proximal behavior policy model in the training engine for the corresponding action in the sample data; The second determining module 903 is used to determine the policy gradient importance weights based on the current behavior policy probability and the proximal behavior policy probability, and to perform a pruning process on the policy gradient importance weights to obtain the processed policy gradient importance weights; the current behavior policy probability is the output probability of the current behavior policy model in the training engine for the corresponding action in the sample data; The third determination module 904 is used to determine the loss value by decoupling the policy gradient loss function based on the processed behavior importance weight, policy gradient importance weight, processed policy gradient importance weight and the advantage function corresponding to the sample data. The model update module 905 is used to backpropagate based on the loss value and update the model parameters of the current behavior policy model.
[0074] In one specific implementation, the first determining module 902 is specifically used to: determine the importance weight of the behavior based on the ratio of the probabilities of the near-end behavior strategy to the probabilities of the sample behavior strategy.
[0075] In one specific implementation, the first determining module 902 is specifically used to: determine the smaller value between the behavior importance weight and the preset upper limit threshold as the processed behavior importance weight.
[0076] In one specific implementation, the second determining module 903 is specifically used to: determine the importance weight of the policy gradient based on the ratio of the current behavior policy probability to the near-end behavior policy probability.
[0077] In one specific implementation, the second determining module 903 is specifically used to: determine the processed policy gradient importance weights by limiting the policy gradient importance weights to a preset clipping interval.
[0078] In one specific implementation, the sample data also includes version information of the sample behavior strategy model; the first determining module 902 is specifically used to: determine the expiration degree of the sample data based on the version information of the sample behavior strategy model and the version information of the current behavior strategy model; if the expiration degree is less than or equal to a preset expiration degree threshold, then determine the behavior importance weight based on the probabilities of near-end behavior strategies and the probabilities of sample behavior strategies.
[0079] In one specific implementation, the sample data includes multiple sub-data generated by different versions of the behavior policy model; the third determining module 904 is specifically used to: determine the loss value of the multiple sub-data by decoupling the policy gradient loss function, and aggregate the loss values of the multiple sub-data to obtain the loss value of the sample data.
[0080] In summary, this application provides an asynchronous reinforcement learning training device. By introducing a decoupled policy gradient loss function during training, it constructs a two-layer importance weight using sample behavior policy probabilities, near-end behavior policy probabilities, and current behavior policy probabilities. On one hand, the behavior importance weight corrects the distribution deviation between asynchronous samples and the current policy; on the other hand, the policy gradient importance weight and its pruning mechanism limit the magnitude of a single update. This effectively handles non-policy biases in asynchronous reinforcement learning, improving training stability and convergence accuracy. Furthermore, this application also improves sample utilization and training performance in long-chain inference tasks through mechanisms such as sample expiration control, sample data area capacity management, cross-version sub-data segmented training, dynamic adjustment of long chain generation length, and abnormal sample filtering.
[0081] This application discloses an asynchronous reinforcement learning training device, the device comprising: a memory and a processor; the memory for storing a program; the processor for executing the program to implement the various steps of the asynchronous reinforcement learning training method as described in the first aspect, thereby achieving the beneficial effects of the asynchronous reinforcement learning training method as described in the first aspect.
[0082] This application discloses a computer-readable medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the various steps of the asynchronous reinforcement learning training method as described in the first aspect, thereby achieving the beneficial effects of the asynchronous reinforcement learning training method as described in the first aspect.
[0083] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0084] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. An asynchronous reinforcement learning training method, characterized in that, Applied to a training engine, the method includes: Sample data is extracted from the sample data area; the sample data is generated by an inference engine that runs asynchronously with the training engine and written into the sample data area; the sample data includes sample behavior policy probabilities; the sample behavior policy probabilities are the output probabilities of the sample behavior policy model for the corresponding actions in the sample data when the inference engine generates the sample data; Based on the proximal behavior policy probability and the sample behavior policy probability, the behavior importance weight is determined, and the behavior importance weight is subjected to an upper limit constraint to obtain the processed behavior importance weight; the proximal behavior policy probability is the output probability of the proximal behavior policy model in the training engine for the corresponding action in the sample data; Based on the current behavior policy probability and the proximal behavior policy probability, the policy gradient importance weight is determined, and the policy gradient importance weight is pruned to obtain the processed policy gradient importance weight; the current behavior policy probability is the output probability of the current behavior policy model in the training engine for the corresponding action in the sample data; The loss value is determined by decoupling the policy gradient loss function based on the processed behavior importance weight, the policy gradient importance weight, the advantage function corresponding to the sample data; Backpropagation is performed based on the loss value to update the model parameters of the current behavior strategy model.
2. The method according to claim 1, characterized in that, The step of determining the behavior importance weight based on the probabilities of near-end behavior strategies and the probabilities of the sample behavior strategies includes: The importance weight of a behavior is determined based on the ratio of the probabilities of near-end behavior strategies to the probabilities of the sample behavior strategies.
3. The method according to claim 1, characterized in that, The step of applying an upper limit constraint to the importance weights of the behaviors to obtain the processed importance weights includes: The smaller value between the behavior importance weight and the preset upper limit threshold is determined as the processed behavior importance weight.
4. The method according to claim 1, characterized in that, The step of determining the policy gradient importance weight based on the current behavior policy probability and the near-end behavior policy probability includes: The importance weight of the policy gradient is determined based on the ratio of the current policy probability to the probabilities of the near-end policy.
5. The method according to claim 1, characterized in that, The step of pruning the policy gradient importance weights to obtain the processed policy gradient importance weights includes: The processed policy gradient importance weights are determined by limiting the policy gradient importance weights to a preset clipping range.
6. The method according to any one of claims 1-5, characterized in that, The sample data also includes version information of the sample behavior strategy model; determining the behavior importance weight based on the near-end behavior strategy probability and the sample behavior strategy probability includes: The expiration level of the sample data is determined based on the version information of the sample behavior strategy model and the version information of the current behavior strategy model. If the expiration degree is less than or equal to the preset expiration degree threshold, then the importance weight of the behavior is determined based on the probabilities of the near-end behavior strategy and the probabilities of the sample behavior strategy.
7. The method according to any one of claims 1-5, characterized in that, The sample data includes multiple sub-data generated from different versions of the behavior policy model; the determination of the loss value through decoupling the policy gradient loss function includes: The loss values of the multiple sub-data are determined by the gradient loss function of the decoupling strategy, and the loss values of the multiple sub-data are aggregated to obtain the loss value of the sample data.
8. An asynchronous reinforcement learning training device, characterized in that, The device, used in a training engine, includes: a data extraction module, a first determination module, a second determination module, a third determination module, and a model update module; The data extraction module is used to extract sample data from the sample data area; the sample data is generated by an inference engine that runs asynchronously with the training engine and written into the sample data area; the sample data includes sample behavior policy probabilities; the sample behavior policy probabilities are the output probabilities of the sample behavior policy model for the corresponding actions in the sample data when the inference engine generates the sample data; The first determining module is used to determine the behavior importance weight based on the proximal behavior policy probability and the sample behavior policy probability, and to perform upper limit constraint processing on the behavior importance weight to obtain the processed behavior importance weight; the proximal behavior policy probability is the output probability of the proximal behavior policy model in the training engine for the corresponding action in the sample data; The second determining module is used to determine the policy gradient importance weight based on the current behavior policy probability and the proximal behavior policy probability, and to perform a pruning process on the policy gradient importance weight to obtain the processed policy gradient importance weight; the current behavior policy probability is the output probability of the current behavior policy model in the training engine for the corresponding action in the sample data; The third determining module is used to determine the loss value by decoupling the policy gradient loss function based on the processed behavior importance weight, the policy gradient importance weight, the processed policy gradient importance weight, and the advantage function corresponding to the sample data. The model update module is used to perform backpropagation based on the loss value and update the model parameters of the current behavior strategy model.
9. An asynchronous reinforcement learning training device, characterized in that, The device includes: a memory and a processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the asynchronous reinforcement learning training method as described in any one of claims 1 to 7.
10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the asynchronous reinforcement learning training method as described in any one of claims 1 to 7.