Strategy optimization method and device based on prompt-level priority sampling and empirical smoothing

Abstract

The invention relates to the technical field of machine learning models, and discloses a strategy optimization method and device based on prompt-level priority sampling and empirical smoothing, and the method comprises the steps: carrying out the sampling from a prompt set according to the priority of each prompt in the prompt set, and obtaining the prompt of a current training batch; generating a response by using the current strategy model and obtaining a corresponding reward; according to the prompted historical reward statistical magnitude and the currently obtained reward, after the prompted reward statistical magnitude is smoothly updated, the priority of the prompts in the prompt set is updated; and according to the updated smooth reward statistical magnitude and deviation compensation information introduced due to non-uniform sampling, performing strategy optimization updating on the current strategy model to obtain an updated strategy model, taking the updated strategy model as a current strategy model of a new round of training, and repeatedly executing training operation until a training stop condition is met. According to the method, the stability and the sample utilization efficiency of reinforcement learning training of the large language model can be improved, and efficient optimization of model performance is realized.

Description

Technical Field

[0001] This application relates to the field of machine learning model technology, specifically to a strategy optimization method and apparatus based on cue-level priority sampling and empirical smoothing. Background Technology

[0002] With the development of artificial intelligence technology, large language models (LLMs) are increasingly widely used in various natural language processing tasks. Reinforcement learning methods based on policy gradients have become a key technology for improving the quality of model output and aligning the model with human expectations. Among them, Group Relative Policy Optimization (GRPO) is a representative scheme that generates multiple responses to the same prompt and calculates the dominance value by normalizing within the group, and has been widely used in model reinforcement optimization.

[0003] However, existing policy optimization methods, such as GRPO, still have some shortcomings. On the one hand, their reward statistics rely on real-time data from the current training batch, failing to incorporate historical reward characteristics, leading to significant fluctuations in reward signals. This directly affects the stability of policy updates and easily causes model training oscillations. On the other hand, the use of uniform sampling to select training cues fails to distinguish the differences in value of different cues for model training. This results in high-value cues not being fully utilized, while low-value cues consume excessive computational resources, leading to low sample utilization efficiency. Therefore, existing reinforcement learning policy optimization methods for large-scale language models suffer from insufficient training stability and low sample utilization efficiency.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0005] This application provides a policy optimization method and apparatus based on cue-level priority sampling and empirical smoothing, which can improve the stability and sample utilization efficiency of reinforcement learning training of large language models and achieve efficient optimization of model performance.

[0006] In a first aspect, embodiments of this application provide a strategy optimization method based on cue-level priority sampling and empirical smoothing, including: Based on the priority of each prompt in the prompt set, the prompts for the current training batch are sampled from the prompt set; For each cue in the current training batch, at least one response is generated using the current policy model, and the reward corresponding to each response is obtained; For each prompt, the reward statistics of the prompt are smoothly updated based on the historical reward statistics of the prompt and the currently obtained reward, so as to obtain the smoothed reward statistics of the prompt after the update; Based on the updated smoothed reward statistics, the priority of the prompt in the prompt set is updated; Based on the updated smoothed reward statistics and the bias compensation information introduced by non-uniform sampling, the current policy model is optimized and updated to obtain the updated policy model. The updated policy model is used as the current policy model for the new round of training, and the training operation is repeatedly executed based on the updated cue priority until the training stop condition is met; wherein, the training operation includes the entire process from cue sampling to policy model optimization and update.

[0007] Furthermore, in some embodiments of this application, the step of sampling the prompts for the current training batch from the prompt set according to the priority of each prompt in the prompt set includes: Obtain the current priority value of each prompt in the prompt set; Based on the current priority value of each prompt, calculate the probability that each prompt will be sampled; the higher the current priority value, the greater the sampling probability. Multiple prompts are randomly selected non-uniformly from the prompt set according to the sampling probability to form the current training batch.

[0008] Furthermore, in some embodiments of this application, obtaining the current priority value of each prompt in the prompt set includes: obtaining the current priority value of each prompt by querying a tree-like cumulative data structure, wherein the leaf nodes of the tree-like cumulative data structure store the priority values ​​of each prompt, and the internal nodes of the tree-like cumulative data structure store the sum of the priority values ​​of the child nodes; The step of calculating the probability of each prompt being sampled based on the current priority value of each prompt includes: determining the sampling probability based on the total value of the root node of the tree-like cumulative data structure and the values ​​stored in each leaf node; The step of non-uniformly and randomly selecting multiple prompts from the prompt set according to the sampling probability to form the current training batch includes: generating a random number between zero and the total value of the root node; starting from the root node of the tree-like cumulative data structure, traversing down layer by layer according to the random number to a leaf node; and selecting the prompts corresponding to the leaf nodes into the current training batch.

[0009] Furthermore, in some embodiments of this application, the step of smoothly updating the reward statistics of each prompt based on the historical reward statistics of the prompt and the currently obtained reward to obtain the smoothed reward statistics of the prompt includes: For a given prompt, based on the reward of at least one response currently obtained for the prompt, calculate the reward statistic for the current batch of the prompt, wherein the reward statistic includes the mean of the within-group reward and the variance of the within-group reward; Obtain the historical smoothed reward statistics of the prompt, wherein the historical smoothed reward statistics include the historical smoothed mean and the historical smoothed variance; The historical smoothed reward statistics of the prompt are updated using the reward statistics of the current batch of the prompt, so as to obtain the updated smoothed reward statistics of the prompt.

[0010] Furthermore, in some embodiments of this application, updating the historical smoothed reward statistics of the prompt using the reward statistics of the current batch of the prompt to obtain the updated smoothed reward statistics of the prompt includes: The updated smoothed mean is obtained by taking a weighted average of the historical smoothed mean and the mean of the intra-group reward of the current batch. The updated smoothed variance is obtained by taking a weighted average of the historical smoothed variance and the within-group reward variance of the current batch.

[0011] Furthermore, in some embodiments of this application, updating the priority of the prompt in the prompt set based on the updated smoothed reward statistics includes: For a given prompt, calculate the current advantage value of the prompt based on the smoothed mean of the currently obtained reward and the smoothed reward statistics after the prompt is updated; The priority update value of the prompt is determined based on the advantage value; Update the priority of the prompt in the prompt set to the priority update value.

[0012] Furthermore, in some embodiments of this application, the step of calculating the current advantage value of a prompt based on the smoothed mean of the currently obtained reward and the updated smoothed reward statistics of the prompt includes: Calculate the difference between the currently obtained reward and the smoothed mean; The difference is normalized using the smoothed variance in the updated smoothed reward statistic to obtain the advantage value.

[0013] Furthermore, in some embodiments of this application, the deviation compensation information is an importance sampling weight, and the calculation method of the importance sampling weight includes: Calculate the actual sampling probability of the prompt during the prompt sampling process; Calculate the theoretical sampling probability of the prompt under uniform sampling conditions; Calculate the ratio of the actual sampling probability to the theoretical sampling probability; The ratio is subjected to a power transformation, and the result of the power transformation is used as the importance sampling weight of the prompt.

[0014] Furthermore, in some embodiments of this application, the reward includes a correctness reward and a format reward. Therefore, generating at least one response using the current policy model for each cue in the current training batch and obtaining the reward corresponding to each response includes: Calculate a correctness bonus score based on the correctness of the response; Determine whether the response conforms to the preset reasoning format specification, and calculate the format reward score based on the determination result; The correctness reward score and the format reward score are summed to obtain the final reward corresponding to the response.

[0015] Secondly, embodiments of this application provide a strategy optimization apparatus based on cue-level priority sampling and empirical smoothing, comprising: The prompt sampling module is used to sample the prompts of the current training batch from the prompt set according to the priority of each prompt in the prompt set; The response reward module is used to generate at least one response for each cue in the current training batch using the current policy model, and to obtain the reward corresponding to each response. The smooth update module is used to smoothly update the reward statistics of each prompt based on the historical reward statistics of the prompt and the currently obtained reward, so as to obtain the smoothed reward statistics of the prompt after the update. The priority update module is used to update the priority of the prompt in the prompt set based on the updated smoothed reward statistics; The strategy optimization module is used to optimize and update the current strategy model based on the updated smoothed reward statistics and the bias compensation information introduced by non-uniform sampling, so as to obtain the updated strategy model. The iterative training module is used to take the updated policy model as the current policy model for the new round of training, and repeatedly execute the training operation based on the updated prompt priority until the training stop condition is met; wherein, the training operation includes the entire process from prompt sampling to policy model optimization and update.

[0016] This application provides a policy optimization method and apparatus based on cue-level priority sampling and empirical smoothing. First, it samples the current training batch of cue according to the priority of each cue in the cue set, enabling more valuable cue for model training to receive more training opportunities and avoiding excessive computational burden on low-value cue, thereby improving sample utilization efficiency. Second, it performs smooth updates for each cue by combining historical reward statistics with the current reward, effectively reducing reward signal fluctuations caused by single batches of data, making reward statistics more stable, and providing a reliable reference for subsequent policy optimization. Simultaneously, it dynamically adjusts cue priority based on the updated smoothed reward statistics, allowing the sampling strategy to adapt to the model training state in real time, forming a virtuous cycle. Then, it optimizes the policy model by incorporating bias compensation information introduced by non-uniform sampling, ensuring the accuracy and rationality of policy updates and reducing the impact of sampling bias on training results. Finally, it continuously optimizes the policy model through iterative training, improving the stability of reinforcement learning training for large language models while optimizing sample utilization efficiency, enabling the model to achieve performance improvements in a more efficient manner. In summary, this application effectively improves the stability and sample utilization efficiency of reinforcement learning training for large language models by dynamically adjusting the prompt priority, smoothly updating reward statistics, and iteratively optimizing the policy model. This achieves efficient optimization of model performance and solves the problems of insufficient training stability and low sample utilization efficiency in existing reinforcement learning policy optimization methods for large language models. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an application environment diagram of the strategy optimization method based on prompt-level priority sampling and empirical smoothing provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the strategy optimization method based on prompt-level priority sampling and empirical smoothing provided in the embodiments of this application; Figure 3 This is a schematic diagram of the system collaborative processing of the strategy optimization method based on prompt-level priority sampling and empirical smoothing provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the strategy optimization device based on prompt-level priority sampling and empirical smoothing provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of systems and methods consistent with those detailed in the appended claims or with some aspects of this application.

[0020] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover descriptions such as non-exclusive inclusion, so that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0021] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0022] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0023] To address the aforementioned technical problems and overcome the shortcomings of existing technologies, this application provides a strategy optimization method and apparatus based on cue-level priority sampling and empirical smoothing, which can improve the stability and sample utilization efficiency of large-scale language model reinforcement learning training and achieve efficient optimization of model performance.

[0024] Figure 1 This is an application environment diagram for a policy optimization method based on cue-level priority sampling and empirical smoothing in one embodiment. (Refer to...) Figure 1This policy optimization method based on cue-level priority sampling and empirical smoothing is applied to a policy optimization system based on cue-level priority sampling and empirical smoothing. The system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal, and the mobile terminal can be at least one of a mobile phone, tablet computer, or laptop computer. The server 120 can be a standalone server or a server cluster consisting of multiple servers. Server 120 is configured to execute the aforementioned policy optimization method based on cue-level priority sampling and empirical smoothing, including: sampling cues for the current training batch from the cue set according to the priority of each cue in the cue set; generating at least one response for each cue in the current training batch using the current policy model and obtaining the reward corresponding to each response; for each cue, smoothing the reward statistics of the cue based on the historical reward statistics and the currently obtained reward to obtain the updated smoothed reward statistics; updating the priority of the cue in the cue set based on the updated smoothed reward statistics; optimizing and updating the current policy model based on the updated smoothed reward statistics and the bias compensation information introduced by non-uniform sampling to obtain the updated policy model; using the updated policy model as the current policy model for the new round of training, and repeatedly executing the training operation based on the updated cue priority until the training stopping condition is met; wherein, the training operation includes the entire process from cue sampling to policy model optimization and updating.

[0025] Please see Figure 2 , Figure 2 This is a flowchart illustrating a policy optimization method based on cue-level priority sampling and empirical smoothing according to an embodiment of this application. This embodiment primarily uses the application of this policy optimization method based on cue-level priority sampling and empirical smoothing to a computer device as an example for illustration. Specifically, the policy optimization method based on cue-level priority sampling and empirical smoothing according to an embodiment of this application may include the following steps: S1. Based on the priority of each prompt in the prompt set, sample the prompts for the current training batch from the prompt set; Specifically, for step S1, during the reinforcement learning training of a large-scale language model, a set of prompts containing multiple task-related prompts is pre-constructed. Each prompt corresponds to a priority reflecting its training value. This priority is determined based on the model's past learning feedback on each prompt during training, reflecting the contribution of the prompt to the improvement of model performance. During sampling, sampling probabilities are assigned according to the priority of each prompt. The higher the priority of a prompt, the greater the probability of it being selected into the current training batch. Finally, a certain number of prompts are selected from the prompt set to form the current training batch, ensuring that training resources are tilted towards high-value prompts. For example, if the prompt set contains 7000 math problems, the prompt related to "finding the root of a quadratic function" has a higher priority due to the model's poor past learning performance and high potential for improvement, and will be more likely to be selected into the current training batch during sampling. On the other hand, the prompt for "basic addition and subtraction operations," which the model has mastered, has a lower priority and a relatively smaller probability of being selected. S2. For each cue in the current training batch, generate at least one response using the current policy model and obtain the reward corresponding to each response; Specifically, in step S2, for each sampled prompt, the policy model currently in the training phase is invoked to generate at least one corresponding response for each prompt. After the response is generated, each response is scored according to a preset reward evaluation standard, and a reward is obtained for each response. The reward setting aims to guide the model to output high-quality results. For example, in a mathematical reasoning task, if the response generated by the model for a prompt for a geometric proof problem accurately derives the conclusion and is logically rigorous, a positive correctness reward will be given; if the response conforms to the preset reasoning format specifications (such as containing a clear "final answer" label), an additional format reward will be given. Finally, the correctness reward and the format reward are summed to obtain the final reward for the response; if the response answer is incorrect or the format is disordered, a negative reward or a lower score reward will be given.

[0026] S3. For each prompt, based on the historical reward statistics of the prompt and the current reward obtained, the reward statistics of the prompt are smoothly updated to obtain the smoothed reward statistics after the prompt is updated; Specifically, for step S3, for each prompt, the system continuously records its reward statistics during past training processes, forming historical reward statistics. These statistics reflect the model's long-term learning performance on that prompt. After obtaining all response rewards for the prompt in the current batch, the system first calculates the reward statistics for that prompt in the current batch (such as the mean and variance of the rewards), then merges the reward statistics of the current batch with the historical reward statistics. Smoothing is then used to eliminate the impact of random fluctuations in reward data from a single batch, resulting in updated smoothed reward statistics. For example, if the historical smoothed reward mean for a prompt is 8 points (out of 10), and the current batch generates 3 responses with rewards of 7, 8, and 9 points respectively, and the current batch's reward mean is 8 points, after smoothing, the smoothed reward mean for that prompt will be adjusted by combining historical and current data, ultimately obtaining a more stable updated smoothed reward mean, avoiding statistical distortion caused by occasional extreme reward values ​​in a single batch.

[0027] S4. Based on the updated smoothed reward statistics, update the priority of the cue in the cue set; Specifically, for step S4, the priority of each cue is not fixed but dynamically adjusted based on its updated smoothed reward statistic. By comparing the response reward of the current batch for that cue with the updated smoothed reward statistic, the model's learning progress and potential for improvement on that cue can be determined: if the current reward is significantly lower than the smoothed reward statistic, it indicates that the model still has significant potential for improvement on that cue, and its priority will be increased accordingly; if the current reward is close to or higher than the smoothed reward statistic, it indicates that the model has mastered the task related to that cue well, and its priority will be appropriately decreased. For example, if the average smoothed reward for a certain "probability calculation" cue is 6 points after the update, and the response reward for the current batch is 4 points, it indicates that the model's performance on this cue has not reached the historical average level, and there is significant potential for improvement. Therefore, the priority of this cue will be increased to provide more training opportunities in subsequent training. S5. Based on the updated smoothed reward statistics and the bias compensation information introduced by non-uniform sampling, the current policy model is optimized and updated to obtain the updated policy model. Specifically, in step S5, the use of non-uniform sampling to select cues may lead to a bias in the training data distribution towards high-priority cues, introducing sampling bias. Therefore, bias compensation information is needed to correct this bias. Combining the updated smoothed reward statistics (providing a reliable learning signal for policy updates) with the bias compensation information (to offset the effects of non-uniform sampling), the parameters of the current policy model are adjusted and updated using a policy gradient optimization method commonly used in reinforcement learning. During the update process, the learning signal reflected by the smoothed reward statistics serves as a guide, while the bias compensation information ensures the unbiasedness of model learning, enabling the model to optimize towards outputting high-quality responses, ultimately resulting in the policy model with updated parameters.

[0028] S6. Use the updated policy model as the current policy model for the new round of training, and repeat the training operation based on the updated cue priority until the training stopping condition is met; wherein, the training operation includes the entire process from cue sampling to policy model optimization and update; Specifically, in step S6, the updated policy model is used as the initial policy model for the next round of training. Simultaneously, the updated cue priorities are used to repeat the entire training process from cue sampling to policy model optimization and updating. This iterative process continues, with each round of training optimizing model parameters and cue priorities based on the previous round's results, continuously improving model performance. Training stops when preset stopping conditions are met (e.g., the model's performance metrics on the validation set reach the target value, the number of training iterations reaches a set threshold, or model performance no longer shows significant improvement), ultimately yielding the optimized policy model.

[0029] This embodiment achieves efficient sampling by dynamically adjusting the prompt priority, and ensures the stability of the learning signal by combining historical and current rewards for smooth updates. Furthermore, it continuously improves the model through bias compensation and iterative optimization, ultimately effectively improving the stability and sample utilization efficiency of large-scale language model reinforcement learning training, and helping the model to quickly achieve performance improvement.

[0030] Furthermore, in some embodiments, step S1, "sampling the prompts for the current training batch from the prompt set according to the priority of each prompt in the prompt set," may specifically include: S11. Get the current priority value of each hint in the hint set; Specifically, for step S11, in the reinforcement learning training scenario of a large language model, a set of prompts related to multiple training tasks is pre-constructed, with each prompt corresponding to a real-time updated priority value. This priority value is determined based on the model's past learning feedback on the prompt (such as the quality of the prompt's response during previous training, the model's potential for improvement, etc.), and is used to quantify the value of the prompt for model training. When obtaining the priority value, the latest priority record of each prompt stored in the system is directly retrieved to ensure that the priority value of each prompt truly reflects its current training value. For example, in a mathematical reasoning training task, the prompt set includes multiple prompts such as "solving quadratic equations," "geometric proofs," and "probability distribution calculations." The "geometric proofs" prompt has a current priority value of 9 (priority value range 0-10) because the model's past learning performance was poor and it has great potential for improvement. The "solving quadratic equations" prompt has a current priority value of 3 because the model has basically mastered it.

[0031] S12. Calculate the probability of each prompt being sampled based on its current priority value. The higher the current priority value, the greater the sampling probability. Specifically, in step S12, after obtaining the current priority values ​​of all prompts, the sampling probability of each prompt is calculated based on these priority values. The core logic is to establish a positive correlation between priority values ​​and sampling probabilities; a higher priority value means a greater potential contribution of the prompt to improving model performance, and thus a higher allocated sampling probability. In the specific calculation, the sum of the priority values ​​of all prompts in the prompt set is first calculated, and then the priority value of a single prompt is divided by this sum to obtain the sampling probability of that prompt. For example, if the prompt set contains three prompts with current priority values ​​of 9, 6, and 5, totaling 20, then the sampling probability of the first prompt is 9 / 20 = 45%, the second is 6 / 20 = 30%, and the third is 5 / 20 = 25%, clearly demonstrating the allocation rule that higher priority equates to higher sampling probabilities.

[0032] S13. Based on the sampling probability, multiple prompts are randomly selected from the prompt set in a non-uniform manner to form the current training batch; Specifically, in step S13, based on the calculated sampling probabilities of each prompt, a non-uniform random selection method is used to filter prompts from the prompt set to form the current training batch. Non-uniform random selection means that the probability of each prompt being selected is determined by its own sampling probability, rather than an equal chance, ensuring that prompts with high sampling probabilities are more likely to enter the training batch. For example, if the current training batch needs to select 2 prompts, combining the sampling probabilities of the above 3 prompts (45%, 30%, 25%), when randomly selecting, the prompt with a priority value of 9 (sampling probability of 45%) has a significantly higher probability of being selected than the other two prompts; ultimately, this high-priority prompt and the prompt with a sampling probability of 30% may be selected together to form the current training batch, so that training resources are concentrated on more valuable prompts.

[0033] This embodiment calculates the sampling probability based on the priority of the prompts and performs non-uniform sampling, so that prompts that are more valuable to model training receive a higher selection probability. This effectively avoids low-value prompts occupying too many training resources, significantly improves the utilization efficiency of training samples, and lays the foundation for efficient optimization of the model in the future.

[0034] Furthermore, in some embodiments, step S1, "obtaining the current priority value of each prompt in the prompt set", may specifically include: obtaining the current priority value of each prompt by querying a tree-like cumulative data structure, wherein the leaf nodes of the tree-like cumulative data structure store the priority values ​​of each prompt, and the internal nodes of the tree-like cumulative data structure store the sum of the priority values ​​of the child nodes; Specifically, in large-scale language model reinforcement learning training, a tree-like cumulative data structure is used to manage the priority information of all prompts. This data structure is a binary tree, where each leaf node uniquely corresponds to a prompt, and the value stored in the leaf node is the current priority value of that prompt. Each internal node stores the sum of the priority values ​​of its left and right child nodes. Through this hierarchical accumulation method, the total priority of all prompts can be quickly summarized. To obtain the current priority value of a prompt, simply locate the leaf node corresponding to that prompt in the tree-like cumulative data structure and directly read the value stored in that leaf node. For example, the hint set contains three hints: "trigonometric function simplification", "sequence summation", and "solid geometry calculation". These correspond to three leaf nodes in a tree structure, with priority values ​​stored as 6, 4, and 2, respectively. The left inner node of this structure (corresponding to the parent node of the first two hints) stores 6+4=10, and the root node (corresponding to the parent node of all hints) stores 10+2=12. To obtain the priority value of the "sequence summation" hint, simply query its corresponding leaf node to get the value 4.

[0035] Then step S12, "Calculate the probability of each prompt being sampled based on the current priority value of each prompt", may specifically include: determining the sampling probability based on the total value of the root node of the tree-based cumulative data structure and the values ​​stored in each leaf node; Specifically, the calculation of sampling probability is based on the core features of the tree-structured cumulative data structure: First, the value stored in the root node is read. This value is the sum of all prompt priority values, representing the total priority weight of the entire prompt set. Then, the priority value stored in the leaf node corresponding to each prompt is compared with the total value of the root node. The result is the sampling probability of that prompt. Since the higher the priority value, the larger its ratio to the total value, the sampling probability is naturally positively correlated with the priority. For example, continuing the above example, the total value of the root node is 12. The leaf node value of the "trigonometric function simplification" prompt is 6, and its sampling probability is 6 / 12 = 50%; the sampling probability of the "sequence summation" prompt is 4 / 12 ≈ 33.3%; and the sampling probability of the "solid geometry calculation" prompt is 2 / 12 ≈ 16.7%, clearly demonstrating the logic that the higher the priority, the greater the sampling probability.

[0036] Then step S13, "selecting multiple prompts from the prompt set non-uniformly and randomly according to the sampling probability to form the current training batch", may specifically include: generating a random number between zero and the total value of the root node; starting from the root node of the tree-like cumulative data structure, traversing down layer by layer according to the random number to a leaf node; and selecting the prompts corresponding to the leaf nodes into the current training batch. Specifically, first, a random number is generated, with its value strictly between 0 and the total value stored in the root node. This random number will serve as the basis for target location in the sampling. Then, starting from the root node of the tree-like cumulative data structure, the process traverses downwards layer by layer according to preset rules: first, the random number is compared with the value stored in the left child node of the current node. If the random number is less than the value of the left child node, the target hint is located in the left subtree, and the traversal continues directly into the left child node. If the random number is greater than or equal to the value of the left child node, the target hint is located in the right subtree. In this case, the value of the left child node is subtracted from the random number to obtain the new target value, and the traversal continues into the right child node. This process is repeated until a leaf node is reached; the hint corresponding to this leaf node is the hint selected for this sampling. This selection process is repeated until the required number of hints are collected, constituting the current training batch. For example, if we need to select one prompt, the total number of root nodes is 12, and the generated random number is 7.5: First, compare 7.5 with the value 10 of the left inner node. Since 7.5 < 10, we move to the left child node. Then, compare 7.5 with the value 6 of the left child node (the leaf node corresponding to "Trigonometric Function Simplification"). Since 7.5 ≥ 6, we use 7.5 - 6 = 1.5 as the new target value and move to the right child node (the leaf node corresponding to "Summation of Sequence"). At this point, we have traversed to the leaf node and finally selected the "Summation of Sequence" prompt to enter the current training batch.

[0037] This embodiment efficiently manages and prioritizes query suggestions through a tree-based cumulative data structure, quickly completing sampling probability calculation and non-uniform sampling. While ensuring that high-value suggestions are selected first, it significantly improves the efficiency of the sampling process, providing strong support for the efficient advancement of model training.

[0038] Furthermore, in some embodiments, step S3, "for each prompt, based on the historical reward statistics of the prompt and the currently obtained reward, smoothly updates the reward statistics of the prompt to obtain the smoothed reward statistics after the prompt is updated," may specifically include: S31. For a given prompt, calculate the reward statistic for the current batch of the prompt based on the reward for at least one response currently obtained from the prompt, wherein the reward statistic includes the mean reward within the group and the variance reward within the group; Specifically, for step S31, for a single cue, the model generates at least one corresponding response in the current training batch, and each response receives a reward score based on a preset standard. To accurately reflect the training performance of the cue in the current batch, it is necessary to calculate the reward statistics for the current batch based on the reward scores of these responses. The core of these statistics includes the mean within-group reward and the variance within-group reward. The mean within-group reward reflects the overall quality level of the cue's response in the current batch, while the variance within-group reward reflects the fluctuation of the rewards for each response in the current batch. For example, in a physics problem-solving training exercise, a "Newton's Laws of Motion Application" prompt generates 4 responses in the current batch, with corresponding reward scores of 7, 8, 8, and 9 points respectively. First, calculate the mean reward within the group: (7+8+8+9)÷4=8 points; then calculate the variance of the reward within the group: first, calculate the square of the difference between each reward and the mean ((7-8)²=1, (8-8)²=0, (8-8)²=0, (9-8)²=1), then calculate the average of these squares, i.e. (1+0+0+1)÷4=0.5. Finally, the reward statistics for the current batch of this prompt are a mean of 8 points and a variance of 0.5.

[0039] S32. Obtain the historical smoothed reward statistics, which include the historical smoothed mean and the historical smoothed variance. Specifically, for step S32, each cue accumulates reward-related statistical data throughout the training process. This data, after smoothing, forms historical smoothed reward statistics, reflecting the model's long-term learning trend on that cue, rather than the accidental performance of a single batch. The historical smoothed reward statistics also include the historical smoothed mean and historical smoothed variance, which are continuously stored and updated in real time by the system. For example, continuing with the "Application of Newton's Laws of Motion" cue example, retrieving the historical smoothed reward statistics for that cue from the system yields a historical smoothed mean of 7.2 and a historical smoothed variance of 0.8. These data represent the accumulated results of smoothing the reward statistics from previous training rounds for that cue.

[0040] S33. Use the reward statistics of the current batch to update the historical smoothed reward statistics of the prompt, and obtain the updated smoothed reward statistics of the prompt; Specifically, for step S33, to avoid the random fluctuations in the reward statistics of a single batch interfering with subsequent training decisions, it is necessary to merge and update the reward statistics of the current batch with the historical smoothed reward statistics. The update process takes into account both the long-term trend of historical data and the latest performance of current data. Through reasonable fusion rules, the influence of extreme values ​​or random fluctuations is eliminated, ultimately resulting in a more stable and valuable updated smoothed reward statistic. For example, in response to the above suggestion, the reward mean of the current batch (8 points) is merged with the historical smoothed mean (7.2 points), and the variance of the current batch (0.5 points) is merged with the historical smoothed variance (0.8 points). The resulting updated smoothed mean might be 7.4 points, and the updated smoothed variance might be 0.7, reflecting both the progress of the current batch and preserving the stability of historical trends, thus avoiding overestimating the statistical results due to the good performance of a single batch.

[0041] This embodiment updates the reward statistics by combining the current batch reward statistics with the historical smoothed reward statistics, effectively filtering out random fluctuations in single batch reward data. This makes the reward statistics results more stable and reliable, providing accurate reference for subsequent model training and helping to improve the stability of training.

[0042] Furthermore, in some embodiments, step S33, "updating the historical smoothed reward statistics of the prompt using the reward statistics of the current batch to obtain the updated smoothed reward statistics of the prompt," may specifically include: S331. Take a weighted average of the historical smoothed mean and the mean of the group rewards in the current batch to obtain the updated smoothed mean; Specifically, in step S331, when smoothing the average reward of the prompt, both the trend accumulated from historical training and the latest performance of the current batch are considered. A preset weighting rule is used to allocate the weights of both, ensuring that the reference value of historical data is not completely ignored while also fully reflecting the training feedback of the current batch. The historical smoothed average is the stable average data formed after multiple smoothing updates over previous training rounds, reflecting the model's long-term learning level of the prompt; the average reward within the current batch is the average level of all responses to the prompt during this training, reflecting the performance under the latest training state. For example, the weighting coefficient for the historical smoothed mean is set to 0.7, and the weighting coefficient for the average reward within the current batch is set to 0.3 (the sum of the two coefficients is 1, ensuring the weighting logic is reasonable). If the historical smoothed mean of a certain "chemical equation balancing" is 7.5 points, and the current batch generates 3 responses with rewards of 8, 9, and 8 points respectively, the average reward within the current batch is (8+9+8)÷3≈8.33 points. Then the updated smoothed mean = 7.5×0.7+8.33×0.3≈7.5+2.5=10? No, recalculate: 7.5×0.7=5.25, 8.33×0.3≈2.5, the sum ≈7.75 points, which reflects the progress of the current batch without deviating from the long-term historical trend, avoiding the one-sided influence of data from a single batch.

[0043] S332. Take a weighted average of the historical smoothed variance and the within-group reward variance of the current batch to obtain the updated smoothed variance; Specifically, for step S332, consistent with the smoothing update logic of the mean, the variance update also adopts a weighted average method. The core is to filter out the fluctuation noise of the reward data in a single batch and maintain the stability of the variance statistics. The historical smoothed variance reflects the long-term characteristics of the reward fluctuation in the past training, while the within-group reward variance of the current batch reflects the dispersion of each response reward in this training. For example, continuing with the "chemical equation balancing" example above, we set the weighting coefficient of historical smoothing variance to 0.7 and the weighting coefficient of the variance of the current batch's within-group reward to 0.3. The historical smoothing variance of this suggestion is 0.6. The within-group variance of the three response rewards (8 points, 9 points, and 8 points) in the current batch is calculated as follows: First, we calculate the square of the difference between each reward and the current mean of 8.33 points ((8-8.33)²≈0.11, (9-8.33)²≈0.45, (8-8.33)²≈0.11), then we calculate the average of the squared values, i.e., (0.11+0.45+0.11)÷3≈0.22. Then, the updated smoothing variance = 0.6×0.7+0.22×0.3≈0.42+0.066≈0.486, which absorbs the positive signal of the small fluctuation of the current batch's rewards while retaining the reference value of historical fluctuation characteristics, making the variance statistics more stable and reliable.

[0044] This embodiment effectively balances historical training trends and current training feedback by updating the historical smoothed mean, variance, and corresponding statistics of the current batch using a weighted average. It filters out random fluctuations in single batch data, making the updated smoothed reward statistics more stable and accurate, and providing a reliable basis for subsequent priority adjustments and strategy optimization.

[0045] Furthermore, in some embodiments, step S4, "updating the priority of the cue in the cue set based on the updated smoothed reward statistics," may specifically include: S41. For a given prompt, calculate the current advantage value of the prompt based on the smoothed mean of the current reward obtained and the smoothed reward statistics after the prompt is updated; Specifically, for step S41, the updated smoothed mean is the stable reward benchmark obtained after the prompt has accumulated through multiple rounds of training and fused with the current batch of data, reflecting the model's long-term average performance on that prompt. The current reward is the latest response quality feedback of the model to that prompt during this training. The core function of the advantage value is to quantify the difference between the model's current performance and its long-term average level. The calculation is based on the comparison between the two, intuitively reflecting the potential value of the prompt for model training. If the current reward is higher than the smoothed mean, it means that the model's current performance is better than the historical average, and the advantage value is positive; if the current reward is lower than the smoothed mean, it means that the model still has significant room for improvement on that prompt, and the advantage value is negative. For example, the updated smoothed mean of a certain "English grammar correction" prompt is 7 points, and the reward corresponding to the response generated by the model in the current batch is 5 points. By comparing and calculating, the advantage value is -2 points, clearly indicating that this prompt is a weak point in the current training of the model and needs to be focused on.

[0046] S42. Determine the priority of the prompt based on the advantage value and update the value accordingly; Specifically, for step S42, the core of setting the priority update value is to accurately match the training priority of the prompt with the model's improvement needs. When the advantage value is negative and the absolute value is large, it means that the model's current performance on that prompt is far below the long-term average, with great potential for improvement and a higher contribution to the overall performance optimization of the model. Therefore, its priority update value will be set to a higher level. When the advantage value is positive and the absolute value is large, it indicates that the model has mastered the task related to the prompt well, and the marginal benefit of further training is low. Its priority update value will be appropriately reduced. If the advantage value is close to zero, it means that the current performance is on par with the long-term level, and the priority update value can be maintained or slightly adjusted. For example, continuing with the example of the "English grammar correction" prompt, if its advantage value is -2 points (negative and with a large absolute value), its priority update value will be increased from the original 4 (assuming the original priority range is 1-10) to 8. If the advantage value of another "simple English translation" prompt is +1.5 points, its priority update value will be decreased from 6 to 3.

[0047] S43. Update the priority of the prompt in the prompt set to the priority update value; Specifically, in step S43, the system maintains the priority records of all prompts in the prompt set in real time. Once the priority update value for a prompt is determined, it directly replaces the original priority value with the updated value, ensuring that the updated priority immediately applies to the next round of training. After the update, the prompt's priority accurately reflects its current value to model training, providing a reliable basis for subsequent sampling. For example, if the original priority of the "English grammar correction" prompt is 4, and its priority update value is determined to be 8, the system will change the priority record of this prompt in the set from 4 to 8, giving it a higher probability of being selected in the next round of sampling.

[0048] This embodiment quantifies the difference between the model's current performance and its long-term level by using the advantage value, and then dynamically adjusts the priority of prompts, giving higher training priority to prompts corresponding to the model's weaknesses. This achieves precise allocation of training resources and effectively improves the targeting of training and the efficiency of data utilization.

[0049] Further, in some embodiments, step S41, "for a prompt, calculating the current advantage value of the prompt based on the smoothed mean of the currently obtained reward and the updated smoothed reward statistics of the prompt," may specifically include: S411. Calculate the difference between the currently obtained reward and the smoothed mean; Specifically, for step S411, the smoothed mean is a stable reward benchmark obtained by fusing historical data with the current batch of data, objectively reflecting the model's long-term average performance on that prompt; the current reward is the model's latest response quality feedback to that prompt during this training. To initially quantify the gap between the model's current performance and the long-term average, the difference between the two needs to be calculated directly. The sign of the difference reflects the quality of the current performance relative to the long-term level, while the absolute value of the difference reflects the magnitude of the gap. For example, if the smoothed mean of a certain "data analysis modeling" prompt is 8.2 points after the update, and the reward corresponding to the response generated by the model in the current batch is 6.7 points, the difference is calculated as 6.7 - 8.2 = -1.5 points. This difference intuitively indicates that the current response performance is lower than the long-term average, and the gap is 1.5 points.

[0050] S412. Normalize the difference using the smoothed variance in the updated smoothed reward statistic to obtain the dominance value; Specifically, in step S412, the smoothed variance reflects the fluctuation characteristics of the past reward data for the prompt. The reward fluctuation range of different prompts may differ, and directly comparing the performance gap between different prompts through the difference is not comparable. Therefore, it is necessary to normalize the obtained difference using smoothed variance to eliminate the influence of different prompt reward scales and fluctuation ranges, so that the final advantage value has a unified quantitative standard and more accurately reflects the relative difference between the model's current performance and its long-term level. For example, continuing with the "Data Analysis Modeling" suggestion above, its updated smoothed variance is 0.81. Normalizing the difference of -1.5 points (calculated by dividing the difference by the square root of the smoothed variance), we get -1.5 ÷ √0.81 = -1.5 ÷ 0.9 ≈ -1.67. This result is the current dominance value for this suggestion. If another suggestion, "Data Visualization Design," has a difference of -1.2 points and a smoothed variance of 0.64, the normalized dominance value is -1.2 ÷ 0.8 = -1.5. This dominance value clearly indicates that the former's current performance is significantly worse than its long-term performance.

[0051] This embodiment uses difference calculation to initially quantify the performance gap, and then performs smoothing and variance normalization to make the advantage value more comparable and accurate. This provides an objective and unified quantitative basis for subsequent priority adjustment and strategy optimization, helping to improve the rationality of training decisions.

[0052] Furthermore, in some embodiments, the deviation compensation information in step S5 is an importance sampling weight, and the calculation method of the importance sampling weight includes: S501. Calculate the actual sampling probability of the prompt during the prompt sampling process; Specifically, for step S501, the actual sampling probability of a prompt is its true probability of being selected into the training batch during non-uniform sampling. This probability is determined by the prompt's own priority and the sum of the priorities of all prompts in the prompt set. Before sampling, the system assigns sampling weights based on the priority of each prompt. The actual sampling probability is essentially the proportion of a single prompt's priority to the sum of the priorities of all prompts; the higher the priority of a prompt, the greater its actual sampling probability. For example, if the prompt set for a training task contains 20 prompts, and the sum of the priorities of all prompts is 100, and the priority of the prompt "machine learning model tuning" is 10, then the actual sampling probability of this prompt is 10 ÷ 100 = 10%, meaning its true probability of being selected in this sampling is 10%.

[0053] S502. Calculate the theoretical sampling probability under uniform sampling conditions; Specifically, for step S502, under uniform sampling, each prompt has an equal chance of being selected, and there is no probability skew due to priority differences. Therefore, the calculation of the theoretical sampling probability is only related to the total number of prompts in the prompt set, that is, the theoretical sampling probability of a single prompt is equal to 1 divided by the total number of prompts in the prompt set. For example, continuing the above example, if there are 20 prompts in the prompt set, regardless of the content or potential value of each prompt, the theoretical sampling probability of each prompt under uniform sampling is 1÷20=5%, which is the baseline probability when uniformly distributing training resources.

[0054] S503. Calculate the ratio of the actual sampling probability to the theoretical sampling probability; Specifically, for step S503, to quantify the probability difference between non-uniform sampling and uniform sampling, the ratio of the actual sampling probability to the theoretical sampling probability needs to be calculated. This ratio directly reflects the "sampling skew" of non-uniform sampling on a single prompt: if the ratio is greater than 1, it means that the prompt has a higher probability of being selected in non-uniform sampling than in uniform sampling, and is a prompt that is "preferred"; if the ratio is less than 1, it means that its probability of being selected is lower than in uniform sampling, and is a prompt that is "undersampled". For example, the actual sampling probability of the above "machine learning model tuning" prompt is 10%, and the theoretical sampling probability is 5%, then the ratio of the two is 10% ÷ 5% = 2, indicating that the probability of the prompt being selected in non-uniform sampling is twice that in uniform sampling.

[0055] S504. Perform a power transformation on the comparison value, and use the result of the power transformation as the importance sampling weight for the prompt; Specifically, in step S504, the core purpose of the power transformation is to control the intensity of sampling bias compensation, avoiding weight imbalance caused by excessively large or small ratios, which would affect the stability of model training. By introducing a preset hyperparameter as the power, the obtained ratio is subjected to a power operation, and the result is the importance sampling weight of the prompt. For example, if the power hyperparameter is set to 0.5 (i.e., square root operation), and the above ratio is 2, then the importance sampling weight is 2^0.5≈1.414; if the actual sampling probability of another prompt is 2.5% and the theoretical sampling probability is 5%, with a ratio of 0.5, after the same power transformation, the weight is 0.5^0.5≈0.707. Through this transformation, the logic for compensating for sampling bias is preserved, while avoiding the interference of extreme weight values ​​on training.

[0056] This embodiment calculates the ratio of actual to theoretical sampling probabilities and performs a power transformation to obtain reasonable importance sampling weights, effectively compensating for the bias caused by non-uniform sampling, ensuring the unbiasedness of model strategy updates, and providing strong support for the stability of the training process.

[0057] Furthermore, in some embodiments, the reward in step S2 includes a correctness reward and a format reward. Therefore, step S2, "for each cue in the current training batch, generate at least one response using the current policy model and obtain the reward corresponding to each response," may specifically include: S21. Calculate the correctness bonus points based on the correctness of the response; Specifically, for step S21, the core of the correctness reward is to evaluate the core quality of the model's response, focusing on whether the response answer is consistent with the task objective and whether it is accurate. In practical applications, clear correctness judgment criteria and corresponding scoring rules are pre-set: if the core answer of the response is completely correct and meets the task requirements (such as correct solution, accurate information retrieval, etc.), a preset maximum positive score is given; if the answer is partially correct or has a slight deviation, a corresponding medium positive score is given according to the degree of deviation; if the answer is completely wrong or irrelevant to the task objective, a negative score or zero score is given. For example, in the mathematical equation-solving task, the preset correctness reward score range is -5 to 10 points. If the model gives the answer "x=2" (completely correct) for the response to "solving the equation 2x+3=7", the correctness reward score is 10 points; if it gives the answer "x=3" (calculation error), the correctness reward score is -5 points; if it gives the answer "x=2.5" (partial deviation), the correctness reward score is 3 points.

[0058] S22. Determine whether the response conforms to the preset reasoning format specification, and calculate the format reward score based on the judgment result; Specifically, for step S22, the format reward aims to guide the model to output a response with a clear structure that conforms to the usage habits of the task scenario. The preset inference format specifications will specify the requirements according to the specific task type, such as whether it includes designated markers (such as "final answer" or "inference steps"), whether the inference process is complete, and whether the expression is concise and standardized. If the response fully conforms to the preset format specifications, a positive format reward score is given; if it partially conforms to the specifications, a lower positive score or zero score is given according to the degree of missing or non-standardized parts; if it does not conform to the specifications at all (such as no key markers, chaotic inference logic, or lengthy and disorganized expression), a negative score is given. For example, continuing with the mathematical equation-solving task mentioned above, the pre-defined format specifications require the response to include clear markings such as "Solution Steps" and "Final Answer: XXX". The format bonus score ranges from -3 to 5 points. If the model response clearly lists the solution steps "1. Rearrange terms: 2x = 7 - 3 = 4; 2. Divide coefficients by 1: x = 4 ÷ 2 = 2" and also marks "Final Answer: x = 2", then the format bonus score is 5 points. If only the answer "x = 2" is given without listing the steps and markings as required, then the format bonus score is 0 points. If the response is disorganized and lacks any standardized format, then the format bonus score is -3 points.

[0059] S23. Sum the correctness bonus score and the format bonus score to obtain the final reward corresponding to the response; Specifically, for step S23, the final reward is a comprehensive quality evaluation of the model's response. By directly summing the correctness reward score and the format reward score, a dual consideration of "core answer quality" and "output format standardization" is achieved, ensuring that the model focuses on both accurate results and standardized output. For example, continuing the above example, if the correctness reward score of the model response is 10 points and the format reward score is 5 points, then the final reward is 10 + 5 = 15 points; if the correctness reward score is -5 points and the format reward score is -3 points, then the final reward is -5 + (-3) = -8 points; if the correctness reward score is 3 points and the format reward score is 0 points, then the final reward is 3 + 0 = 3 points.

[0060] This embodiment comprehensively evaluates the core quality of the model's response and the standardization of its output by splitting the rewards into two categories: correctness and format, and summing them together. This effectively guides the model to pay attention to both answer accuracy and format standardization, improving the comprehensiveness and guidance of the reward signal and providing a reasonable basis for the efficient training of the model.

[0061] To facilitate understanding of the policy optimization method based on cue-level priority sampling and experience smoothing provided in this embodiment, this embodiment also provides a specific implementation of the policy optimization method based on cue-level priority sampling and experience smoothing. Taking the implementation of this method in a policy optimization system based on cue-level priority sampling and experience smoothing as an example, the policy optimization system mainly includes a cue sampling module, a reward statistics update module, an advantage calculation module, a priority update module, and a policy optimization module.

[0062] The cue sampling module is used to select training samples from the cue set in each policy optimization iteration, enabling the training process to prioritize cue samples that significantly improve the model. This module is based on dynamic cue priority adjustment, SumTree tree structure storage, and a non-uniform weighted sampling mechanism, achieving a refined data selection strategy while maintaining efficient updates and real-time response. The entire module includes the following core steps: cue priority generation, priority structured storage, priority update, SumTree-based non-uniform sampling, and importance weight calculation.

[0063] (a) Priority generation prompt During each training session, the system generates multiple model outputs for each prompt and generates a priority value for each prompt based on the learning signals obtained from the reward calculation module and the advantage calculation module (such as the learning gain of the latest output, the average gain of all outputs, etc.).

[0064] Priority is used to represent the learning value of the current prompt sample. A higher value indicates a higher learning potential for the model and a greater likelihood of it being sampled first. Priorities can be generated based on the following methods: Average learning value strategy: Prioritize based on the average learning gain of multiple outputs.

[0065] Maximum learning value strategy: Prioritize the output with the highest benefit among multiple outputs.

[0066] Latest learning signal strategy (best results): The priority is updated directly based on the learning signal obtained from the most recently generated model output, enabling the system to respond quickly to changes in the current training state.

[0067] (ii) Priority storage structure based on SumTree To achieve fast priority updates and weighted sampling under large-scale cue sets, this invention uses SumTree as the core data structure for cue priority.

[0068] SumTree is a binary tree structure with the following characteristics: each leaf node corresponds to a cue and stores the priority value of that cue; each internal node stores the sum of the priorities of its left and right child nodes; the root node stores the sum of the priorities of all cues, which can be regarded as the "global priority" of the entire system. This structure allows priority updates and sampling to be completed in logarithmic time without recalculating the probability distribution of the entire cue set.

[0069] (III) Priority Update Mechanism When the effective learning value of a cue changes (e.g., the latest learning signal increases or decreases), the system updates the priority value of the leaf node corresponding to that cue. The update process is as follows: Locate the leaf node corresponding to the prompt; write the new priority of the leaf node into the tree structure. Update all its parent nodes upwards along the tree path, ensuring that the value of each parent node is always equal to the sum of the values ​​of its left and right child nodes. The root node ultimately stores the latest total priority of all prompts. This design allows priority changes to take effect immediately and influence the next sampling loop without causing a global recalculation or structural rebuild.

[0070] (iv) Non-uniform weighted sampling based on SumTree When a prompt is needed to enter the current training batch, the system performs non-uniform sampling based on the priority distribution within the SumTree. The specific method is as follows: The system first obtains the sum of priorities of the root node, which represents the "weight interval" of the entire cue set. The system randomly generates a value between 0 and the total priority as the target sampling position. Starting from the root, the system searches for the landing point layer by layer downwards according to the target value: if the target value falls within the interval of the left subtree, it enters the left subtree; otherwise, it enters the right subtree and deducts the interval value of the left subtree from the target value; when the leaf node is reached, the corresponding cue is the final sampling result.

[0071] The above process ensures that higher-priority hints occupy a larger interval in the tree and have a greater probability of being randomly hit; lower-priority hints have a shorter interval, but still have a certain probability of being sampled; the sampling process does not require explicit calculation of the probability distribution, but relies entirely on the internal structure of SumTree to naturally achieve priority normalization. This mechanism ensures efficiency while avoiding the cost and frequent recalculation problems of traditional probability construction methods.

[0072] (v) Calculation of Importance Weights Because the cue sampling adopts a non-uniform strategy, each sampling cue needs to be assigned an importance weight during model training to correct the sample bias caused by the difference in priority.

[0073] The importance weights are calculated based on the ratio between the "weight interval length" of the cue within the SumTree and the total interval of the root node: high-priority cues sampled frequently have their weights appropriately reduced to avoid over-amplifying the training signal; low-priority cues sampled infrequently have their weights appropriately increased to compensate for their lower sampling probability. The final weights are used in the policy optimization module to ensure the model's learning process remains unbiased and stable, further improving the convergence quality of the training.

[0074] The reward statistics update module is responsible for recording and updating reward statistics related to the "hint level", including the average reward and the magnitude of reward change.

[0075] The purpose of the rewards is to reflect the quality of each output, encouraging concise and correct reasoning while penalizing meaningless, verbose, or malformed output. This invention is primarily designed for mathematical reasoning tasks; therefore, the rewards mainly come from the following two aspects: Correctness reward: Determine whether the model's output answer is correct and give the corresponding positive or negative score.

[0076] Format reward: Determine whether the model output conforms to the standard inference format, such as whether it contains a label like "final answer".

[0077] The reward statistics are updated by recording the historical reward trends of each prompt, and then performing a "weighted smoothing". Specific steps: After each new output is generated, the average and variance (i.e., the fluctuation of the current reward) are calculated. The system uses an "exponential weighting" method to merge the current statistics with historical statistics. The most recent data has a larger weight, while the influence of older data gradually decreases. The smoothed statistics obtained in this way are more stable than those obtained from a single sampling and are less affected by extreme samples. The final smoothed reward statistics are used to: calculate more stable and lower-noise advantage values; guide the update of cue priority; and ensure smoother strategy optimization.

[0078] For the advantage calculation module, a stable advantage value is calculated using the smoothed mean / variance; this module compares the "current reward" with the "historical smoothed reward statistics" to obtain a more stable learning signal.

[0079] The meaning of advantage: If the current reward is higher than the past average level → the advantage is positive → the model is prompted that "this direction is worth strengthening"; if the current reward is lower than the past → the advantage is negative → the model is prompted that "this direction should be reduced".

[0080] After the advantages are stabilized, they will be passed on to the priority update module and the strategy optimization module.

[0081] For the priority update module, the priority value of the prompt is set according to the advantage value; this module uses the advantage value to update the priority of the prompt, so that the entire system forms a dynamic loop: Poor performance (low advantage) prompts → increased priority → the model encounters them more frequently → accelerated learning; High performance (high advantage) suggests a potential decrease in priority and avoids overtraining.

[0082] This mechanism is similar to "adaptive learning," where the system automatically spends time on the prompts that the model needs to learn most, thereby improving data efficiency.

[0083] The policy optimization module implements policy updates based on the PSPO objective function. This module uses: smoothed advantage, cue importance weights, constraints between the old and new policies (e.g., KL control), and a PPO-style stable update mechanism to update the model parameters. The updated policy serves as the basis for the next round of training.

[0084] like Figure 3 As shown, Figure 3This describes the collaborative processing of the strategy optimization system. The entire system forms a tightly coupled loop: the prompt sampling module selects prompts based on priority; the model generation module generates multiple outputs based on the current policy; the reward statistics update module calculates the reward and updates historical statistics; the advantage calculation module stabilizes the reward; the priority update module adjusts the prompt priority based on the latest advantage; the policy optimization module updates the policy based on the advantage and weights; and the updated policy is returned to the prompt sampling module to enter the next round. This closed-loop mechanism supports faster convergence, higher stability, and efficient training even with small datasets.

[0085] In a specific embodiment, the workflow of the strategy optimization system is as follows: Step 1. Cue Sampling: Based on cue level priority Calculate sampling probability And sampling suggestions based on this distribution. : in It is a training dataset set. This determines the priority level to use. This corresponds to uniform sampling.

[0086] Step 2. Generate intra-group response: For the sampled prompts Using old strategies generate Output each item, receive the corresponding reward, and calculate the average reward within the group. and variance , Indicates the first Samples were collected the next time. The calculated within-group mean, Similarly; Step 3. Experience-weighted reward smoothing: For hints The reward statistics will be updated using EMA. in It is the smoothing coefficient.

[0087] Step 4. Advantage Calculation: Use smoothed reward statistics to calculate the advantage, improving the stability of the advantage value calculation; in This indicates a random disturbance.

[0088] Step 5. Priority Update (PPS): Priority can be implemented using any of the following methods: average, maximum, latest learning value (i.e., the absolute value of the advantage).

[0089] Step 6. Importance Sampling Weight Correction: To eliminate gradient bias caused by non-uniform sampling, the importance weights are calculated. in That is, the number of datasets. It is a hyperparameter used to control importance weights.

[0090] Step 6. Iterate through the optimization objective until training is complete. This objective function improves GRPO.

[0091] in This is a hyperparameter used to control the KL divergence so that the current policy does not deviate too much from the reference policy.

[0092] Taking a mathematical task as an example, the specific implementation process of the policy optimization method based on cue-level priority sampling and empirical smoothing provided in this embodiment is as follows: a. Prompt for assembly preparation The training objective is to improve the accuracy of the model's mathematical reasoning. The hint set contains over 7000 math problems, each problem serving as a hint unit, including information such as the question stem. The system assigns a unique index to each hint in the set and initializes the corresponding Sum Tree leaf nodes, setting all hints to the same default priority to ensure that each hint has an equal sampling opportunity in the early stages of training.

[0093] b. Suggested Sampling (based on Sum Tree) After training begins, the system enters its first round of strategy updates.

[0094] The system accesses the root node of the Sum Tree and reads the total priority value of all prompts.

[0095] Depending on the required number of hints to be extracted (e.g., 32 at a time), the system repeats the following process 32 times: generating a random value between 0 and the total priority; searching in the Sum Tree to locate the corresponding leaf node; adding the hint corresponding to the leaf node to the training samples of the current batch. In this way, the first batch of hints is fed into the model for inference and solution.

[0096] c. Model Generation Output For each hint obtained from sampling, the system generates multiple outputs (e.g., 4 different solutions), which are evaluated and scored in subsequent modules.

[0097] d. Reward Calculation and Reward Statistics Update The system evaluates each output as follows: First, it judges the final answer of the output. If the answer is correct, a positive reward is given; if it is incorrect, a negative reward is given. It checks whether the output format conforms to the expected reasoning template, such as whether it contains a clear location of the final answer. If the format is correct, an additional format reward is given; otherwise, points are deducted. For multiple outputs of the same prompt, the system calculates the average reward and its fluctuation. Then, based on these values, the system uses an exponentially weighted update strategy to smooth the average reward and the magnitude of reward changes for that prompt, thereby obtaining stable historical reward statistics at the prompt level.

[0098] e. Advantage Calculation The system compares the current reward performance with its smoothed historical reward statistics. A positive advantage value indicates significant model improvement when the current reward is significantly higher than historical rewards. A negative advantage value indicates that the prompt poses a challenge to the model or that the model has not yet learned the critical path. The advantage value is stabilized to reduce the impact of outliers.

[0099] f. Priority update suggestion (Sum Tree) Based on the latest advantage value of each suggestion, the system updates the priority of the corresponding Sum Tree leaf node. For example, suggestions with poor performance but high potential (low advantage value) will be given higher priority so that they appear more frequently in subsequent training. Suggestions with stable performance or high advantage values ​​will be given lower priority to avoid overtraining.

[0100] The update steps include locating the leaf node, writing the new priority value, and updating the parent node upwards along the path to keep the tree structure up-to-date.

[0101] g. Importance weight calculation and strategy optimization Since the prompts are sampled non-uniformly according to priority, the system calculates an importance weight for each prompt to compensate for the bias caused by different sampling probabilities.

[0102] Subsequently, the system performs a policy gradient update on the model using the smoothed advantage value, cue-level importance weights, and policy update stability constraints. The updated policy will be used for the next round of cue sampling, thus forming a complete closed loop.

[0103] In summary, compared with existing technologies, the policy optimization method based on cue-level priority sampling and experience smoothing provided in this embodiment can effectively improve the stability and sample utilization efficiency of reinforcement learning training for large language models by dynamically adjusting cue priority, smoothly updating reward statistics, and iteratively optimizing the policy model. This achieves efficient optimization of model performance and solves the problems of insufficient training stability and low sample utilization efficiency in existing reinforcement learning policy optimization methods for large language models.

[0104] To facilitate better implementation of the policy optimization method based on cue-level priority sampling and empirical smoothing according to the embodiments of this application, this application also provides a policy optimization apparatus based on cue-level priority sampling and empirical smoothing, which is based on the aforementioned policy optimization method based on cue-level priority sampling and empirical smoothing. The meanings of the terms used are the same as in the aforementioned policy optimization method based on cue-level priority sampling and empirical smoothing, and specific implementation details can be found in the description of the method embodiments.

[0105] Please see Figure 4 , Figure 4 The schematic diagram shows the structure of the policy optimization device based on cue-level priority sampling and experience smoothing provided in the embodiments of this application. Specifically, the policy optimization device based on cue-level priority sampling and experience smoothing may include a cue sampling module 201, a response reward module 202, a smoothing update module 203, a priority update module 204, a policy optimization module 205, and an iterative training module 206, which can be as follows: The cue sampling module 201 is used to sample the cue from the cue set according to the priority of each cue in the cue set to obtain the cue for the current training batch; The response reward module 202 is used to generate at least one response for each cue in the current training batch using the current policy model, and obtain the reward corresponding to each response; The smooth update module 203 is used to smooth update the reward statistics of each prompt based on the historical reward statistics of the prompt and the currently obtained reward, so as to obtain the smoothed reward statistics after the prompt is updated. Priority update module 204 is used to update the priority of the cue in the cue set based on the updated smoothed reward statistics; The strategy optimization module 205 is used to optimize and update the current strategy model based on the updated smoothed reward statistics and the bias compensation information introduced by non-uniform sampling, so as to obtain the updated strategy model. The iterative training module 206 is used to take the updated policy model as the current policy model for the new round of training, and repeatedly execute the training operation based on the updated prompt priority until the training stopping condition is met; wherein, the training operation includes the entire process from prompt sampling to policy model optimization and update.

[0106] Furthermore, in some embodiments, the prompting sampling module 201 is specifically used for: Get the current priority value of each hint in the hint set; Based on the current priority value of each prompt, calculate the probability that each prompt will be sampled. The higher the current priority value, the greater the sampling probability. Multiple prompts are randomly selected from the prompt set non-uniformly based on the sampling probability to form the current training batch.

[0107] Furthermore, in some embodiments, the response reward module 202 is specifically used to: obtain the current priority value of each prompt by querying a tree-like cumulative data structure, wherein the leaf nodes of the tree-like cumulative data structure store the priority values ​​of each prompt, and the internal nodes of the tree-like cumulative data structure store the sum of the priority values ​​of the child nodes; determine the sampling probability based on the total value of the root node of the tree-like cumulative data structure and the values ​​stored in each leaf node; generate a random number between zero and the total value of the root node; starting from the root node of the tree-like cumulative data structure, traverse down layer by layer according to the random number to a leaf node; and select the prompt corresponding to the leaf node into the current training batch.

[0108] Furthermore, in some embodiments, the smooth update module 203 is specifically used for: For a given prompt, calculate the reward statistics for the current batch of prompts based on the reward for at least one response obtained at the current time. The reward statistics include the mean of the rewards within the group and the variance of the rewards within the group. Obtain the historical smoothed reward statistics for the prompt, which include the historical smoothed mean and the historical smoothed variance; The historical smoothed reward statistics of the prompt are updated using the reward statistics of the current batch, resulting in the updated smoothed reward statistics of the prompt.

[0109] Furthermore, in some embodiments, the smooth update module 203 is specifically used for: The updated smoothed mean is obtained by taking a weighted average of the historical smoothed mean and the mean of the group rewards in the current batch. The updated smoothed variance is obtained by taking a weighted average of the historical smoothed variance and the within-group reward variance of the current batch.

[0110] Furthermore, in some embodiments, the priority update module 204 is specifically used for: For a given prompt, calculate the current advantage value of the prompt based on the smoothed mean of the current reward obtained and the smoothed reward statistics after the prompt is updated; The priority of the prompt is determined based on the advantage value; Update the priority of the prompt in the prompt set to the priority update value.

[0111] Furthermore, in some embodiments, the priority update module 204 is specifically used for: Calculate the difference between the currently obtained reward and the smoothed mean; The difference is normalized using the smoothed variance in the updated smoothed reward statistic to obtain the dominance value.

[0112] Furthermore, in some embodiments, the bias compensation information is an importance sampling weight, and the calculation method for the importance sampling weight includes: Calculate the actual sampling probability of the prompt during the prompt sampling process; The calculation provides the theoretical sampling probability under uniform sampling conditions. Calculate the ratio of the actual sampling probability to the theoretical sampling probability; The comparison values ​​are subjected to a power transformation, and the result of the power transformation is used as the importance sampling weight for the prompt.

[0113] Furthermore, in some embodiments, the reward includes a correctness reward and a format reward, then the response reward module 202 is specifically used for: Calculate a correctness bonus based on the correctness of the response; Determine whether the response conforms to the preset reasoning format specifications, and calculate the format bonus score based on the determination result; The correctness bonus and the format bonus are summed to obtain the final reward for the response.

[0114] For specific limitations regarding the policy optimization apparatus based on cue-level priority sampling and empirical smoothing, please refer to the limitations of the policy optimization method based on cue-level priority sampling and empirical smoothing mentioned above, which will not be repeated here. Each module in the aforementioned policy optimization apparatus based on cue-level priority sampling and empirical smoothing can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0115] The policy optimization device based on cue-level priority sampling and experience smoothing provided in this embodiment can effectively improve the stability and sample utilization efficiency of reinforcement learning training for large language models by dynamically adjusting the cue priority, smoothly updating reward statistics, and iteratively optimizing the policy model. This achieves efficient optimization of model performance and solves the problems of insufficient training stability and low sample utilization efficiency in existing reinforcement learning policy optimization methods for large language models.

[0116] Furthermore, embodiments of this application also provide an electronic device, such as... Figure 5 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically: The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will understand that... Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 301 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 302, and by calling data stored in the memory 302, thereby providing overall monitoring of the electronic device. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 301.

[0117] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and strategy optimization methods based on cue-level priority sampling and experience smoothing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0118] The electronic device also includes a power supply 303 that supplies power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 303 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0119] The electronic device may also include an input unit 304, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0120] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 301 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 runs the applications stored in the memory 302 to realize various functions, as follows: Based on the priority of each cue in the cue set, cue samples are obtained from the cue set for the current training batch. For each cue in the current training batch, at least one response is generated using the current policy model, and the reward corresponding to each response is obtained. For each cue, the reward statistics of the cue are smoothly updated based on the historical reward statistics and the currently obtained reward, resulting in the updated smoothed reward statistics. Based on the updated smoothed reward statistics, the priority of the cue in the cue set is updated. Based on the updated smoothed reward statistics and the bias compensation information introduced by non-uniform sampling, the current policy model is optimized and updated, resulting in the updated policy model. The updated policy model is used as the current policy model for the new round of training, and the training operation is repeated based on the updated cue priority until the training stopping condition is met. The training operation includes the entire process from cue sampling to policy model optimization and updating.

[0121] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0122] This application embodiment improves the stability and sample utilization efficiency of reinforcement learning training for large language models by dynamically adjusting the prompt priority, smoothly updating reward statistics, and iteratively optimizing the policy model. This achieves efficient optimization of model performance and solves the problems of insufficient training stability and low sample utilization efficiency in existing reinforcement learning policy optimization methods for large language models.

[0123] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0124] To this end, embodiments of this application provide a storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the policy optimization methods based on cue-level priority sampling and empirical smoothing provided in embodiments of this application. For example, the instructions can execute the following steps: Based on the priority of each cue in the cue set, cue samples are obtained from the cue set for the current training batch. For each cue in the current training batch, at least one response is generated using the current policy model, and the reward corresponding to each response is obtained. For each cue, the reward statistics of the cue are smoothly updated based on the historical reward statistics and the currently obtained reward, resulting in the updated smoothed reward statistics. Based on the updated smoothed reward statistics, the priority of the cue in the cue set is updated. Based on the updated smoothed reward statistics and the bias compensation information introduced by non-uniform sampling, the current policy model is optimized and updated, resulting in the updated policy model. The updated policy model is used as the current policy model for the new round of training, and the training operation is repeated based on the updated cue priority until the training stopping condition is met. The training operation includes the entire process from cue sampling to policy model optimization and updating.

[0125] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0126] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0127] Since the instructions stored in the storage medium can execute the steps in any of the policy optimization methods based on cue-level priority sampling and experience smoothing provided in the embodiments of this application, the beneficial effects that any of the policy optimization methods based on cue-level priority sampling and experience smoothing provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0128] The foregoing has provided a detailed description of a strategy optimization method and apparatus based on prompt-level priority sampling and empirical smoothing provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A strategy optimization method based on cue-level priority sampling and empirical smoothing, characterized in that, include: Based on the priority of each prompt in the prompt set, the prompts for the current training batch are sampled from the prompt set; For each cue in the current training batch, at least one response is generated using the current policy model, and the reward corresponding to each response is obtained; For each prompt, the reward statistics of the prompt are smoothly updated based on the historical reward statistics of the prompt and the currently obtained reward, so as to obtain the smoothed reward statistics of the prompt after the update; Based on the updated smoothed reward statistics, the priority of the prompt in the prompt set is updated; Based on the updated smoothed reward statistics and the bias compensation information introduced by non-uniform sampling, the current policy model is optimized and updated to obtain the updated policy model. The updated policy model is used as the current policy model for the new round of training, and the training operation is repeatedly executed based on the updated cue priority until the training stop condition is met; wherein, the training operation includes the entire process from cue sampling to policy model optimization and update.

2. The strategy optimization method based on cue-level priority sampling and empirical smoothing according to claim 1, characterized in that, The step of sampling the prompts for the current training batch from the prompt set according to the priority of each prompt in the prompt set includes: Obtain the current priority value of each prompt in the prompt set; Based on the current priority value of each prompt, calculate the probability that each prompt will be sampled; the higher the current priority value, the greater the sampling probability. Multiple prompts are randomly selected non-uniformly from the prompt set according to the sampling probability to form the current training batch.

3. The strategy optimization method based on cue-level priority sampling and empirical smoothing according to claim 2, characterized in that, The step of obtaining the current priority value of each prompt in the prompt set includes: obtaining the current priority value of each prompt by querying a tree-like cumulative data structure, wherein the leaf nodes of the tree-like cumulative data structure store the priority values ​​of each prompt, and the internal nodes of the tree-like cumulative data structure store the sum of the priority values ​​of the child nodes; The step of calculating the probability of each prompt being sampled based on the current priority value of each prompt includes: determining the sampling probability based on the total value of the root node of the tree-like cumulative data structure and the values ​​stored in each leaf node; The step of non-uniformly and randomly selecting multiple prompts from the prompt set according to the sampling probability to form the current training batch includes: generating a random number between zero and the total value of the root node; starting from the root node of the tree-like cumulative data structure, traversing down layer by layer according to the random number to a leaf node; and selecting the prompts corresponding to the leaf nodes into the current training batch.

4. The strategy optimization method based on cue-level priority sampling and empirical smoothing according to claim 1, characterized in that, For each prompt, the reward statistics of the prompt are smoothly updated based on the historical reward statistics and the currently obtained reward, resulting in the smoothed reward statistics of the prompt after the update. This includes: For a given prompt, based on the reward of at least one response currently obtained for the prompt, calculate the reward statistic for the current batch of the prompt, wherein the reward statistic includes the mean of the within-group reward and the variance of the within-group reward; Obtain the historical smoothed reward statistics of the prompt, wherein the historical smoothed reward statistics include the historical smoothed mean and the historical smoothed variance; The historical smoothed reward statistics of the prompt are updated using the reward statistics of the current batch of the prompt, so as to obtain the updated smoothed reward statistics of the prompt.

5. The strategy optimization method based on cue-level priority sampling and empirical smoothing according to claim 4, characterized in that, The step of updating the historical smoothed reward statistics of the prompt using the reward statistics of the current batch of the prompt to obtain the updated smoothed reward statistics of the prompt includes: The updated smoothed mean is obtained by taking a weighted average of the historical smoothed mean and the mean of the intra-group reward of the current batch. The updated smoothed variance is obtained by taking a weighted average of the historical smoothed variance and the within-group reward variance of the current batch.

6. The strategy optimization method based on cue-level priority sampling and empirical smoothing according to claim 1, characterized in that, Updating the priority of the prompt in the prompt set based on the updated smoothed reward statistics includes: For a given prompt, calculate the current advantage value of the prompt based on the smoothed mean of the currently obtained reward and the smoothed reward statistics after the prompt is updated; The priority update value of the prompt is determined based on the advantage value; Update the priority of the prompt in the prompt set to the priority update value.

7. The strategy optimization method based on cue-level priority sampling and empirical smoothing according to claim 6, characterized in that, For a given prompt, the method of calculating the current advantage value of the prompt based on the smoothed mean of the currently obtained reward and the smoothed reward statistics after the prompt is updated includes: Calculate the difference between the currently obtained reward and the smoothed mean; The difference is normalized using the smoothed variance in the updated smoothed reward statistic to obtain the advantage value.

8. The strategy optimization method based on cue-level priority sampling and empirical smoothing according to claim 1, characterized in that, The deviation compensation information is an importance sampling weight, and the calculation method of the importance sampling weight includes: Calculate the actual sampling probability of the prompt during the prompt sampling process; Calculate the theoretical sampling probability of the prompt under uniform sampling conditions; Calculate the ratio of the actual sampling probability to the theoretical sampling probability; The ratio is subjected to a power transformation, and the result of the power transformation is used as the importance sampling weight of the prompt.

9. The strategy optimization method based on cue-level priority sampling and empirical smoothing according to claim 1, characterized in that, The rewards include correctness rewards and format rewards. Therefore, for each cue in the current training batch, generating at least one response using the current policy model and obtaining the reward corresponding to each response includes: Calculate a correctness bonus score based on the correctness of the response; Determine whether the response conforms to the preset reasoning format specification, and calculate the format reward score based on the determination result; The correctness reward score and the format reward score are summed to obtain the final reward corresponding to the response.

10. A strategy optimization device based on cue-level priority sampling and empirical smoothing, characterized in that, include: The prompt sampling module is used to sample the prompts of the current training batch from the prompt set according to the priority of each prompt in the prompt set; The response reward module is used to generate at least one response for each cue in the current training batch using the current policy model, and to obtain the reward corresponding to each response. The smooth update module is used to smoothly update the reward statistics of each prompt based on the historical reward statistics of the prompt and the currently obtained reward, so as to obtain the smoothed reward statistics of the prompt after the update. The priority update module is used to update the priority of the prompt in the prompt set based on the updated smoothed reward statistics; The strategy optimization module is used to optimize and update the current strategy model based on the updated smoothed reward statistics and the bias compensation information introduced by non-uniform sampling, so as to obtain the updated strategy model. The iterative training module is used to take the updated policy model as the current policy model for the new round of training, and repeatedly execute the training operation based on the updated prompt priority until the training stop condition is met; wherein, the training operation includes the entire process from prompt sampling to policy model optimization and update.

Images