Speculative prefix reuse method for reinforcement learning training with verifiable rewards

Abstract

This invention proposes a speculative prefix reuse method for verifiable reward reinforcement learning training, comprising: obtaining the cue set of the current training batch and the old responses generated in the previous training iteration, wherein the old responses include multiple old word sequences, each old word sequence carrying the first logarithmic probability and length information of each word under the old policy; calculating the second logarithmic probability of each word under the current policy based on each cue in the cue set and the corresponding old word sequence; judging each word according to the first logarithmic probability and the second logarithmic probability in the order of the old word sequence; if the judgment result for each word is acceptance, the old word sequence is directly reused; if a rejection word appears during the judgment process, the judgment is stopped to obtain a reusable prefix, and the model is called to generate the suffix corresponding to each reusable prefix to obtain a new response, thereby reducing the overhead of the generation sampling stage and improving the response efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and machine learning, in particular to a speculative prefix reuse method for RLVR training, a computer readable storage medium and a computer device. BACKGROUND

[0002] In the related art, with the popularity of large language models in mathematical reasoning, tool use and instruction following, reinforcement learning based on verifiable rewards (RLVR) has become an important path for long response training by obtaining deterministic rewards through programmatic verification. The typical training process includes performing prompt generation sampling (rollout) to output responses, running verifiable programs to calculate rewards, and updating the policy accordingly.

[0003] Existing efficiency improvements are mainly focused on adjusting targets / constraints, data grouping and sorting, and system engineering optimization, etc. "sequence / batch" level measures lack direct control over the number of new tokens generated within a single response (i.e. the main source of time consumption for rollout). On the other hand, speculative decoding can significantly reduce latency in the reasoning phase through a draft-and-verify mechanism, but the original design is for a single reasoning request: if directly copied to the training phase, it usually requires an additional draft model or multiple generation calls, which brings implementation complexity and does not match the memory and scheduling constraints of RLVR with large batch and long response. Therefore, there is an urgent need for an overhead reduction mechanism that is available during training, token-level effective, and compatible with high-throughput engines. SUMMARY

[0004] The present application aims to at least solve one of the above technical problems in the technical field. To this end, one object of the present application is to provide a speculative prefix reuse method for RLVR training, which uses the old response of the previous training iteration as an implicit draft, only performs consistency evaluation and acceptance determination with the current policy before generation, and obtains reusable continuous prefixes; the remaining suffix is generated and spliced in a batch continuation writing to output, thereby reducing the overhead of the generation sampling stage and improving response efficiency.

[0005] A second object of the present application is to provide a computer readable storage medium.

[0006] A third object of the present application is to provide a computer device.

[0007] To achieve the above objectives, a first aspect of the present invention proposes a speculative prefix reuse method for verifiable reward reinforcement learning training, comprising the following steps: obtaining the cue set of the current training batch and the old responses generated in the previous training iteration, wherein the old responses include multiple old lexical sequences, each old lexical sequence carrying the first logarithmic probability and length information corresponding to each lexical under the old policy; calculating the second logarithmic probability corresponding to each lexical under the current policy based on each cue in the cue set and the corresponding old lexical sequence; judging each lexical according to the first logarithmic probability and the second logarithmic probability corresponding to each lexical in the order of the old lexical sequence; if the judgment result for each lexical is acceptance, then the old lexical sequence is directly reused; if a rejection lexical appears during the judgment process, the judgment is stopped to obtain a reusable prefix, and the model is called to generate a suffix corresponding to each reusable prefix, and the reusable prefix and the generated suffix are concatenated to obtain a new response; thereby reducing the overhead of the generation sampling stage and improving the response efficiency.

[0008] Furthermore, the speculative prefix reuse method for verifiable reward reinforcement learning training proposed in the above embodiments of the present invention may also have the following additional technical features:

[0009] Optionally, it also includes: inputting all the new responses obtained into a verifiable reward function to obtain a reward value, calculating the advantage and updating the policy parameters to complete a training iteration.

[0010] Optionally, each word is judged according to the first logarithmic probability and the second logarithmic probability corresponding to each word in the order of the old word sequence, including: obtaining the acceptance probability corresponding to each word based on the first logarithmic probability and the second logarithmic probability corresponding to each word; sampling a random number, wherein the value of the random number is between 0 and 1; if the acceptance probability corresponding to the word is greater than or equal to the random number, it is judged as acceptance, and the judgment of the next word continues; if the acceptance probability corresponding to the word is less than the random number, it is judged as rejection, and the judgment of the next word stops.

[0011] Alternatively, the acceptance probability for each lexical unit can be obtained using the following formula:

[0012]

[0013] in, Indicates the first In the nth word sequence The acceptance probability corresponding to each lexical unit Indicates the first In the nth word sequence The second logarithmic probability corresponding to each lexical unit Indicates the first In the nth word sequence The first logarithmic probability corresponding to each lexical unit; This represents the tolerance parameter.

[0014] Optionally, the model is invoked to generate the suffix corresponding to each reusable prefix, including: concatenating each prompt in all prompts with the corresponding reusable prefix to form an input context, and obtaining the length of the corresponding suffix to be continued; inputting the input context and the suffix length into the sampling engine for parallel decoding to complete batch continuation and obtain the continuation suffix corresponding to each reusable prefix.

[0015] To achieve the above objectives, a second aspect of the present invention provides a computer-readable storage medium storing a speculative prefix reuse program for verifiable reward reinforcement learning training, which, when executed by a processor, implements the speculative prefix reuse method for verifiable reward reinforcement learning training as described above.

[0016] To achieve the above objectives, a third aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the speculative prefix reuse method for verifiable reward reinforcement learning training as described above. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a speculative prefix reuse method for verifiable reward reinforcement learning training according to an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram illustrating the acceleration of speculative reinforcement learning according to an embodiment of the present invention.

[0019] Figure 3 This is a performance illustration of speculative reinforcement learning according to an embodiment of the present invention, wherein, Figure 3 (a) is a schematic diagram of mathematical reasoning performance. Figure 3 (b) is a schematic diagram of the training Rollout time. Detailed Implementation

[0020] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0021] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0022] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0023] refer to Figure 1 As shown, the speculative prefix reuse method for verifiable reward reinforcement learning training according to an embodiment of the present invention includes the following steps:

[0024] S101, obtain the cue set of the current training batch and the old response generated in the previous training iteration, wherein the old response includes multiple old word sequences, each old word sequence carrying the first log probability and length information of each word under the old policy.

[0025] In other words, during the sampling phase, the response cached in the previous iteration is directly reused as a "speculation draft". Specifically, given a hint... The old response generated in the previous training iteration is represented as:

[0026]

[0027] Among them, length , This indicates the maximum response length and includes the old strategy. The token-by-token logarithmic probability, i.e., the first logarithmic probability, is expressed as:

[0028]

[0029] S102, calculate the second logarithmic probability of each word under the current policy based on each prompt in the prompt set and the corresponding old word sequence.

[0030] In other words, under the current strategy, the token-by-token log probability, i.e. the second log probability, is calculated once for each old word sequence in the old response.

[0031] Specifically, in the current iteration The same sequence can be computed under the new strategy. Logarithmic probability:

[0032]

[0033] It should be noted that obtaining the first and second log probabilities is for the purpose of comparing the old and new probabilities later, determining whether to "accept" each token, and extracting the longest continuous reusable prefix; if the entire old lexical sequence is accepted, it is reused directly; otherwise, the model is called to generate a suffix after the rejection point.

[0034] S103, each word is determined according to the first logarithmic probability and the second logarithmic probability corresponding to each word in the order of the old word sequence.

[0035] In other words, starting from the beginning of the word sequence, each token is scanned according to the decision rules to determine whether each word is accepted or rejected.

[0036] As an example, each word is judged according to the first logarithmic probability and the second logarithmic probability corresponding to each word in the order of the old word sequence, including: obtaining the acceptance probability corresponding to each word based on the first logarithmic probability and the second logarithmic probability corresponding to each word; sampling a random number, wherein the value of the random number is between 0 and 1; if the acceptance probability corresponding to the word is greater than or equal to the random number, it is judged as acceptance, and the judgment of the next word continues; if the acceptance probability corresponding to the word is less than the random number, it is judged as rejection, and the judgment of the next word stops.

[0037] It should be noted that if the word sequence encounters its first rejected word during the determination process, the rejection position will be used as the prefix splitting point.

[0038] The acceptance probability for each lexical unit is obtained using the following formula:

[0039]

[0040] in, Indicates the first In the nth word sequence The acceptance probability corresponding to each lexical unit Indicates the first In the nth word sequence The second logarithmic probability corresponding to each lexical unit Indicates the first In the nth word sequence The first logarithmic probability corresponding to each lexical unit; This represents the tolerance parameter.

[0041] Specifically, in order to improve the reuse rate, this application introduces a tolerance parameter. After obtaining the above acceptance probabilities, sampling was performed. ,like Then accept the token.

[0042] when At that point, the rules degenerate into naive speculative decoding.

[0043] along with Increasing the size increases the probability of acceptance, and a longer prefix results in faster speed;

[0044] but If the value is too large, almost all tokens will be accepted, which will inhibit exploration and impair training stability.

[0045] Practice has shown that, The optimal speed-accuracy tradeoff is usually achieved in the range of 0.3–0.7: rollout speedup can be 2–3×, with almost no impact on accuracy.

[0046] In other words, a tolerance parameter is introduced. Additive relaxation is applied to the difference to form a single knob with a clear function, which facilitates a stable trade-off between speed and exploration.

[0047] S104 If the judgment result for each word element is "accept", then the old word element sequence is directly reused.

[0048] S105, if a rejected term appears during the judgment process, the judgment is stopped to obtain a reusable prefix, and the model is called to generate a suffix corresponding to each reusable prefix, and the reusable prefix and the generated suffix are concatenated to obtain a new response.

[0049] It should be noted that after determining the old word sequence, it is further classified according to the determination result. Specifically, after obtaining the prefix segmentation point... Then, the samples were divided into two categories:

[0050] (1) REUSE subset: if In this case, the entire old response can be completely reused without needing to be generated;

[0051] (2) CONT subset: if If the prefix is ​​used, then only the suffix needs to be continued.

[0052] As an example, the model is invoked to generate the suffix corresponding to each reusable prefix, including: concatenating each prompt in all prompts with the corresponding reusable prefix to form an input context, and obtaining the length of the corresponding suffix to be continued; inputting the input context and suffix length into the sampling engine for parallel decoding to complete batch continuation, and obtaining the continuation suffix corresponding to each reusable prefix.

[0053] Specifically, construct the context for the CONT subset:

[0054]

[0055] And set the maximum generation length:

[0056]

[0057] Then, through a single batch call:

[0058]

[0059] Generate the suffixes for all CONT samples and then concatenate them:

[0060] like Output ;

[0061] otherwise .

[0062] Therefore, the validity of this application can be ensured. Figure 2 The SPEC-RL method shown is fully compatible with existing high-throughput engines and requires only one generation call.

[0063] In addition, after obtaining all new responses, all new responses are input into a verifiable reward function to obtain reward values, calculate the advantage, and update the policy parameters to complete one training iteration.

[0064] In other words, the new response is input into the standard RLVR process, a verifiable procedure is run to obtain a reward, the advantage is calculated, and the policy parameters are updated. Monitoring metrics (such as average prefix length, percentage of full reuse / "skip rate", generation time) are recorded for subsequent parameter tuning or... Strategic adjustments.

[0065] In summary, the speculative prefix reuse method for verifiable reward reinforcement learning training according to embodiments of the present invention utilizes the cached old responses from the previous training round / iteration as implicit drafts, performs a consistency evaluation and acceptance decision with the current policy only once before generation, and obtains reusable continuous prefixes; the remaining suffixes are generated and concatenated in a batch continuation; to achieve an interpretable tradeoff between speed and exploration, a tolerance parameter is introduced. The reuse intensity is controlled; the overall approach integrates the existing reinforcement learning pipeline with verifiable rewards in three stages: presampling, sampling, and post-processing, ensuring compatibility with high-throughput inference engines. As a result, without changing the reinforcement learning objectives, rewards, and update rules of verifiable rewards, or introducing independent draft models, the number of new lexical units that need to be generated for each response is directly reduced, thereby significantly compressing the generation and sampling time and computational overhead during the training phase, while also taking into account training stability and downstream task performance.

[0066] As another specific example, taking a single training iteration of SPEC-RL as an example, the input includes the cue P and the old response. (Including cached log-prob and length) Current strategy tolerance , LLM engine, maximum response length R.

[0067] 1. Pre-sampling stage

[0068] (1) Calculation

[0069] (2) Calculation

[0070] 2. Speculative Decoding

[0071] (1) For each token t: if or , Accept the token t

[0072] (2) Let =The first rejection position, if there is no rejection then

[0073] 3. Batch Division

[0074] (1)

[0075] (2)

[0076] (3) Constructing context for CONT samples and set

[0077] 4. Rollout

[0078] (1) Call Get suffix

[0079] like ,but

[0080] otherwise

[0081] 5. RLVR Update

[0082] (1) Calculate rewards and advantages based on Y, and update the strategy. .

[0083] It should be noted that, as Figure 3 (a) and Figure 3As shown in (b), SPEC-RL reduces rollout time by 2–3 times while maintaining or even slightly improving overall mathematical reasoning performance. This demonstrates that the method in this application accelerates training without sacrificing accuracy. Specific beneficial effects include token-level reduction: directly reducing the number of newly generated tokens within a single response, significantly reducing the overhead of the generation and sampling phase; and serving as a lightweight modification to the rollout channel without altering the RLVR objective / reward / optimizer, resulting in low engineering integration costs; tolerance parameters... It offers a single knob for a stable trade-off between speed and exploration; it works in conjunction with high-throughput generation engines to adapt to various strategy optimizations (such as GRPO, PPO, DAPO) and multi-scale models; the new pre / post-processing only involves one log probability evaluation and lightweight stitching, with negligible generation overhead.

[0084] In addition, the present invention also proposes a speculative prefix reuse device for verifiable reward reinforcement learning training, which is used to implement the speculative prefix reuse method for verifiable reward reinforcement learning training as described above. The device includes a cache management module, a probability calculation module, an acceptance decision module, a context construction module, a batch continuation module, a splicing module, a reward update module, and a monitoring and scheduling module.

[0085] The cache management module stores old responses, old logarithmic probabilities, lengths, and alignment keys; the probability calculation module evaluates the logarithmic probability of old responses token-by-token under the current policy; and the acceptance decision module determines the probability based on the difference and... The system performs token-by-token acceptance and outputs prefix split points and REUSE / CONT markers; the context building module assembles "hint + prefix" contexts for CONT sub-batches and sets heterogeneous continuation budgets; the batch continuation module initiates a batch generation for CONT sub-batches; the splicing module completes REUSE passthrough and CONT prefix and suffix splicing and mask patching; and the reward update module interfaces with the RLVR training framework to complete reward calculation, advantage estimation, and policy update.

[0086] In addition, the present invention also proposes a computer-readable storage medium storing a speculative prefix reuse program for verifiable reward reinforcement learning training, which, when executed by a processor, implements the speculative prefix reuse method for verifiable reward reinforcement learning training as described above.

[0087] In addition, this invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the speculative prefix reuse method for verifiable reward reinforcement learning training as described above.

[0088] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0093] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0094] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0095] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0096] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0097] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0098] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0099] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A speculative prefix-multiplexing method for verifiable reward reinforcement learning training, the method comprising: receiving a first input and a second input; generating a first prefix and a second prefix; generating a first output and a second output; and outputting the first output and the second output. The method comprises the following steps: obtaining a prompt set of a current training batch and an old response generated in a previous training iteration, wherein the old response comprises a plurality of old token sequences, each old token sequence carrying a first log probability of each token under an old policy and length information; calculating a second log probability of each token under a current policy according to each prompt in the prompt set and the corresponding old token sequence; determining each token according to the first log probability of each token and the second log probability of each token in old token sequence order; if the determination result of each token is all accept, directly reusing the old token sequence; if a rejected token appears in the determination process, stopping the determination to obtain a reusable prefix, calling a model to generate a suffix corresponding to each reusable prefix, and splicing the reusable prefix and the generated suffix to obtain a new response; inputting all the obtained new responses into a verifiable reward function to obtain a reward value, calculating an advantage, and updating a policy parameter to complete a training iteration; wherein determining each token according to the first log probability of each token and the second log probability of each token in old token sequence order comprises: obtaining an acceptance probability of each token according to the first log probability of each token and the second log probability of each token; sampling a random number, wherein the random number is in a range of 0 to 1; if the acceptance probability of the token is greater than or equal to the random number, determining as accept and continuing the determination of the next token; if the acceptance probability of the token is less than the random number, determining as reject and stopping the determination of the next token; wherein the acceptance probability of each token is obtained according to the following formula: in, Indicates the first In the nth word sequence The acceptance probability corresponding to each lexical unit Indicates the first In the nth word sequence The second logarithmic probability corresponding to each lexical unit Indicates the first In the nth word sequence The first logarithmic probability corresponding to each lexical unit; This represents the tolerance parameter.

2. The speculative prefix-multiplication method oriented towards verifiable reward reinforcement learning training of claim 1, wherein, calling a model to generate a suffix corresponding to each reusable prefix comprises: splicing each prompt in all prompts with the corresponding reusable prefix into an input context, and obtaining a length of a suffix to be continued; inputting the input context and the length of the suffix into a sampling engine for parallel decoding to complete batch continuation and obtain a continuation suffix corresponding to each reusable prefix.

3. A computer-readable storage medium, characterized in that, A speculative prefix reuse program for verifiable reward reinforcement learning training is stored thereon, and the speculative prefix reuse program for verifiable reward reinforcement learning training is executed by a processor to implement the speculative prefix reuse method for verifiable reward reinforcement learning training according to any one of claims 1-2.

4. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the speculative prefix reuse method for verifiable reward reinforcement learning training according to any one of claims 1-2.

Images