Reinforcement learning training optimization method and apparatus

Abstract

The application relates to a reinforcement learning training optimization method and device. The method comprises the following steps: obtaining a text reasoning request in a batch, and distributing the text reasoning request in the batch to at least one calculation unit for reasoning output; in the case that the reasoning output completion progress of the text reasoning request in the batch reaches a preset output progress, truncating the unfinished model output in each calculation unit; and updating the model parameters of a model used for reasoning based on the reasoning output result and a reference correct result in the batch. The method can avoid the influence of long-time-consuming requests on the overall training efficiency, significantly improve the resource utilization rate and model training efficiency of the calculation unit reinforcement learning training, and balance the low cost of the training process and the high efficiency of the parameter updating.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a reinforcement learning training optimization method and apparatus. Background Technology

[0002] In the reinforcement learning training process of text models, there is a significant imbalance in the length of the input and output data: the length of the input data for each request is relatively limited, while the length of the corresponding inference output varies greatly. In conventional training deployments, a batch of training text inference requests is distributed across at least one computing unit for parallel data processing. Since each computing unit must wait for the longest inference output in its batch to complete before it can finish its task, even if some computing units have completed their own inference tasks, they still have to wait for other units that have not yet completed their tasks, resulting in a serious waste of computing resources.

[0003] Furthermore, in the time-consuming components of reinforcement learning training, output time accounts for the majority of the total training time due to the shorter input and longer output. The core of this output time is concentrated in the memory access operations of the attention mechanism. Current training processes lack effective means to control the uneven load during the output phase, failing to reasonably avoid low resource utilization caused by long waiting times, and failing to improve output efficiency by optimizing memory access-related processing logic. Ultimately, this results in low overall efficiency in reinforcement learning training, which has become a key technical bottleneck restricting performance improvement in reinforcement learning training. Summary of the Invention

[0004] Therefore, it is necessary to provide a reinforcement learning training optimization method and apparatus to address the aforementioned technical problems.

[0005] Firstly, this application provides a reinforcement learning training optimization method, including:

[0006] Obtain the text reasoning requests for this batch, and assign the text reasoning requests for this batch to at least one computing unit for reasoning output;

[0007] If the reasoning output completion progress in the text reasoning request of this batch reaches the preset output progress, the unfinished model output in each computing unit is truncated.

[0008] Based on the inference output results already completed in this batch and the reference correct results, update the model parameters of the model used for inference.

[0009] Secondly, this application also provides a reinforcement learning training optimization device, the device comprising:

[0010] The acquisition module is used to acquire the text reasoning requests in this batch and assign the text reasoning requests in this batch to at least one computing unit for reasoning output;

[0011] The inference output control module is used to truncate the unfinished model output in each computing unit when the inference output completion progress in the text inference requests of this batch reaches the preset output progress.

[0012] The model update module is used to update the model parameters of the model used for inference based on the inference output results already completed in this batch and the reference correct results.

[0013] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0014] Obtain the text reasoning requests for this batch, and assign the text reasoning requests for this batch to at least one computing unit for reasoning output;

[0015] If the reasoning output completion progress in the text reasoning request of this batch reaches the preset output progress, the unfinished model output in each computing unit is truncated.

[0016] Based on the inference output results already completed in this batch and the reference correct results, update the model parameters of the model used for inference.

[0017] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps described in the first aspect.

[0018] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps described in the first aspect.

[0019] The aforementioned reinforcement learning training optimization method and apparatus can acquire text inference requests in the current batch and allocate the requests to at least one computing unit for inference output. When the inference output completion progress of the current batch of text inference requests reaches a preset output progress, the unfinished model output in each computing unit is truncated. Based on the completed inference output results and the reference correct results in the current batch, the model parameters of the model used for inference are updated. This scheme, by allocating text inference requests to at least one computing unit for parallel inference, and truncating unfinished model output based on a preset output progress, allows for updating model parameters based on completed inference results without waiting for all computing units to complete their inference tasks. This effectively solves the problems in the background technology where excessively long output lengths of some inference requests lead to wasted computing power in many computing units, long training batch times, and low utilization of computing resources. Simultaneously, this mechanism allows the model to iteratively optimize based on completed effective inference data in a timely manner, avoiding the impact of long-running requests on overall training efficiency, significantly improving the resource utilization and model training efficiency of the computing unit in reinforcement learning training, and balancing low training cost with high parameter update efficiency. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a reinforcement learning training optimization method in one embodiment;

[0022] Figure 2 This is a flowchart illustrating a method for obtaining text reasoning requests in the current batch, as shown in one embodiment.

[0023] Figure 3 This is a schematic diagram illustrating the time distribution of pre-filling before and after sorting training data requests by input length from longest to shortest;

[0024] Figure 4 A flowchart illustrating a method for allocating text reasoning requests in one embodiment;

[0025] Figure 5 This is a flowchart illustrating a quantization process in the inference output process of one embodiment.

[0026] Figure 6 This is a structural block diagram of a reinforcement learning training optimization device in one embodiment;

[0027] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0029] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0030] Currently, the training process lacks effective means to control the unbalanced load in the output stage, cannot reasonably avoid resource waste caused by long waiting times, and has failed to improve output efficiency by optimizing memory access-related processing logic. Ultimately, this results in low resource utilization, high training costs, low overall efficiency of model training, and long training time, which has become a key technical bottleneck restricting the improvement of reinforcement learning training performance.

[0031] The reinforcement learning training optimization method provided in this application can update model parameters based on the completed inference results without waiting for all inference tasks of all computing units to be completed. This effectively solves the problems of a large number of computing units waiting idly, long training batch time, and low utilization of computing resources caused by some inference request output lengths being too long.

[0032] The execution entity of the reinforcement learning training optimization method provided in this application embodiment can be a reinforcement learning training optimization device or a computer device. The reinforcement learning training optimization device can be a functional module or functional entity within the computer device.

[0033] The computer device in this application refers to a hardware carrier that has data processing, computing and resource scheduling capabilities and can execute reinforcement learning training optimization methods. It can be a single independent high-performance computing device or a distributed cluster composed of multiple computing devices. From a hardware perspective, this computer device includes at least a processor, memory, and interfaces for communicating with external or internal components. The processor can be a general-purpose or special-purpose computing chip such as a multi-core central processing unit (CPU), graphics processing unit (GPU), or tensor processing unit (TPU), used to handle core tasks such as allocating text inference requests, performing model inference calculations, pre-filling key-value caches (KV), quantization parameter calculations, and updating model parameters. The memory includes volatile storage media such as RAM and video memory, and non-volatile storage media such as hard disks and solid-state drives (SSDs), used to store text inference request data to be processed, model parameters, KV cache data, output data before truncation, and intermediate data during training. The communication interface supports data transmission between computing units within the device or between nodes in a distributed cluster, ensuring efficient execution of operations such as inference request allocation and continuation of unfinished tasks. In terms of device form, computer devices in a standalone scenario can be high-performance workstations equipped with one or more GPUs or artificial intelligence (AI) training servers; computer devices in a distributed scenario can be computing clusters composed of multiple AI servers interconnected through a high-speed network, or elastic computing instances deployed based on public or private clouds, which can dynamically adjust the number of computing units according to the computing power requirements of training tasks and adapt to the batch processing needs of text inference requests of different scales.

[0034] In one exemplary embodiment, such as Figure 1 As shown, a reinforcement learning training optimization method is provided. This method can be applied to the aforementioned computer device and includes, but is not limited to, the following steps:

[0035] 101. Obtain the text reasoning requests for this batch.

[0036] In the process of obtaining the text reasoning requests for this batch, the initial text reasoning requests for this batch can be collected first, and then the initial text reasoning requests for this batch can be pre-filled sequentially in the pre-filling stage. After that, the pre-filled text reasoning requests are determined as the text reasoning requests for this batch.

[0037] 102. Assign at least one computing unit to the text reasoning requests in this batch for reasoning output.

[0038] Among these features, after obtaining the pre-filled text inference requests for this batch, each text inference request can be evenly distributed to at least one computing unit for inference output.

[0039] In this embodiment, the number of computing units can be determined based on the size of the parameters of the target training model, the training time requirements, and the computing power and storage resources configured for each computing unit. If the number of parameters of the target training model is very small, there are no requirements for training time, and the computing power and storage resources configured for each computing unit are large enough, one computing unit can be used for model training; when there are higher requirements, multiple computing units can be used to perform parallel training in a data parallel and tensor parallel manner.

[0040] For example, firstly, based on the number of text inference requests and the number of computing units in the deployment instance, the preprocessed text inference requests can be evenly distributed to each computing unit according to the load balancing principle. Each computing unit can independently handle a portion of the allocated text inference requests. After the distribution is completed, each computing unit can synchronously start the inference output process, completing the decoding calculation and token generation for the requests in its own sub-task set. Each computing unit can perform decoding calculation and token generation in parallel for multiple requests in its own sub-task set, expanding the parallel inference batch, reducing memory accesses, and improving computing resource utilization and model training efficiency. Simultaneously, the inference progress is fed back in real time, providing data support for the progress judgment in subsequent step 103.

[0041] 103. Determine whether the reasoning output completion progress has reached the preset output progress.

[0042] The aforementioned inference output completion progress refers to the inference output completion progress of the text inference requests in this batch.

[0043] If the inference output completion progress in this batch of text inference requests reaches the preset output progress, the following step 104 can be executed, that is, the unfinished model output in each computing unit is truncated. If the inference output completion progress in this batch of text inference requests does not reach the preset output progress, the above step 102 is executed to perform inference output.

[0044] 104. Truncate any incomplete model outputs in each computational unit.

[0045] The aforementioned preset output progress may include: preset inference ratio and / or preset quantity.

[0046] In some implementations, if the proportion of completed inference output in at least one computational unit is greater than a first preset inference proportion, the incomplete model output in computational units that have not yet finished inference output can be truncated.

[0047] The proportion of computational units that have completed inference output in at least one computational unit can refer to the proportion of the number of computational units that have completed inference output to the total number of computational units; or it can refer to the proportion of the number of text inference requests that have completed inference output in each computational unit to the total number of text inference requests allocated in its corresponding subtask set.

[0048] For example, a first preset inference ratio can be set to 80%. A reinforcement learning training deployment instance is configured with 10 GPU computing units, and each computing unit processes the assigned text inference requests in parallel. When 8 out of the 10 computing units have completed all the inference request output tasks they are responsible for, the inference output completion ratio of the computing units reaches 80%. At this time, the inference output completion ratio is greater than the first preset inference ratio, triggering a truncation mechanism. The unfinished model outputs being processed in the remaining 2 computing units that have not completed inference output are truncated, and the unfinished inference requests in these 2 computing units are removed from the current training instance, waiting to be merged with the next batch of text inference requests before re-inference calculation. Alternatively, a first preset inference ratio can be set to 80%. A reinforcement learning training deployment instance is configured with 10 GPU computing units. Each computing unit is assigned 100 text inference requests in its corresponding subtask. Each computing unit processes the assigned text inference requests in parallel. When 80 inference requests have been completed in each computing unit, the inference output completion ratio of each computing unit reaches 80%. At this point, the inference output completion ratio is greater than the first preset inference ratio, triggering a truncation mechanism. The remaining 20 inference requests in each computing unit that have not completed inference output are truncated, and these 200 incomplete inference requests are removed from the current training instance and wait to be merged with the next batch of text inference requests before recalculating the inference.

[0049] In the above implementation methods, the overall completion rate of the computing units can be used as the truncation trigger condition. This allows for macro-level control of inference progress, preventing the entire training cluster from entering a prolonged waiting state due to a few computing units handling excessively long output requests. By promptly truncating and evicting computing units with incomplete tasks, the computing resources of these units can be quickly released, reducing the computing power wastage rate in multi-computation unit parallel training scenarios and improving the resource utilization efficiency of the entire training cluster. Alternatively, the completion rate of each computing unit can be used as the truncation trigger condition. This allows for micro-level control of inference progress, preventing the computing power utilization rate of each computing unit from remaining at a low level for an extended period due to a few excessively long output requests. By promptly truncating and evicting incomplete inference requests, the computing power resources of the computing units can be quickly released, improving the computing power utilization rate of each computing unit in parallel inference scenarios and enhancing the resource utilization efficiency throughout the entire model training period.

[0050] In some implementations, if the number of completed inference outputs in the current batch of text inference requests exceeds a preset number, the unfinished model outputs in each computing unit are truncated.

[0051] The number of text inference requests that have completed inference output in this batch refers to the number of text inference requests that have completed inference output in this batch.

[0052] For example, suppose the total number of text inference requests in a certain reinforcement learning training deployment instance is 8000, and the preset number is 6000. During the training process, each computing unit advances the 8000 inference output tasks in parallel. When the cumulative number of completed inference output requests reaches 6000, a truncation mechanism is triggered. The model output corresponding to the remaining 2000 text inference requests that have not yet been completed in all computing units is truncated. These 2000 incomplete text inference requests are removed from the current training instance and will be merged with the next batch of text inference requests before inference training is restarted.

[0053] In the above implementation, the number of completed text inference requests is used as the quantization truncation threshold. The triggering condition is easy to quantify and monitor. By setting a reasonable preset number, the effective parallel duration of a single batch of training can be accurately controlled, avoiding the problem of a sharp drop in parallelism in the later stages of training due to the drag from a small number of excessively long requests. At the same time, setting a fixed threshold facilitates migration and adaptation to training tasks of different scales, and can stably ensure the high efficiency of parallel data processing during training. Under the premise of ensuring the validity of completed request data, the impact of invalid waiting time on training efficiency is minimized.

[0054] In some implementations, if the proportion of completed inference output in the current batch of text inference requests exceeds the second preset inference proportion, the unfinished model output in each computing unit is truncated.

[0055] The percentage of completed inference outputs in this batch of text inference requests refers to the proportion of completed inference outputs in this batch of text inference requests to the total number of text inference requests in this batch.

[0056] For example, the second preset inference ratio can be set to 75%, and the total number of text inference requests in this batch of a certain reinforcement learning training deployment instance is 4000. During the parallel inference process of each computing unit, when the number of text inference requests that have completed inference output reaches 3000, the completion ratio reaches 75% and meets the second preset inference ratio requirement, the truncation mechanism is triggered. The remaining 1000 models that have not completed inference output in all computing units are truncated, and these 1000 incomplete requests are removed from the current training instance and wait to be merged with the next batch of text inference requests before recalculating the inference.

[0057] In the above implementation, the overall completion rate of text inference requests in this batch is used as the truncation basis. Compared with the completion rate of computing units, this is more in line with the global progress of a single batch training task and can flexibly adapt to training scenarios with different numbers of computing units and different request allocation strategies. By setting a reasonable second preset inference ratio, a balance can be achieved between the parallel efficiency and recalculation overhead of the training task. This avoids the problem of too many recalculation requests and excessive additional overhead caused by premature truncation, and also avoids the defect of continuously declining resource utilization caused by late truncation. This ensures that the single batch training task completes the main amount of computation within the efficient parallel range, and significantly improves the overall resource utilization efficiency and iteration speed of training.

[0058] 105. Based on the inference output results and reference correct results already completed in this batch, update the model parameters of the model used for inference.

[0059] Among these, iterative optimization of the model can be achieved through a reinforcement learning reward evaluation feedback mechanism and a gradient descent algorithm.

[0060] For example, we can first collect all the request results that have completed inference output in this batch, compare the inference output of each request with the corresponding reference correct result, and calculate the reward value of the inference result of the request using a preset reward function (such as indicators based on result consistency matching degree, sequence integrity, etc.). Then, based on the reward value and the gradient function of the model, we can deduce the gradient change values ​​of the network parameters of each layer in reverse. The gradient calculation process can be completed by gradient aggregation of multiple computing units with the help of a distributed framework. Finally, we use a gradient descent optimizer to update the model's weights, biases and other parameters according to the aggregated gradient values. After the update is completed, we get the iteratively optimized model, which can be directly used to process the next batch of inference requests, so as to realize the continuous iterative upgrade of the model.

[0061] The aforementioned reinforcement learning training optimization method, by allocating text inference requests to at least one computing unit for parallel inference, truncates incomplete model outputs based on a preset output progress. This allows for updating model parameters based on completed inference results without waiting for all computing units to finish their inference tasks. This effectively solves the problems in the background technology where excessively long outputs of some inference requests lead to wasted computing power in many units, long training batch times, and low utilization of computing resources. Simultaneously, this mechanism enables the model to iterate and optimize based on completed, valid inference data in a timely manner, avoiding the impact of long-running requests on overall training efficiency. It significantly improves resource utilization and model training efficiency in multi-computation-unit reinforcement learning training, balancing low training costs with high parameter update efficiency.

[0062] In some embodiments, after truncating the incomplete model outputs in each computing unit, the reinforcement learning training optimization method provided in this application embodiment can further merge the incomplete text inference requests with the next batch of text inference requests to obtain merged input data; and then perform the next batch of inference output on the merged input data.

[0063] In this embodiment, after truncating the incomplete model output in each computing unit, the incomplete text inference requests are not directly discarded. Instead, a continuation and merging strategy is adopted to ensure the complete inference of the requests. First, the incomplete text inference requests that were truncated and removed from the current batch are selected as text inference requests to be continued. Then, the text inference requests to be continued are integrated with the newly connected text inference requests in the next batch to generate merged input data. This achieves seamless continuation of incomplete text inference requests, avoids the loss of inference data due to truncation, and ensures the continuity of the training task.

[0064] In some implementations, the evicted incomplete text reasoning requests are refilled with the key-value cache in the next batch, and the reasoning output is recalculated based on the recalculated key-value cache.

[0065] For incomplete text inference requests in the merged input data, a full pre-filling and recalculation approach can be used to complete subsequent inference. Specifically, the truncated incomplete text inference requests are treated as independent requests, and the complete pre-filling process is re-executed in the next batch of pre-filling stages. That is, the Key and Value vectors required by the attention mechanism are recalculated based on the original input data of these text inference requests, generating a brand-new KV cache. After the KV cache is built, the decoding and inference process is started based on the cache, gradually generating the output tokens of the incomplete requests until the entire inference task is completed.

[0066] In some implementations, the request to resume data can be obtained based on the input data corresponding to the incomplete text reasoning request and the reasoning output obtained before truncation; the incomplete request to resume data is then merged with the input data of the next batch of text reasoning requests to obtain merged input data.

[0067] In the process of generating the next batch of inference output from the merged input data, the merged input data can be pre-filled, the key-value cache can be calculated, and the inference output can be recalculated based on the calculated key-value cache.

[0068] The above implementation method is an optimization of the recalculation process, which can reuse the valid inference data before truncation and reduce the overhead of repeated calculations.

[0069] The aforementioned continuation method for incomplete requests effectively solves the problem of continuing inference for incomplete requests under the truncation mechanism, thereby improving training efficiency and resource utilization. On the one hand, the continuation merging strategy avoids discarding truncated requests, ensuring the integrity of the inference task, while merging incomplete requests with the next batch of requests for processing. This eliminates the need to allocate computing resources separately for individual incomplete requests, significantly reducing the scheduling overhead of scattered requests. On the other hand, the differentiated recalculation strategy supports both stable recalculation with full pre-filling and incremental continuation based on truncated data. By reusing effective output data before truncation, it reduces the amount of repetitive calculations in the inference stage, further improving the computing power utilization in the inference stage.

[0070] In an exemplary embodiment, for step 101 described above, as follows: Figure 2 As shown, a method for obtaining text reasoning requests in the current batch is provided. This method can be applied to the aforementioned computer device and includes, but is not limited to, the following steps:

[0071] 201. In the pre-filling stage, the initial text reasoning requests of this batch are pre-filled sequentially.

[0072] In some implementations, the input data length of each initial text inference request in the current batch is obtained; the difference ratio between the maximum and minimum input data lengths is determined; if the difference ratio exceeds a preset difference ratio, the input data of the initial text inference requests in the current batch are pre-filled sequentially from longest to shortest.

[0073] It's important to note that for the pre-filling process after sorting the input data from longest to shortest, the preset filling progress can be configured to 100%. This means waiting for all initial text inference requests in this batch to complete pre-filling before proceeding to subsequent steps. This is because the long-tail effect of pre-filling in the long-to-shortest order can effectively eliminate the long-tail effect in the pre-filling stage. The sorted short requests can be processed quickly after the long requests have completed pre-filling, without needing to truncate incomplete requests to avoid wasting computing power. Conversely, if the initial text inference requests are not sorted by length, or if a short-to-long pre-filling order is used, the preset filling progress can be configured to be below the 100% threshold. By truncating incomplete long requests in advance, their impact on the overall training pace can be avoided.

[0074] In some implementations, when obtaining the input data length of each initial text inference request in the current batch, it is possible to iterate through all initial text inference requests in the current batch and extract the input data length of each request. For example, extracting the number of tokens corresponding to the input text.

[0075] In some implementations, the difference ratio calculated above can be compared with a preset difference ratio. If the difference ratio exceeds the preset difference ratio, the order optimization strategy is triggered. Each initial text inference request is assigned to the computing unit for pre-filling processing in the order of input data length from longest to shortest, generating the corresponding KV cache. If the difference ratio does not reach the preset difference ratio, pre-filling can be performed in a random order or a default order.

[0076] For example, in the above method of calculating the difference ratio, the difference between the maximum input data length and the minimum input data length can be calculated first, and then the ratio of the difference to the minimum input data length can be calculated to obtain the above difference ratio.

[0077] For example, a preset difference ratio can be set to 50%. A batch of initial text inference requests contains 7 requests with input data lengths of 5, 10, 13, 15, 26, 29, and 35 words respectively. When calculating the difference ratio in the above manner, the maximum input data length can be determined to be 35, the minimum input data length to be 5, and the difference ratio to be (35-5) / 5×100%=600%, which is far greater than the preset difference ratio of 50%. At this point, the order optimization strategy is triggered, and the 7 requests are sorted from longest to shortest input data length, resulting in the order 35→29→26→15→13→10→5. Then, the calculation unit performs pre-filling processing on the requests in this order, prioritizing the completion of KV cache calculation for long requests, avoiding waiting in the subsequent decoding stage due to post-processing of long requests, and significantly shortening the total time of the pre-filling stage.

[0078] In the above implementation, during the pre-filling stage, by optimizing the pre-filling order of the initial text inference requests, the KV cache calculation of long requests is completed first, avoiding the waiting in the decoding stage caused by the post-processing of long requests, shortening the total pre-filling time and improving the computing power utilization, which can reduce the invalid waiting time of short requests and improve the overall computing power utilization of the pre-filling stage.

[0079] Furthermore, the above implementation can also trigger order optimization only when the difference in request length is too large, which can take into account the efficiency of pre-filling.

[0080] For example, Figure 3 This is a schematic diagram illustrating the time distribution of pre-filling before and after sorting training data requests by input length from longest to shortest. Figure 3 (a) and Figure 3 (b) uses time as the horizontal axis and parallel processing tasks (processes) as the vertical axis. Figure 3 (a) shows the time distribution when pre-padded without sorting by input length (or shortest to longest, or random order). Figure 3 Figure (b) shows the time distribution for pre-filling after sorting the inputs by length from longest to shortest. From the perspective of task startup and time distribution, Figure 3 In (a) of the example, short input requests and long input requests are mixed to initiate pre-filling. This results in a long wait for long requests to complete their pre-filling before the short requests can synchronously enter the decoding stage. This leads to an unbalanced state where short requests are idle while long requests consume computing power, and the pre-filling and decoding stages of each request are disconnected, ultimately resulting in a long total training time for the entire batch. Figure 3 As can be seen in (b), by prioritizing the pre-filling of long input requests, the longest pre-filling task can be advanced and completed first. The pre-filling of subsequent short input requests can be quickly followed during or after the pre-filling of long requests without additional waiting. The pre-filling of each request is seamlessly connected with the decoding stage. Short requests can immediately enter the decoding stage after completing pre-filling, avoiding ineffective idle time. The total training time of the entire batch is greatly shortened, and the computing load of each computing unit is more evenly distributed, giving full play to the advantages of data parallel processing.

[0081] 202. Determine whether the pre-filling completion progress is greater than or equal to the preset filling progress.

[0082] The aforementioned pre-filling completion progress includes: the pre-filling quantity, and / or, the pre-filling ratio.

[0083] The aforementioned pre-filled quantity may refer to the number of initial text inference requests that have been pre-filled, and the aforementioned pre-filled ratio refers to the percentage of the number of initial text inference requests that have been pre-filled to the total number of initial text inference requests in this batch.

[0084] It is understandable that the aforementioned pre-fill quantity and pre-fill ratio can be used as one or both as the basis for judgment, which can provide a flexible basis for progress judgment. Quantitative indicators can be selected according to the scale of the training task to avoid invalid pre-fill occupying computing resources.

[0085] If the pre-filling completion progress in the initial text inference request of this batch is greater than or equal to the preset filling progress, continue to execute step 203 below; if the pre-filling completion progress in the initial text inference request of this batch is less than the preset filling progress, continue to perform pre-filling processing based on step 201 above.

[0086] For example, suppose a batch of initial text inference requests totals 8000, and the preset filling progress is set to "pre-fill ratio reaches 80%" or "pre-fill quantity reaches 6000". During the pre-filling process, the number of completed requests is counted in real time. When the number of completed pre-filling requests reaches 6400, the corresponding pre-fill ratio is 6400 / 8000×100%=80%, reaching the preset filling progress threshold; the progress judgment result is triggered, and the pre-filling of the remaining 1600 requests is stopped, directly proceeding to step 203; if the preset filling quantity is set to 6000, then when the number of completed requests reaches 6000, the subsequent process can be triggered without waiting to reach the fixed ratio.

[0087] 203. The text reasoning requests that have completed pre-filling processing in the initial text reasoning requests of this batch are identified as the text reasoning requests of this batch.

[0088] In step 202, after determining that the pre-filling completion progress is greater than or equal to the preset filling progress, all requests that have completed pre-filling processing in the initial text inference requests of this batch are selected, and these requests are identified as text inference requests of this batch. Their corresponding KV cache data is used for subsequent inference output. For the remaining initial text inference requests that have not completed pre-filling, they are directly removed from the current training batch and do not participate in the inference output of this batch. They can wait to be merged with the initial requests of the next batch and then re-execute the pre-filling process.

[0089] For example, suppose that out of 8000 initial requests in a batch, 6400 are pre-filled and reach a preset fill ratio of 80%. These 6400 pre-filled initial text inference requests are selected and identified as the text inference requests for this batch. The key-value cache corresponding to these 6400 requests is allocated at least one computing unit to execute subsequent inference output. The remaining 1600 initial text inference requests that have not been pre-filled are removed from the current batch and will be merged with the initial text inference requests of the next batch before re-entering the pre-filling process.

[0090] In some embodiments, a strategy of expanding the inference batch is adopted when executing inference output. By integrating all pre-filled text inference requests in a training batch into one inference batch for parallel inference output, memory access is reduced, data transmission efficiency is improved, and computing resource utilization is increased. This avoids the situation in the traditional mode where decoding and inference output are started after a small number of requests are pre-filled for a batch of model training data, which leads to frequent switching of computing resources between pre-filled encoding and decoding inference modes, frequent data transmission, and computing power waiting for data transmission time, resulting in increased memory access and wasted computing resources.

[0091] In some embodiments, the length of the input data for the initial text inference requests in the current batch can be obtained; based on the length of the input data for the initial text inference requests in the current batch, the number of initial text inference requests whose input data length difference does not exceed a preset difference threshold can be determined; and based on the number of initial text inference requests, a preset filling progress can be determined.

[0092] This process can iterate through all initial text reasoning requests in this batch to obtain the input data length of each request; set a preset difference threshold (e.g., the difference in the number of tokens does not exceed 20), filter out a subset of requests with an input data length difference of no more than 20 from the batch of requests, and count the number of requests in this subset; and directly determine the number of requests in this subset as the preset fill progress threshold.

[0093] In some embodiments, the difference in input data length can refer to the difference between the maximum input data length and the minimum input data length, or it can refer to the difference between each input data length and the minimum input data length.

[0094] For example, suppose the preset difference threshold is set to 20 words, and a batch of initial requests contains 10 requests with input data lengths of 10, 15, 18, 20, 25, 30, 45, 50, 55, and 60. A subset of requests with input data length differences not exceeding 20 is selected: based on the minimum input data length of 10, requests with a length ≤ 30 (10+20) meet the requirements, corresponding to requests 10, 15, 18, 20, 25, and 30, totaling 6 requests. This subset of 6 is determined as the preset filling progress threshold for this batch. When the number of pre-filled requests reaches 6, a progress determination is immediately triggered, proceeding to step 203, avoiding excessive waiting for long requests exceeding 30 (45, 50, etc.) and ensuring the parallel efficiency of the pre-filling stage.

[0095] In the above implementation method, the preset filling progress is associated with the request length distribution to avoid resource waste in non-uniform length scenarios with a fixed threshold (preset filling progress).

[0096] In some embodiments, the input data length of each initial inference request in the initial inference requests of this batch can be obtained; the difference ratio between the maximum input data length and the minimum input data length can be determined; if the difference ratio does not exceed the preset difference ratio, and the pre-filling completion progress in the initial inference requests of this batch is greater than or equal to the preset filling progress, the inference request in the initial inference requests of this batch that has completed pre-filling processing is determined as the inference request of this batch.

[0097] The system can perform dual condition checks: the first check is whether the difference ratio is less than or equal to the preset difference ratio, and the second check is whether the pre-filling completion progress is greater than or equal to the preset filling progress. If both conditions are met, all initial inference requests that have completed pre-filling are identified as inference requests for this batch. If either condition is not met, the system executes according to the normal logic.

[0098] For example, a preset difference ratio is set to 50%, and a preset filling progress is set to 100% (i.e., all pre-filling is completed). A batch initially contains 5 requests with input lengths of 10, 12, 13, 14, and 15. The calculated difference ratio is (15-10) / 10×100%=50%. This difference ratio equals the preset difference ratio, satisfying the condition of uniform length distribution. Because the length distribution is uniform, the pre-filling time for each request is basically the same. Pre-filling continues until all 5 requests are pre-filled, and the pre-filling completion progress reaches 100%, satisfying the second condition. At this point, all 5 pre-filled requests are identified as inference requests for this batch and allocated to computing units for parallel inference output, maximizing computing power utilization.

[0099] In the above implementation, full pre-filling is performed only when the request length distribution is uniform, ensuring that the inference parallelism and computing power utilization are maximized in uniform length scenarios.

[0100] In an exemplary embodiment, regarding step 102 above, as follows: Figure 4 As shown, a method for allocating text reasoning requests is provided. This method can be applied to the aforementioned computer device and includes, but is not limited to, the following steps:

[0101] 401. Obtain the number and length of input data for the text inference requests in this batch.

[0102] In obtaining the number and length of input data for text inference requests in this batch, firstly, the text inference requests in this batch can be traversed to establish a request feature information table. This table can contain: 1) the number of input data, i.e., the initial number of text inference requests in this batch; and 2) the length of input data, i.e., the total number of tokens obtained after the input text corresponding to each text inference request is segmented by the tokenizer. This parameter directly determines the computational power consumption of the pre-filling stage and the memory size loaded in the decoding output stage after pre-filling. Both of these are used to characterize the size of the input text. Secondly, the data in the feature information table can be statistically summarized to calculate key indicators such as the total length of input data for this batch of requests, the maximum length of input data for a single request, and the minimum length of input data, providing a quantitative basis for the subsequent allocation strategy.

[0103] For example, assuming there are 6 text reasoning requests in this batch, the input data length (number of tokens) of each request is extracted by traversal and is 10, 13, 5, 15, 29, and 26 tokens respectively. Statistical analysis of this set of data shows that the total input data length is 10+13+5+15+29+26=98 tokens, the maximum input data length is 29 tokens, and the minimum input data length is 5 tokens, providing a data basis for the balanced distribution in step 402.

[0104] It should be noted that the text reasoning request involved in step 401 above can be the initial text reasoning request mentioned above. The initial text reasoning request is the basic data source for the pre-filling stage, and its core attributes (such as the number of input data and the length of input data) have been described in step 201 and related embodiments above. The text reasoning request processed in step 401 follows this basic definition and can directly use the initial text reasoning request that has undergone traversal extraction and length statistics as the processing object, ensuring the continuity and consistency of data flow in the entire technical solution. At the same time, it can also reuse the pre-processing logic such as length difference analysis and sorting optimization for the initial text reasoning request mentioned above, ensuring the rationality and efficiency of the subsequent allocation strategy.

[0105] 402. Based on the number and length of input data in the text inference requests in this batch, allocate a corresponding number of text inference requests to each computing unit for pre-filling processing.

[0106] The difference in the number of input data processed by each computing unit does not exceed a first preset value, and the difference in the total length of the input data does not exceed a second preset value.

[0107] The first preset value mentioned above is a threshold for constraining the balance of the number of requests processed by each computing unit, and the second preset value is a threshold for constraining the balance of the total length of requests processed by each computing unit. These two thresholds can be set according to actual needs, and the embodiments of this application do not specifically limit their values.

[0108] In some implementation methods, the text reasoning request assignment includes dual constraints:

[0109] First, the difference in the number of input data processed by each computing unit shall not exceed a first preset value. This preset value can be set according to the computing power performance of the computing unit, for example, it can be set to 10% to 15% of the average number of requests processed by a single computing unit.

[0110] Second, the total length difference of the input data processed by each computing unit does not exceed the second preset value. This preset value needs to be set in combination with the characteristic that the computing power consumption in the pre-filling stage is positively correlated with the square of the token length of each request. For example, it can be set to 10% to 15% of the average total length.

[0111] Based on the above constraints, either sorting and polling or grouping and extraction can be selected as the allocation strategy.

[0112] In some implementations, if a sorted round-robin allocation strategy is used, the text inference requests in this batch need to be sorted first according to the length of the input data in ascending or descending order. Then, the text inference requests are allocated to each computing unit sequentially according to the sorted data using alternating long and short round-robin or sequential round-robin methods. If a group extraction strategy is used, the text inference requests are first divided into length groups of (number of requests / number of computing units) × N (N is a positive integer). Then, each computing unit extracts requests from different length groups to ensure that the number of text inference requests in each computing unit is basically equal and the total length of the input data is relatively balanced. Finally, after the allocation is completed, each computing unit synchronously receives the allocated input data, starts pre-filling processing, and generates the corresponding KV cache.

[0113] For example, suppose there are 6 requests and 3 computing units (A, B, and C). The first preset value is set to 1 (meaning the difference in the number of requests processed by each computing unit does not exceed 1), and the second preset value is 10 tokens (meaning the difference in the total length of the input data processed by each computing unit does not exceed 5). First, the 6 requests are sorted in ascending order of input data length, resulting in the sequence: 5, 10, 13, 15, 26, 29. Then, using a round-robin allocation method with alternating long and short input data lengths, the requests are sequentially distributed to the 3 computing units. The allocation details include:

[0114] Calculation Unit A: Allocation requests 5 and 29, total input data length is 5 + 29 = 34 tokens;

[0115] Calculation Unit B: Allocation requests 10 and 26, with a total input data length of 10 + 26 = 36 tokens;

[0116] Calculation unit C: Assignment requests 13 and 15, with a total input data length of 13 + 15 = 28 tokens.

[0117] As can be seen from the above example, each computing unit processes 2 requests, with a difference of 0, which meets the first preset value requirement; the maximum difference in total length is 36-28=8 tokens, which meets the second preset value requirement, achieving the maximum possible load balancing distribution.

[0118] In some implementation methods, if a group extraction allocation strategy is adopted, it is necessary to first divide all text inference requests into a specified number of length groups according to the input data length based on the total number of requests in this batch and the number of computing units of the deployment instance; then each computing unit extracts an equal number of requests from all groups to ensure that the requests undertaken by each computing unit include samples of different length ranges and can achieve a certain degree of balanced allocation.

[0119] For example, suppose there are 6 text inference requests in this batch, with input data lengths (token counts) of 10, 13, 5, 15, 29, and 26 respectively. The deployment instance is configured with 3 computing units (A, B, and C). The number of requests processed by each computing unit is set to not exceed 1, and the total length of the input data differs by no more than 25 tokens. According to the grouping rules, the result of dividing the number of requests by the number of computing units is multiplied by a positive integer N (here, N=1). The 6 requests can be first split into two groups based on their input data length: the first group consists of short-length requests (5, 10, 13), and the second group consists of long-length requests (15, 26, 29). Then, the 3 computing units are each assigned a request from one of the two groups. One request is extracted from each group. Calculation unit A may extract 5 from the first group and 26 from the second group, for a total length of 31 tokens. Calculation unit B may extract 10 from the first group and 29 from the second group, for a total length of 39 tokens. Calculation unit C may extract 13 from the first group and 15 from the second group, for a total length of 28 tokens. Ultimately, each of the three calculation units is allocated two requests, with a difference of 0 in the number of requests and a maximum difference in total length of 11 tokens. This satisfies the preset constraints, achieving load balancing among the calculation units. Furthermore, it eliminates the need to sort all requests according to the length of the input data, saving sorting time and improving training efficiency.

[0120] In this embodiment of the application, in order to reduce cache pressure, reduce data transmission volume, and shorten data transmission time, corresponding quantization parameters can be set for each token and / or each attention head to quantize the input data parameters and / or model parameters. For example, the data format can be quantized from 16-bit floating-point (BF16) to 8-bit floating-point (FP8), which reduces the storage volume by nearly half and the data transmission volume is also reduced proportionally.

[0121] In one exemplary embodiment, such as Figure 5 As shown, a quantization processing method is provided during the inference output process. This method can be applied to the aforementioned computer device and includes, but is not limited to, the following steps:

[0122] 501. Determine the quantization object. The quantization object includes at least one of the input data parameters corresponding to this batch of text inference requests and the model parameters called during the inference output process.

[0123] The model parameters include the key-value cache of the multi-head attention mechanism and / or the network layer activation values.

[0124] The aforementioned key-value cache is cached data during the pre-filling process of a large language model. Its purpose is to avoid repeatedly calculating the key vector and value (V) vector of historical tokens when decoding and generating each token. By caching the already calculated K and V vectors, the amount of decoding computation can be greatly reduced and the inference speed can be improved. However, it will also occupy a lot of GPU memory. Its size is proportional to the batch size, the length of the input sequence, the number of model layers, the number of attention heads, and the KV vector dimension.

[0125] For example, the aforementioned large language models can be such as Generative Pre-trained Transformer (GPT) models, Metaverse AI Large Language Model (LLaMA), etc. Among them, Transformer is a deep learning model architecture based on a self-attention mechanism.

[0126] The aforementioned network layer activation values ​​refer to the input and output data of each layer of the model. Their tensor shape is usually [batch size, sequence length, hidden layer dimension], which is key data dynamically generated during the inference process.

[0127] The above input data parameters refer to the preprocessed data of this batch of text reasoning requests, which is the basic input information for model reasoning.

[0128] It should be noted that quantization is essentially a data compression technique. Its idea is to replace traditional high-precision data formats (such as floating-point FP32 and BF16) with lower-precision data formats (such as FP8) to represent and compute neural network models. The goal is to significantly improve model efficiency and reduce resource consumption while keeping model performance (accuracy) at a manageable level. Compared to coarse-grained (per-tensor) quantization, which uses only a single global quantization parameter, the word-by-word meta-quantization and attention-by-attention head quantization employed in this application are fine-grained quantization strategies. These strategies can more accurately adapt to the distribution differences of different data and significantly reduce quantization errors.

[0129] 502. For each quantization object, set the corresponding quantization parameters using word-by-word meta-quantization and / or attention-by-attention quantization.

[0130] The quantization parameters include a scaling factor (Scale) and a zero point (ZeroPoint). The core of quantization is to map a quantized object represented in a high-precision data format to a low-precision data format through a linear mapping process, achieving data compression. The key formula involved is: Q = round(R / Scale) + ZeroPoint; where Q is the quantized low-precision value, R is the original high-precision floating-point value, and round is the rounding operation. The dequantization process approximates the low-precision value back to the high-precision value using R' ≈ (Q - ZeroPoint) * Scale.

[0131] The word-by-word quantization method described above refers to calculating independent quantization parameters for each word in the input sequence. Specifically, for the i-th word in the sequence, all values ​​in its complete feature vector (length equal to the hidden layer dimension) are extracted. Based on the distribution of these values, the maximum value (R_max) and minimum value (R_min) are determined, and then a specific scaling factor and zero point are calculated, giving each word its own quantization parameters. This method is well-suited for scenarios where the dynamic range of activation values ​​for different words within the sequence varies greatly. For example, the activation values ​​of some keywords may be significantly greater than those of function words; independent quantization parameters can accurately match the data distribution of each word.

[0132] The aforementioned attention-head-by-attention quantization method refers to calculating independent quantization parameters for each attention head in a multi-head attention mechanism. Since each attention head in a multi-head attention mechanism can be considered an independently operating head, the numerical distributions of different attention heads may differ significantly; some focus on processing large-amplitude feature signals, while others focus on subtle feature signals. Therefore, it is necessary to calculate the scaling factor and zeros separately based on the numerical distribution of the quantization object corresponding to each attention head (such as the head's KVCache or output data). This method is particularly suitable for quantizing KVCaches, maximizing the preservation of information from each attention head and significantly reducing quantization errors.

[0133] For example, suppose a Transformer layer contains four attention heads (Head1, Head2, Head3, and Head4). During inference, the distribution of K-cache values ​​generated by each attention head is significantly different: Head1's K-cache values ​​range from approximately [-10.0, 10.0], focusing more on capturing large feature changes in the text; Head2's K-cache values ​​range from approximately [-0.5, 0.5], mainly focusing on subtle feature signals; Head3's K-cache values ​​range from approximately [-2.0, 8.0], exhibiting an asymmetrical distribution; and Head4's K-cache values ​​range from approximately [-1.0, 1.0], focusing only on features with narrow fluctuations. If coarse-grained (per-tensor) quantization is used, a single scaling factor is calculated based on the global maximum and minimum values. This leads to a serious waste of precision for attention heads with narrow distributions, such as Head2 and Head4, and a significant increase in quantization error. However, when using per-attention-head quantization, four independent scaling factors and zeros are calculated for each of the four attention heads. This accurately matches the dynamic numerical distribution of each head, preserves the feature information of each attention head to the maximum extent, and significantly reduces quantization error. This fully demonstrates the advantages of per-attention-head quantization in KVCache quantization scenarios.

[0134] In addition, a single quantization method or a combination of quantization methods can be flexibly selected according to the type of quantization object. For example, when quantizing KVCache, word-by-word quantization and attention-by-attention head quantization can be used simultaneously. That is, quantization parameters are calculated for each word of each attention head. This method has fine quantization granularity and high precision.

[0135] The advantage of the above-mentioned word-by-word quantization lies in its ability to handle situations where the dynamic range of activation values ​​of different words within a sequence varies greatly. For example, the activation values ​​of some keywords may be significantly higher than those of function words. By configuring independent quantization parameters for each word, the data distribution of each word can be accurately matched, avoiding local precision loss caused by coarse-grained quantization. However, this method also has certain limitations: first, it requires storing a separate set of quantization parameters for each word, which increases storage overhead slightly; second, it requires performing a quantization and dequantization operation for each word during the calculation process, which leads to a slight increase in computational overhead. Therefore, it is more suitable for quantization objects such as activation values ​​that exhibit significant dynamic changes and are sensitive to precision.

[0136] The advantage of the above-mentioned attention-based head quantization is that it is suitable for quantizing KVCache. Since each attention head in the multi-head attention mechanism can be regarded as an independently working head, the numerical distribution of different heads differs significantly. Some focus on processing large-amplitude feature signals, while others focus on subtle feature signals. Calculating the quantization parameters for each head separately can preserve the feature information of each head to the maximum extent and significantly reduce quantization error.

[0137] The combination of word-by-word quantization and attention-by-attention head quantization can create a more refined quantization strategy. For example, when quantizing KVCache, a combination of attention-by-attention head and word-by-word quantization can be used, where the quantization parameters are calculated separately for each word of each attention head. Horizontally, quantization is based on attention heads to ensure accurate matching of feature distributions across different heads; vertically, it is based on words to accommodate dynamically changing numerical ranges within the sequence. This combination offers the finest granularity and highest accuracy, maximizing the balance between memory savings and model performance. However, it incurs greater computational and storage overhead than a single quantization method, making it more suitable for training large models with long sequences where high accuracy is required and memory resources are limited. Therefore, the decision to perform quantization and the appropriate quantization scheme should be based on the model's training accuracy requirements, the quantity and length of the input data.

[0138] 503. Perform quantization processing on the quantization object based on quantization parameters.

[0139] The goal of quantization is to reduce cache pressure, decrease data transfer volume, and shorten data transfer time while ensuring that the model inference performance is not significantly degraded. Specifically, high-precision data formats (such as FP32 and BF16) can be quantized into low-precision data formats (such as FP8). Both BF16 and FP8 are floating-point formats. BF16 uses 2 bytes to store a value, while FP8 uses 1 byte. Compared to BF16 or FP16 formats, FP8 quantization can reduce storage volume by nearly half, and the data transfer volume is also reduced proportionally. This achieves extreme memory savings and improves bandwidth efficiency, allowing twice the amount of data to be transferred under the same memory bandwidth, effectively alleviating the memory bandwidth bottleneck.

[0140] In the specific quantization process, based on the quantization parameters determined in step 502, the original high-precision value of the quantized object is converted into a low-precision value through the key formula. During the inference calculation, if the quantized object needs to be used, the low-precision value can be approximately restored to a high-precision value through the dequantization formula to participate in the calculation. Although dequantization will result in a small amount of precision loss due to rounding operations, as long as the scaling factor is set reasonably, this error is usually acceptable. It can achieve the goal of improving running efficiency and reducing resource consumption under the premise of controllable model performance degradation.

[0141] For example, if the quantization object is a set of model weights in FP32 format [-2.0, -1.0, 0.0, 1.0, 2.0], and it is quantized to INT8 (range -128 to 127) using attention-based head quantization, first determine the maximum value R_max = 2.0 and the minimum value R_min = -2.0 for this set of weights, and calculate the scaling factor Scale = (2.0 - (-2.0)) / (127 - (-128)) = 4.0 / 255 ≈ 0.0157. Since it is necessary to ensure that the floating-point number 0 can be accurately mapped to the integer 0, calculate the zero point ZeroPoint = 0 - round(0.0 / 0.0157) = 0. Then, calculate the quantized value of each weight through the quantization formula to obtain the quantized weights in INT8 format [-127, -64, 0, 64, 127], and complete the quantization process. When the inference calculation needs to use the weights, the original precision value can be approximately restored through the dequantization formula.

[0142] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0143] Based on the same inventive concept, this application also provides a reinforcement learning training optimization apparatus for implementing the reinforcement learning training optimization method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the reinforcement learning training optimization apparatus provided below can be found in the limitations of the reinforcement learning training optimization method described above, and will not be repeated here.

[0144] In one exemplary embodiment, such as Figure 6 As shown, a reinforcement learning training optimization device is provided, comprising:

[0145] The acquisition module 601 is used to acquire the text reasoning requests of the current batch and allocate the text reasoning requests of the current batch to at least one computing unit for reasoning output;

[0146] The inference output control module 602 is used to truncate the unfinished model output in each calculation unit when the inference output completion progress in the text inference request of the current batch reaches the preset output progress.

[0147] The model update module 603 is used to update the model parameters of the model used for inference based on the inference output results completed in this batch and the reference correct results.

[0148] In some embodiments, the inference output control module 602 is specifically used for:

[0149] If the proportion of completed inference output in at least one computing unit is greater than a first preset inference proportion, the unfinished model output in the computing units that have not finished inference output is truncated.

[0150] or,

[0151] If the number of completed inference outputs in the text inference requests of this batch is greater than the preset number, the unfinished model outputs in each computing unit will be truncated.

[0152] or,

[0153] If the proportion of completed reasoning output in the text reasoning requests of this batch exceeds the second preset reasoning proportion, the unfinished model output in each computing unit will be truncated.

[0154] In some embodiments, the apparatus further includes:

[0155] The merging module is used to merge incomplete text reasoning requests with the next batch of text reasoning requests to obtain merged input data;

[0156] The inference output control module 602 is also used to perform the next batch of inference output on the merged input data.

[0157] In some embodiments, the merging module is specifically used for:

[0158] The request to resume transmission data is obtained based on the input data corresponding to the incomplete text reasoning request and the reasoning output results obtained before truncation.

[0159] The incomplete request continuation data is merged with the input data of the next batch of text reasoning requests to obtain the merged input data.

[0160] In some embodiments, the acquisition module 601 is specifically used for:

[0161] During the pre-filling stage, the initial text reasoning requests for this batch are pre-filled sequentially.

[0162] If the pre-filling completion progress in the initial text reasoning requests of this batch is greater than or equal to the preset filling progress, the text reasoning requests in the initial text reasoning requests of this batch that have completed pre-filling processing are determined as the text reasoning requests of this batch.

[0163] In some embodiments, the acquisition module 601 is specifically used to: acquire the length of the input data of each initial text inference request in the initial text inference requests in this batch;

[0164] Determine the ratio of the difference between the maximum and minimum input data lengths;

[0165] If the difference ratio exceeds the preset difference ratio, the input data lengths of the initial text reasoning requests in this batch are pre-filled sequentially from longest to shortest.

[0166] In some embodiments, the acquisition module 601 is further configured to: acquire the length of the input data for the initial text reasoning request in the current batch;

[0167] Based on the input data length of the initial text inference requests in this batch, determine the number of initial text inference requests whose input data length difference does not exceed a preset difference threshold.

[0168] The preset filling progress is determined based on the number of initial text reasoning requests.

[0169] In some embodiments, the inference output process further includes:

[0170] A quantization module is used to determine the quantization object, which includes at least one of the input data parameters corresponding to the current batch of text inference requests and the model parameters called during the inference output process. The model parameters include the key-value cache of the multi-head attention mechanism and / or the network layer activation values. For the quantization object, the corresponding quantization parameters are set using word-by-word meta-quantization and / or attention-by-head quantization. The quantization object is then quantized based on the quantization parameters.

[0171] In some embodiments, the acquisition module 601 is specifically used for:

[0172] Obtain the number and length of input data for this batch of Chinese text inference requests;

[0173] Based on the number and length of the input data in the text inference requests of this batch, a corresponding number of text inference requests are allocated to each computing unit for pre-filling processing. The difference in the number of input data processed by each computing unit does not exceed a first preset value, and the difference in the total length of the input data does not exceed a second preset value.

[0174] Each module in the aforementioned reinforcement learning training optimization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0175] In one exemplary embodiment, a computer device is provided, which may be a terminal or a server, and its internal structure diagram may be as follows: Figure 7 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements a reinforcement learning training optimization method.

[0176] Those skilled in the art will understand that Figure 7The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0177] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the various processes shown in the above method embodiments.

[0178] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the various processes shown in the above method embodiments.

[0179] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the various processes shown in the above method embodiments.

[0180] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0181] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0182] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A reinforcement learning training optimization method, characterized in that, include: Obtain the text reasoning requests for this batch, and assign the text reasoning requests for this batch to at least one computing unit for reasoning output; If the reasoning output completion progress in the text reasoning request of this batch reaches the preset output progress, the unfinished model output in each computing unit is truncated. Based on the inference output results already completed in this batch and the reference correct results, update the model parameters of the model used for inference.

2. The method according to claim 1, characterized in that, When the reasoning output completion progress in the text reasoning requests of this batch reaches the preset output progress, the unfinished model output in each computing unit is truncated, including: If the proportion of completed inference output in at least one computing unit is greater than a first preset inference proportion, the unfinished model output in the computing units that have not finished inference output is truncated. or, If the number of completed inference outputs in the text inference requests of this batch is greater than the preset number, the unfinished model outputs in each computing unit will be truncated. or, If the proportion of completed reasoning output in the text reasoning requests of this batch exceeds the second preset reasoning proportion, the unfinished model output in each computing unit will be truncated.

3. The method according to claim 1, characterized in that, After truncating the incomplete model output in each computational unit, the method further includes: The incomplete text reasoning requests are merged with the next batch of text reasoning requests to obtain merged input data; The merged input data is then used for the next batch of inference output.

4. The method according to claim 3, characterized in that, The step of merging incomplete text reasoning requests with the next batch of text reasoning requests to obtain merged input data includes: The request to resume transmission data is obtained based on the input data corresponding to the incomplete text reasoning request and the reasoning output results obtained before truncation. The incomplete request continuation data is merged with the input data of the next batch of text reasoning requests to obtain the merged input data.

5. The method according to claim 1, characterized in that, The requests for obtaining this batch of text reasoning data include: During the pre-filling stage, the initial text reasoning requests for this batch are pre-filled sequentially. If the pre-filling completion progress in the initial text reasoning requests of this batch is greater than or equal to the preset filling progress, the text reasoning requests in the initial text reasoning requests of this batch that have completed pre-filling processing are determined as the text reasoning requests of this batch.

6. The method according to claim 5, characterized in that, In the pre-filling stage, the initial text reasoning requests for this batch are pre-filled sequentially, including: Get the length of the input data for each initial text inference request in this batch; Determine the ratio of the difference between the maximum and minimum input data lengths; If the difference ratio exceeds the preset difference ratio, the input data lengths of the initial text reasoning requests in this batch are pre-filled sequentially from longest to shortest.

7. The method according to claim 5, characterized in that, Before determining the pre-filled text reasoning requests that have completed pre-filling processing in the initial text reasoning requests of this batch as the text reasoning requests of this batch, when the pre-filling completion progress in the initial text reasoning requests of this batch is greater than or equal to the preset filling progress, the method further includes: Get the length of the input data for the initial text inference request in this batch; Based on the input data length of the initial text inference requests in this batch, determine the number of initial text inference requests whose input data length difference does not exceed a preset difference threshold. The preset filling progress is determined based on the number of initial text reasoning requests.

8. The method according to claim 1, characterized in that, The reasoning output process also includes: The quantization object is determined, and the quantization object includes at least one of the input data parameters corresponding to the current batch of text inference requests and the model parameters called during the inference output process, wherein the model parameters include the key-value cache of the multi-head attention mechanism and / or the network layer activation value. For the quantization object, the corresponding quantization parameters are set using word-by-word meta-quantization and / or attention-by-attention quantization. The quantization object is quantized based on the quantization parameters.

9. The method according to any one of claims 1 to 8, characterized in that, The step of allocating at least one computing unit to perform inference output for the text inference requests in this batch includes: Obtain the number and length of input data for this batch of Chinese text inference requests; Based on the number and length of the input data in the text inference requests of this batch, a corresponding number of text inference requests are allocated to each computing unit for pre-filling processing. The difference in the number of input data processed by each computing unit does not exceed a first preset value, and the difference in the total length of the input data does not exceed a second preset value.

10. A reinforcement learning training optimization device, characterized in that, The device includes: The acquisition module is used to acquire the text reasoning requests in this batch and assign the text reasoning requests in this batch to at least one computing unit for reasoning output; The inference output control module is used to truncate the unfinished model output in each computing unit when the inference output completion progress in the text inference requests of this batch reaches the preset output progress. The model update module is used to update the model parameters of the model used for inference based on the inference output results already completed in this batch and the reference correct results.

Images