Response information generation method and device, medium and computer program product

Abstract

The invention discloses a response information generation method and device, a medium and a computer program product, and relates to the technical field of artificial intelligence. According to the computing power utilization rate of executing the pre-filling task by the equipment, determining the lemma budget quantity of the pre-filling task and the lemma budget quantity of the decoding task, so that the proportion of the lemma budget quantity of the decoding task is in negative correlation with the computing power utilization rate of the pre-filling task; the corresponding sequence to be processed is input into a pre-training language model for parallel reasoning calculation according to the budget number of the lexical elements, so that balance of throughput and delay is obtained through parallel reasoning calculation, and the problem of contradiction between equipment pressure and generation performance in parallel reasoning scheduling in related technologies can be solved; the technical effect of improving the parallel reasoning performance of the pre-training language model is achieved, and the execution performance of the artificial intelligence question and answer task is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a response information generation method, device, medium and computer program product. Background Art

[0002] AI question-answering technology is a booming branch of AI technology, and its applications cover multiple technical fields such as translation, article generation, summary generation, information search, image generation, image analysis, and code generation. Pre-trained language models are commonly used in AI question-answering technology. Using pre-trained language models to perform parallel reasoning calculations on multiple user tasks can better utilize device resources and improve reasoning efficiency compared to single-task execution, but it still puts tremendous pressure on device performance, and the efficiency of generating response information also needs to be improved.

[0003] Solving the problems of equipment pressure and generation efficiency of response information generated by artificial intelligence question-answering technology is a technical problem that technical personnel in this field need to solve. Summary of the invention

[0004] The present invention provides a response information generation method, device, medium and computer program product to at least solve the problem of the contradiction between device pressure and generation performance in the related art.

[0005] The present invention provides a method for generating response information, comprising: Receive the pending sequence of response tasks; Inputting the sequence to be processed into a pre-trained language model for inference calculation; In a batch of inference calculations, the word unit budget number of the pre-filling task and the word unit budget number of the decoding task are determined according to the computing power utilization rate of the device where the pre-filling task is performed, and the corresponding sequence to be processed is input into the pre-trained language model for parallel inference calculation according to the word unit budget number; splicing the output results of the decoding tasks to obtain the response information of the sequence to be processed; Outputting the response information; Among them, the proportion of the word unit budget quantity of the decoding task is negatively correlated with the computing power utilization rate of the pre-filling task.

[0006] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of the above-mentioned response information generation method when executing the computer program.

[0007] The present invention also provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program implements the steps of the above-mentioned response information generation method when executed by a processor.

[0008] The present invention also provides a computer program product, including a computer program, which implements the steps of the above-mentioned response information generation method when executed by a processor.

[0009] Through the present invention, a dynamic scheduling scheme is provided for the two tasks of pre-filling and decoding, which are respectively computationally intensive and memory intensive, in the inference calculation of a pre-trained language model. In a batch of inference calculation, the word unit budget number of the pre-filling task and the word unit budget number of the decoding task are determined according to the computing power utilization rate of the device where the pre-filling task is performed, so that the word unit budget number ratio of the decoding task is negatively correlated with the computing power utilization rate of the pre-filling task. According to the word unit budget number, the corresponding sequence to be processed is input into the pre-trained language model for parallel inference calculation, so that the parallel inference calculation obtains a balance between throughput and delay, thereby solving the problem of the contradiction between equipment pressure and generation performance in parallel inference scheduling in the related art, achieving the technical effect of improving the parallel inference performance of the pre-trained language model, splicing the output results of the decoding task, obtaining and outputting the response information of the sequence to be processed, and improving the execution performance of the artificial intelligence question and answer task. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1 A schematic diagram of the parallel reasoning process of a pre-trained language model; Figure 2 A flowchart of a method for generating response information provided by an embodiment of the present invention; Figure 3 A flowchart of another method for generating response information provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a batch mixing manager provided by an embodiment of the present invention; Figure 5 A schematic diagram of the structure of a response information generating device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] It should be noted that, in the description of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0014] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0015] Here, some key terms used in the embodiments of the present invention are explained.

[0016] Pre-trained language model (PLM) generally refers to designing language model training tasks based on large-scale corpus (including language training materials such as sentences and paragraphs), training large-scale neural network algorithm structures to learn and implement, and the final large-scale neural network algorithm structure and parameters are pre-trained language models. Subsequent tasks can be based on this model to extract features or fine-tune tasks to achieve specific task purposes. The idea of ​​pre-training is to first train a task to obtain a set of model parameters, then use this set of model parameters to initialize the network model parameters, and then use the initialized network model to train other tasks to obtain models adapted to other tasks. By pre-training on a large-scale corpus, the neural language representation model can learn powerful language representation capabilities and can extract rich syntactic and semantic information from the text. The pre-trained language model can provide tokens and sentence-level features containing rich semantic information for use in downstream tasks, and can also be fine-tuned directly on the pre-trained model for downstream tasks, so as to obtain downstream-specific models conveniently and quickly.

[0017] The neural network algorithm structure of the pre-trained language model training can be a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM) network, etc., or a model built with an attention network, such as a transformer model, a bidirectional encoder representation from transformers (BERT) model, a contrastive language-image pre-training (CLIP) model, etc., which is not limited in the present invention. An attention network refers to a network model that uses an attention mechanism for training. The model extracts more important feature information from the input sequence by assigning different weights to each part of the input sequence, so that the model finally obtains a more accurate output.

[0018] Compute Express Link (CXL) is a high-speed serial protocol that allows fast and reliable data transfer between different components within a computer system. It is designed to address bottlenecks in high-performance computing, including memory capacity, memory bandwidth, and I / O latency. CXL can also enable memory expansion and memory sharing, and can communicate with peripherals such as computing accelerators (such as GPUs and FPGAs), providing faster and more flexible data exchange and processing methods.

[0019] One of the reasons that limits the more efficient use of resources during the reasoning process of the pre-trained language model is that there are two completely different processes in the reasoning process of the pre-trained language model, namely the prefill phase (also known as the prompt phase) and the decoding phase (also known as the token generation phase).

[0020] When the computing system receives a user task (Request), in the pre-filling stage, the pre-trained language model processes the input (input) or prompt words (prompt) of all user tasks, processes the input tokens in parallel, and calculates the corresponding key-value cache data (KV Cache), which can fully utilize the computing power of the computing system in parallel and is a computationally intensive task. Next, it will enter the decoding stage, which generates tokens one by one in sequence, and only calculates one token each time the memory is accessed. Its computing power requirements are relatively low and are mainly limited by memory bandwidth. It is a memory-intensive task. The pre-trained language model repeats these two stages until the end-of-sequence (EOS) is generated or the stop condition set by the user is reached.

[0021] Figure 1 Schematic diagram of the parallel reasoning process of a pre-trained language model.

[0022] like Figure 1 As shown, x1 and x2 are two sequences of different lengths, that is, they include different numbers of tokens. The tokens of sequence x1 include "I", "think", "this", "is", and "great", and the tokens of sequence x2 include "I", "love", and "you". <eos>" indicates the terminator.

[0023] will be filled ( Figure 1 The sequence with the same length after left padding is sent to the pre-trained language model to perform the pre-filling task and produce the first token. The whole process of the pre-filling task is called an iteration (it can also be understood as an inference stage), that is, Figure 1 Iteration 1 is shown. Next, the decoding task is performed on these two sequences.

[0024] It can be found that after one iteration, sequence x2 has been inferred, but sequence x1 is still inferred. In traditional batch processing, sequences in a batch act together, so even though sequence x2 has been inferred, it cannot be "released".

[0025] Next, sequence x1 is iterated twice more (iteration 2, iteration 3, iteration 4 is used to output the terminator). Now sequence x1 has also completed inference. Then the data in the entire batch can be truly "released". When the inference of this batch is completed, the remaining requests can continue to form a new batch for the next round of inference.

[0026] Since the decoding task is performed token by token, the second iteration can only be performed after a token is produced in the first iteration. This results in idle time (bubble) of computing resources. Although the inference calculation of sequence x2 has been completed, it still occupies resources, which wastes resources and slows down the overall computing process.

[0027] In the related art, the scheduling of pre-filling tasks and decoding tasks often adopts a fixed resource ratio or a scheduling with decoding tasks as the priority. The ratio here is usually in tokens. Token is the smallest unit of text after word segmentation. In the reasoning of the pre-trained language model, the input text is divided into tokens, and the model processes these tokens one by one. More tokens often mean higher computing resource requirements and higher storage resource requirements. In the execution phase of the decoding task, the key-value cache is used to store the key-value pairs of each token to reduce repeated calculations, and the key-value cache data usually requires a large amount of storage space. Considering that the memory resources of computing accelerators such as GPUs are limited, the corresponding decoding tasks are usually considered when the key-value cache with a large amount of data fills the memory space, and after determining the proportion of the word unit budget allocated to the pre-filling task and the decoding task based on the memory resources, it is often no longer adjusted.

[0028] In actual applications, the lengths of the pending sequences of user tasks vary, and the load severity varies at different times. If a fixed word-unit budget ratio is used, if the word-unit budget ratio allocated to the pre-allocated task is too high, although the pre-filling task has been completed long ago, a large number of decoding tasks are piled up in queue, and the delay of the entire inference task is high; if the word-unit budget ratio allocated to the decoding task is too high, the throughput will be too small, the parallel effect will be poor, and the computing resources will not be fully utilized.

[0029] In order to improve the performance of generating response information while making full use of device resources, the present invention provides a dynamic scheduling solution for the two tasks of pre-filling and decoding in the inference calculation of the pre-trained language model, which are respectively computationally intensive and memory intensive. In a batch of inference calculation, the word unit budget number of the pre-filling task and the word unit budget number of the decoding task are determined according to the computing power utilization rate of the device where the pre-filling task is performed, so that the word unit budget number ratio of the decoding task is negatively correlated with the computing power utilization rate of the pre-filling task. According to the word unit budget number, the corresponding sequence to be processed is input into the pre-trained language model for parallel inference calculation, so that the parallel inference calculation obtains a balance between throughput and delay, thereby solving the problem of the contradiction between device pressure and generation performance in parallel inference scheduling in the related art, achieving the technical effect of improving the parallel inference performance of the pre-trained language model, splicing the output results of the decoding task, obtaining and outputting the response information of the sequence to be processed, and improving the execution performance of the artificial intelligence question and answer task.

[0030] To facilitate understanding of the environment on which the response information generation method provided by the embodiment of the present invention depends, the software and hardware architecture on which the response information generation method provided by the embodiment of the present invention can be executed is described here.

[0031] The response information generation method provided in the embodiment of the present invention can be applied to a single computing device or to a cluster including multiple computing devices. If applied to a computing cluster, the computing cluster may include multiple computing nodes, each of which is deployed with a pre-trained language model, and the computing node is used to receive the sequence to be processed of the response task; input the sequence to be processed into the pre-trained language model for reasoning calculation; in a batch of reasoning calculation, determine the word unit budget number of the pre-filling task and the word unit budget number of the decoding task according to the computing power utilization rate of the device where the pre-filling task is performed, and input the corresponding sequence to be processed into the pre-trained language model for parallel reasoning calculation according to the word unit budget number; splice the output results of the decoding task to obtain the response information of the sequence to be processed; output the response information; wherein, the word unit budget number ratio of the decoding task is negatively correlated with the computing power utilization rate of the pre-filling task.

[0032] In the embodiment of the present invention, a computing node is a computing device, and the types of computing nodes may include but are not limited to a graphics processing unit (GPU), a field programmable gate array device (FPGA), an application specific integrated circuit device (ASIC) and a processor distributed processing unit device (DPU), or a computing device using one or more of them as an accelerator, and other types of computing devices may also be used.

[0033] The computing cluster executes the response information generation method provided by the embodiment of the present invention, and can use data parallel inference (Data Parallel Inference) to divide the user task into multiple subsets, and distribute these subsets to multiple computing devices (such as GPU) for parallel processing. It can also use model parallel inference (Model Parallel Inference) to distribute different parts of the pre-trained language model to different computing devices for parallel processing, that is, each computing device only calculates a part of the model, and completes the reasoning of the entire model through communication between devices. It can also use hybrid parallel inference (Hybrid Parallel Inference) to use data parallelism and model parallelism technology at the same time to further improve the reasoning efficiency.

[0034] An embodiment of the present invention provides a method for generating response information. Combined with the execution flow of the method for generating response information, the method is described in detail below.

[0035] Figure 2 A flowchart of a method for generating response information provided by an embodiment of the present invention.

[0036] like Figure 2 As shown, the response information generation method provided by the embodiment of the present invention includes:

[0037] S201: Receive a pending sequence of response tasks.

[0038] S202: Input the sequence to be processed into a pre-trained language model for inference calculation.

[0039] S203: In a batch of inference calculations, determine the word unit budget number of the pre-filling task and the word unit budget number of the decoding task according to the computing power utilization rate of the device where the pre-filling task is performed, and input the corresponding sequence to be processed into the pre-trained language model for parallel inference calculation according to the word unit budget number.

[0040] S204: splicing the output results of the decoding tasks to obtain response information of the sequence to be processed.

[0041] S205: Output response information.

[0042] Among them, the proportion of word unit budget in decoding task is negatively correlated with the computing power utilization of pre-filling task.

[0043] In a specific implementation, the response information generation method provided in the embodiment of the present invention can be executed based on one computing device or a computing cluster including multiple computing devices.

[0044] The response tasks can be image generation, speech generation, content analysis, translation and other tasks.

[0045] For S201 , the sequence to be processed may be an input or prompt of a response task given by a user.

[0046] For S202 and S203, a batch in the embodiment of the present invention is used to process multiple sequences to be processed in parallel, and the inference calculation of all sequences to be processed in the current batch needs to be completed before the next batch is combined.

[0047] The pre-trained language model is used to perform parallel reasoning calculations. The parallel method may include at least one of the following two methods: treating multiple short input sequences as multiple sequences to be processed in parallel; splitting a long input sequence into multiple short sequences to be processed in parallel.

[0048] Then, in S203, the corresponding sequence to be processed is input into the pre-trained language model for parallel reasoning calculation according to the word unit budget number, which may include: performing pre-filling tasks on multiple sequences to be processed in parallel to obtain the first sequence corresponding to the sequence to be processed; if there is a first sequence that meets the number of long sequences, the first sequence is processed in blocks; and the decoding task is performed on multiple first sequences in parallel. By performing parallel reasoning calculation on multiple short sequences and dividing the long sequence into multiple short sequences for parallel calculation, it is avoided that different sequences with large length differences are processed in parallel in the same batch, resulting in idle resources.

[0049] Depending on the specific reasoning strategy and goal, a batch of reasoning calculations can be used to generate a new token for each sequence to be processed, or to perform multiple iterations and generate multiple tokens for each sequence to be processed.

[0050] Since the batches composed of different sequences to be processed may have large differences in the input length of the pre-training task and the generated length of the decoding task during the reasoning process of the pre-training language model, the embodiment of the present invention provides a dynamic scheduling scheme for pre-filling tasks and decoding tasks to adaptively adjust the word unit budget number of the pre-filling task and the word unit budget number of the decoding task. For long input sequences to be processed (such as the first sequence that meets the number of long sequences mentioned above), the word unit budget number of the pre-training task can be further subdivided to ensure that the calculation amount of each input is moderate and avoid the increase in delay caused by the excessive number of tokens in a single sequence to be processed.

[0051] In an embodiment of the present invention, the word unit budget quantity of the pre-filling task and the word unit budget quantity of the decoding task are determined according to the computing power utilization rate of the device where the pre-filling task is performed, and the adjustment principle is that the word unit budget quantity ratio of the decoding task is negatively correlated with the computing power utilization rate of the pre-filling task. Since the word unit budget quantity of the pre-filling task and the word unit budget quantity of the decoding task are allocated from the total word unit budget for the parallel reasoning task, the word unit budget quantity ratio of the decoding task is negatively correlated with the computing power utilization rate of the pre-filling task, which is equivalent to the word unit budget quantity ratio of the pre-filling task being positively correlated with the computing power utilization rate of the pre-filling task.

[0052] That is to say, when the computing power utilization of the pre-filling task drops significantly, it means that the pre-filling task is completed quickly and is waiting for the execution of the decoding task. At this time, by increasing the word unit budget ratio of the decoding task, that is, allocating more word unit budget to the decoding task, more decoding tasks can be loaded to reduce the latency of the current batch. When the computing power utilization of the pre-filling task is high, it means that the word unit budget allocated to the pre-filling task is insufficient. At this time, by allocating more word unit budget to the pre-filling task, the throughput of the current batch can be increased.

[0053] For S204, after the stop condition set by the user is reached, for each sequence to be processed, the output results of each decoding task are spliced ​​to obtain the response information of the sequence to be processed.

[0054] At S205, response information is output in response to the response task.

[0055] The response information generation method provided by the embodiment of the present invention provides a dynamic scheduling solution for the two tasks of pre-filling and decoding in the reasoning calculation of the pre-trained language model, which are respectively computationally intensive and memory intensive. In a batch of reasoning calculation, the word unit budget number of the pre-filling task and the word unit budget number of the decoding task are determined according to the computing power utilization rate of the device where the pre-filling task is performed, so that the word unit budget number ratio of the decoding task is negatively correlated with the computing power utilization rate of the pre-filling task. According to the word unit budget number, the corresponding sequence to be processed is input into the pre-trained language model for parallel reasoning calculation, so that the parallel reasoning calculation obtains a balance between throughput and delay, thereby solving the problem of the contradiction between equipment pressure and generation performance in parallel reasoning scheduling in the related art, achieving the technical effect of improving the parallel reasoning performance of the pre-trained language model, splicing the output results of the decoding task, obtaining and outputting the response information of the sequence to be processed, and improving the execution performance of the artificial intelligence question and answer task.

[0056] In some optional implementations of the embodiments of the present invention, S203 determines the word unit budget number of the pre-filling task and the word unit budget number of the decoding task according to the computing power utilization rate of the device where the pre-filling task is performed, including: when the computing power utilization rate of the pre-filling task is less than the computing power threshold, increasing the proportion of the word unit budget number of the decoding task.

[0057] In an embodiment of the present invention, a computing power threshold is used to trigger the adjustment of the word unit budget quantity ratio of the pre-filling task and the decoding task. Only when the computing power utilization rate of the pre-filling task is less than the computing power threshold, the increase of the word unit budget quantity ratio of the decoding task (i.e., the reduction of the word unit budget quantity ratio of the pre-filling task) is triggered. Accordingly, when the computing power utilization rate of the pre-filling task is greater than or equal to the computing power threshold, the word unit budget quantity ratio of the pre-filling task and the decoding task can adopt the initial setting value of the device, or can reduce the word unit budget quantity ratio of the decoding task on the basis of the word unit budget quantity ratio at the previous moment (i.e., increase the word unit budget quantity ratio of the pre-filling task).

[0058] In the embodiment of the present invention, the computing power utilization is the ratio of the actual computing power of the device when executing the pre-filling task to the peak computing power of the device.

[0059] Specifically, the actual computing power when the device performs the pre-filling task can be the difference between the computing power consumed by the device to perform each word of the current batch of pre-filling tasks and the computing power consumed by the device to generate the first word when performing the current batch of pre-filling tasks. At this time, the memory utilization of the pre-filling task can be expressed by the following formula: ; Where R represents the memory utilization of the pre-fill task, Indicates the computing power indicator of a single word of the device executing the pre-filling task of the current batch. Indicates the computing power consumed by the device to execute each word of the current batch of pre-filling tasks. represents the attenuation coefficient (the coefficient of attenuation over time, an inherent parameter related to the physical properties of the device), Indicates the time to first token, which is the time it takes for the device to generate the first token when executing the pre-fill task of the current batch. Indicates the peak computing power of the device when executing the current batch of pre-filling tasks. Indicates the peak computing power of the device.

[0060] Alternatively, when the device also includes extended memory, the problem that computing devices that can efficiently perform pre-filling tasks often have insufficient memory resources can be solved. At this time, communication loss and other losses will occur in the computing system. The actual computing power can be the difference between the computing power consumed by the device to perform each word of the current batch of pre-filling tasks plus the communication loss of the device when performing the current batch of pre-filling tasks minus the computing power consumed by the device to generate the first word when performing the current batch of pre-filling tasks. At this time, the memory utilization rate of the pre-filling task can be expressed by the following formula: ; Where R represents the memory utilization of the pre-fill task, Indicates the computing power indicator of a single word of the device executing the pre-filling task of the current batch. Indicates the computing power consumed by the device to execute each word of the current batch of pre-filling tasks. Indicates communication loss and other losses (can be set to 50% of the device's peak computing power for the scaling memory pool). represents the attenuation coefficient (the coefficient of attenuation over time, an inherent parameter related to the physical properties of the device), Indicates the time to first token, which is the time it takes for the device to generate the first token when executing the pre-fill task of the current batch. Indicates the peak computing power of the device when executing the current batch of pre-filling tasks. Indicates the peak computing power of the device.

[0061] In the embodiment of the present invention, the computing power threshold may include one or more thresholds, and when different computing power thresholds are triggered, different adjustment scales of word unit budget quantity ratios may correspond.

[0062] In some other optional implementations of the embodiments of the present invention, S203 determines the word-unit budget number of the pre-filling task and the word-unit budget number of the decoding task according to the computing power utilization of the device where the pre-filling task is performed, and may also include: determining the word-unit budget number of the pre-filling task and the word-unit budget number of the decoding task according to the computing power utilization of the pre-filling task and the memory utilization of the device; wherein the proportion of the word-unit budget number of the decoding task is negatively correlated with the memory utilization of the device.

[0063] Among them, the proportion of the word unit budget quantity of the decoding task is negatively correlated with the computing power utilization rate of the pre-filling task, and the specific analysis is recorded in the above embodiment.

[0064] The word-unit budget ratio of the decoding task is negatively correlated with the memory utilization of the device, that is, the word-unit budget ratio of the pre-filling task is positively correlated with the memory utilization of the device. For memory utilization, when the memory utilization of the device is low, it means that the amount of data generated by the key-value cache data is small, which means that the throughput of the current batch is small, and the word-unit budget allocated to the pre-filling task is insufficient. At this time, the word-unit budget ratio of the decoding task is reduced, that is, the word-unit budget ratio of the pre-filling task is increased to improve throughput. When the memory utilization of the device is high, it means that the amount of data generated by the key-value cache data is large, which means that the load of the current batch is large. At this time, the word-unit budget ratio of the decoding task is prioritized to reduce latency.

[0065] In an embodiment of the present invention, determining the word unit budget quantity of the pre-filling task and the word unit budget quantity of the decoding task based on the computing power utilization of the pre-filling task and the memory utilization of the device can include: increasing the proportion of the word unit budget quantity of the decoding task when at least one of the conditions that the computing power utilization of the pre-filling task is less than the computing power threshold and the memory utilization of the device is less than the first memory threshold is met.

[0066] In the embodiment of the present invention, the method for calculating the computing power utilization and the method for determining the computing power threshold can be found in the description of the above-mentioned implementation method.

[0067] The memory utilization rate can be the ratio of the memory usage of the current batch to the total memory allocated by the device to the response task. The memory usage of the current batch can include the memory usage of key-value cache data, tokens of the current batch, and model parameters of the pre-trained language model.

[0068] In some further optional implementations of the embodiments of the present invention, S203 determines the word-unit budget number of the pre-filling task and the word-unit budget number of the decoding task according to the computing power utilization of the device where the pre-filling task is performed, and may also include: determining the word-unit budget number of the pre-filling task and the word-unit budget number of the decoding task according to the computing power utilization of the pre-filling task and the operating status parameters of the device; wherein the operating status parameters include at least one of temperature, power, and the speed of the cooling fan; and the proportion of the word-unit budget number of the decoding task is positively correlated with the operating status parameters.

[0069] In actual applications, the power of the device when performing a pre-filling task is usually higher than the power of the device when performing a decoding task. Correspondingly, the temperature of the device when performing a pre-filling task will also be higher than the temperature of the device when performing a decoding task, and the fan speed of the device when performing a pre-filling task will also be higher than the fan speed of the device when performing a decoding task. Therefore, it is also possible to combine operating status parameters such as temperature, power, and cooling fan speed with the computing power utilization of the pre-filling task to determine whether the word unit budget ratio needs to be adjusted. For example, when the temperature of the device is too high, it means that the word unit budget number allocated to the pre-filling task is too high. At this time, the word unit budget ratio of the pre-filling task is reduced, that is, the word unit budget ratio of the decoding task is increased, so as to achieve temperature regulation of the device.

[0070] In an embodiment of the present invention, determining the word unit budget quantity of the pre-filling task and the word unit budget quantity of the decoding task according to the computing power utilization of the pre-filling task and the operating status parameters of the device can include: when at least one of the conditions that the computing power utilization of the pre-filling task is less than the computing power threshold and the operating status parameter is greater than the operating status threshold is met, increasing the proportion of the word unit budget quantity of the decoding task.

[0071] In the embodiment of the present invention, the calculation method of computing power utilization and the method for determining computing power threshold can be found in the description of the above implementation. The calculation method of memory utilization and the method for determining memory threshold can be found in the description of the above implementation.

[0072] One or more operating status parameters may be considered, and corresponding operating status thresholds may be set for temperature, power, speed of cooling fans, etc.

[0073] With reference to the above-mentioned implementation mode, S203 determines the word-unit budget quantity of the pre-filling task and the word-unit budget quantity of the decoding task according to the computing power utilization rate of the device where the pre-filling task is performed, and may also include: determining the word-unit budget quantity of the pre-filling task and the word-unit budget quantity of the decoding task according to the computing power utilization rate of the pre-filling task, the memory utilization rate of the device and the operating status parameters of the device; wherein the operating status parameters include at least one of the temperature, power and the speed of the cooling fan; the proportion of the word-unit budget quantity of the decoding task is negatively correlated with the memory utilization rate of the device; and the proportion of the word-unit budget quantity of the decoding task is positively correlated with the operating status parameters.

[0074] In an embodiment of the present invention, determining the word-unit budget quantity of the pre-filling task and the word-unit budget quantity of the decoding task based on the computing power utilization of the pre-filling task, the memory utilization of the device and the operating status parameters of the device can include: increasing the proportion of the word-unit budget quantity of the decoding task when at least one of the following conditions is met: the computing power utilization of the pre-filling task is less than the computing power threshold, the memory utilization of the device is less than the first memory threshold, and the operating status parameter is greater than the operating status threshold.

[0075] In the embodiment of the present invention, the calculation method of computing power utilization and the method for determining computing power threshold can be found in the description of the above embodiment. The calculation method of memory utilization and the method for determining memory threshold can be found in the description of the above embodiment. The calculation method of operating state parameter and the method for determining operating state threshold can be found in the description of the above embodiment.

[0076] In addition to using memory utilization and operating status parameters to assist computing power utilization to adjust the word unit budget ratio in the above embodiment, memory utilization and operating status parameters can also be used to adjust the operating status of the device. Then the response information generation method provided in the embodiment of the present invention may also include: when at least one of the conditions that the memory utilization reaches the memory threshold and the operating status parameter reaches the operating status threshold is met, the device is downgraded. That is to say, the system stability of the device can be monitored through memory utilization and operating status parameters to prevent the overall system from crashing.

[0077] Figure 3 A flowchart of another method for generating response information provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a batch mixing manager provided in an embodiment of the present invention.

[0078] Since the pre-filling task and the decoding task are two types of tasks with completely opposite resource requirements, there is a contradiction when using a single type of resource. Therefore, in an embodiment of the present invention, a solution for separating the pre-filling task and the decoding task is provided. When an accelerator is used as a computing device, the extended memory can be combined to implement an architecture that separates the pre-filling task and the decoding task.

[0079] In an embodiment of the present invention, in S203, the corresponding sequence to be processed is input into the pre-trained language model for parallel reasoning calculation according to the word unit budget number, which may include: performing a pre-filling task on the sequence to be processed to obtain a first sequence; reading the model weights of the pre-trained language model from the system memory to calculate the query vector and key-value matrix corresponding to the first sequence; storing the key-value matrix into the extended memory; and performing a decoding task according to the query vector, the key-value matrix read from the extended memory, and the word units of the first sequence read from the system memory.

[0080] like Figure 3 As shown, an embodiment of the present invention provides an architecture that separates pre-filling tasks and decoding tasks, which is composed of three parts: a computing module, a system memory, and an extended memory. Among them, the system memory is used to store the original data (sequence to be processed) that needs to be calculated for the pre-training task, and the result vector obtained by the model output. The computing module is usually used for large-scale parallel computing to obtain the intermediate results and model output of the pre-trained language model. The extended memory is used to store the key-value cache data in the middle of the pre-trained language model to accelerate the reasoning speed of the model.

[0081] In an embodiment of the present invention, determining the word-unit budget quantity of the pre-filling task and the word-unit budget quantity of the decoding task according to the computing power utilization rate of the device where the pre-filling task is performed may include: determining the total word-unit budget quantity allocated to the inference calculation according to the overall computing power index of the device and the memory utilization rate of the extended memory; after initializing the word-unit budget quantity of the decoding task, determining the word-unit budget quantity of the pre-filling task according to the remaining memory resources of the device; when the computing power utilization rate of the pre-filling task is less than the first computing power threshold, increasing the word-unit budget quantity ratio of the decoding task; when the computing power utilization rate of the pre-filling task is greater than or equal to the second computing power threshold, reducing the word-unit budget quantity ratio of the decoding task.

[0082] In the embodiment of the present invention, the extended memory may adopt computer express extended memory, specifically, may adopt CXL Type 3 device (supporting computer express memory (CXL.mem) protocol).

[0083] Figure 3 In the process of query vector and key-value matrix (QKV Linear) calculation, only the model weights need to be read from the computing module memory, and the calculation is independent of the length of the input sequence; when doing attention (Attn) calculation, there is no need to read the weights from the system memory; the execution phase of the decoding task requires reading the key-value cache data (KV-cache) from the extended memory to accelerate the calculation.

[0084] like Figure 3 As shown, in the current batch, assuming that , , , Four pending sequences are processed for parallel reasoning calculations, each of which includes 5 tokens. At a certain moment in the parallel reasoning calculation, the batch mixing manager is designed and combined with the extended memory for scheduling. The steps of data flow include:

[0085] ①: Assume that at this moment, 13 tokens ( , , , , , , , , , , , , ) performs pre-filled parallel computation, and the computation module reads the input from the system memory (of size [13, H]).

[0086] ②: In the linear transformation calculation module (QKV Linear), three matrices (of size [13, 3H]) are calculated: query vector (Q), key (K), and value (V).

[0087] ③: The batch mixing manager determines and controls the status of the extended memory through the status register read-write module and the control register read-write module.

[0088] ④: The host of the device stores the K and V matrices into the extended memory pool composed of CXL type-3 devices.

[0089] ⑤⑥: The Q matrix continues to flow to the subsequent computing module; read the key-value cache data in the extended memory; calculate and Attention vector and the attention vector ); The computing module initiates reading of the original input from the system memory again and starts decoding calculations. The 7 token model inputs and In ( , , , )and( , , ) are decoded and calculated to obtain the decoding vector.

[0090] ⑦⑧: The subsequent sequence calculation is the same as step ⑤⑥, including 4 identical operations, generating and Attention vector and the attention vector ).

[0091] ⑨: When all sequences have completed decoding calculations, the resulting sequences are concatenated and merged.

[0092] ⑩: The merged vector (scale is [13, H]) is linearly transformed to obtain the output, which is the response information of the sequence to be processed.

[0093] like Figure 4 As shown, a batch mixing manager is deployed to interact with the extended memory and schedule the ratio of the number of token budgets when performing prefilling tasks and decoding tasks.

[0094] In the specific implementation, the device (server machine) is roughly divided into three parts: system memory, computing module, and extended memory. The key-value cache data is stored in the extended memory. The input data of the model is stored in the system memory. The computing module is responsible for batch mixing during the inference process.

[0095] The batch mixing manager controls the access to the extended memory by reading and writing registers in the extended memory, and implements the mixing strategy by calculating the dynamic control index. The dynamic control index takes into account the time decay coefficient of the hardware device and monitors the computing power index of a single token and the time taken to generate the first token ( ). The batch mixing manager includes a communication transceiver module for coordinating multiple computing modules within the device.

[0096] Using the architecture of separating the pre-filling task from the decoding task provided by the embodiment of the present invention, the steps performed in the batch mixing manager include:

[0097] After the input data is sent to the manager, the Q matrix is ​​stored in the manager memory, and the KV matrix (KV-cache) is stored in the extended memory through the control register read-write module.

[0098] The scheduling module of the batch mixing manager determines the ratio of the word unit budget for the pre-filling task and the decoding task by calculating the threshold, divides the tokens of the input sequence at the next moment, and passes the tokens of the decoding task execution phase to the decoding calculation module for calculation.

[0099] The computing power index data and memory usage rate are transmitted to the manager. Memory usage will affect the operation stability of the entire server. If the memory usage exceeds the set threshold, the system will be downgraded to prevent the entire system from crashing. The computing power index is used to dynamically schedule the budget number of word units for pre-filling tasks and decoding tasks in the calculation module.

[0100] By utilizing the cache consistency of CXL Type 3 devices, the manager is connected to the managers of other computing modules to form an interconnection between multiple computing devices. The communication transceiver module transmits the status of the local computing module to other heterogeneous computing devices by modifying the field information carried by the flight data unit (CXL flit) in the CXL protocol, thereby realizing resource coordination among multiple devices.

[0101] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0102] Figure 5 A schematic diagram of the structure of a response information generating device provided by an embodiment of the present invention.

[0103] like Figure 5 As shown, an embodiment of the present invention further provides a response information generating device, comprising: The receiving module 501 is used to receive a sequence of response tasks to be processed; The calculation module 502 is used to input the sequence to be processed into the pre-trained language model for inference calculation; in a batch of inference calculation, the word unit budget number of the pre-filling task and the word unit budget number of the decoding task are determined according to the computing power utilization rate of the device where the pre-filling task is performed, and the corresponding sequence to be processed is input into the pre-trained language model for parallel inference calculation according to the word unit budget number; the output results of the decoding task are spliced ​​to obtain the response information of the sequence to be processed; Output module 503, used for outputting response information; Among them, the proportion of word unit budget in decoding task is negatively correlated with the computing power utilization of pre-filling task.

[0104] In an embodiment of the present invention, the computing module 502 determines the word unit budget quantity of the pre-filling task and the word unit budget quantity of the decoding task according to the computing power utilization rate of the device where the pre-filling task is performed, which may include: when the computing power utilization rate of the pre-filling task is less than the computing power threshold, increasing the proportion of the word unit budget quantity of the decoding task.

[0105] In an embodiment of the present invention, the computing module 502 determines the word-unit budget number of the pre-filling task and the word-unit budget number of the decoding task according to the computing power utilization of the device on which the pre-filling task is performed, and may also include: determining the word-unit budget number of the pre-filling task and the word-unit budget number of the decoding task according to the computing power utilization of the pre-filling task and the memory utilization of the device; wherein the proportion of the word-unit budget number of the decoding task is negatively correlated with the memory utilization of the device.

[0106] Among them, determining the word unit budget number of the pre-filling task and the word unit budget number of the decoding task according to the computing power utilization of the pre-filling task and the memory utilization of the device can include: when at least one of the conditions that the computing power utilization of the pre-filling task is less than the computing power threshold and the memory utilization of the device is less than the first memory threshold is met, increasing the proportion of the word unit budget number of the decoding task.

[0107] In an embodiment of the present invention, the computing module 502 determines the word-unit budget quantity of the pre-filling task and the word-unit budget quantity of the decoding task according to the computing power utilization of the device where the pre-filling task is performed, and may also include: determining the word-unit budget quantity of the pre-filling task and the word-unit budget quantity of the decoding task according to the computing power utilization of the pre-filling task and the operating status parameters of the device; wherein the operating status parameters include at least one of temperature, power, and the speed of the cooling fan; the word-unit budget quantity ratio of the decoding task is positively correlated with the operating status parameters.

[0108] Among them, determining the word unit budget number of the pre-filling task and the word unit budget number of the decoding task according to the computing power utilization of the pre-filling task and the operating status parameters of the device can include: when at least one of the conditions that the computing power utilization of the pre-filling task is less than the computing power threshold and the operating status parameter is greater than the operating status threshold is met, increasing the proportion of the word unit budget number of the decoding task.

[0109] In the embodiment of the present invention, the computing power utilization rate may be a ratio of the actual computing power of the device when executing the pre-filling task to the peak computing power of the device.

[0110] In the embodiment of the present invention, the actual computing power may be the difference between the computing power consumed by the device to execute each word element of the current batch of pre-filling tasks and the computing power consumed by the device to generate the first word element when executing the current batch of pre-filling tasks.

[0111] In an embodiment of the present invention, the device includes an extended memory; the actual computing power can be the difference between the computing power consumed by the device to execute each word element of the current batch of pre-filling tasks plus the communication loss of the device when executing the current batch of pre-filling tasks minus the computing power consumed by the device to generate the first word element when executing the current batch of pre-filling tasks.

[0112] In an embodiment of the present invention, the calculation module 502 inputs the corresponding sequence to be processed into the pre-trained language model for parallel reasoning calculation according to the word unit budget number, which may include: performing a pre-filling task on the sequence to be processed to obtain a first sequence; reading the model weights of the pre-trained language model from the system memory to calculate the query vector and key-value matrix corresponding to the first sequence; storing the key-value matrix into the extended memory; and performing a decoding task according to the query vector, the key-value matrix read from the extended memory, and the word units of the first sequence read from the system memory.

[0113] Among them, the calculation module 502 determines the word-unit budget number of the pre-filling task and the word-unit budget number of the decoding task according to the computing power utilization rate of the device where the pre-filling task is performed, which may include: determining the total word-unit budget number allocated to the inference calculation according to the overall computing power index of the device and the memory utilization rate of the extended memory; after initializing the word-unit budget number of the decoding task, determining the word-unit budget number of the pre-filling task according to the remaining memory resources of the device; when the computing power utilization rate of the pre-filling task is less than the first computing power threshold, increasing the word-unit budget number ratio of the decoding task; when the computing power utilization rate of the pre-filling task is greater than or equal to the second computing power threshold, reducing the word-unit budget number ratio of the decoding task.

[0114] In an embodiment of the present invention, the calculation module 502 inputs the corresponding sequence to be processed into the pre-trained language model for parallel reasoning calculation according to the word unit budget number, which may include: executing the pre-filling task on multiple sequences to be processed in parallel to obtain the first sequence corresponding to the sequence to be processed; if there is a first sequence that meets the number of long sequences, the first sequence is processed in blocks; and executing the decoding task on multiple first sequences in parallel.

[0115] For the description of the features in the embodiment corresponding to the response information generating device, reference may be made to the relevant description of the embodiment corresponding to the response information generating method, which will not be described in detail here.

[0116] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above-mentioned response information generation method embodiments.

[0117] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned response information generation method embodiments when running.

[0118] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0119] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned response information generation method embodiments are implemented.

[0120] An embodiment of the present invention further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned response information generation method embodiments are implemented.

[0121] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0122] The above is a detailed introduction to a response information generation method, device, medium and computer program product provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.< / eos>

Claims

1. A method for generating response information, characterized in that: include: Receive the pending sequence of response tasks; Inputting the sequence to be processed into a pre-trained language model for inference calculation; In a batch of inference calculations, the word unit budget number of the pre-filling task and the word unit budget number of the decoding task are determined according to the computing power utilization rate of the device where the pre-filling task is performed, and the corresponding sequence to be processed is input into the pre-trained language model for parallel inference calculation according to the word unit budget number; splicing the output results of the decoding tasks to obtain the response information of the sequence to be processed; Outputting the response information; Among them, the proportion of the word unit budget quantity of the decoding task is negatively correlated with the computing power utilization rate of the pre-filling task.

2. The response information generating method according to claim 1, characterized in that: Determining the word unit budget quantity of the pre-filling task and the word unit budget quantity of the decoding task according to the computing power utilization rate of the device where the pre-filling task is performed, including: When the computing power utilization rate of the pre-filling task is less than the computing power threshold, the word unit budget quantity ratio of the decoding task is increased.

3. The response information generating method according to claim 1, characterized in that: Determining the word unit budget quantity of the pre-filling task and the word unit budget quantity of the decoding task according to the computing power utilization rate of the device where the pre-filling task is performed, including: Determine the word unit budget number of the pre-filling task and the word unit budget number of the decoding task according to the computing power utilization rate of the pre-filling task and the memory utilization rate of the device; The word unit budget ratio of the decoding task is negatively correlated with the memory utilization of the device.

4. The response information generating method according to claim 3, characterized in that: Determining the word unit budget quantity of the pre-filling task and the word unit budget quantity of the decoding task according to the computing power utilization rate of the pre-filling task and the memory utilization rate of the device includes: When at least one of the following conditions is met: the computing power utilization of the pre-filling task is less than the computing power threshold, and the memory utilization of the device is less than the first memory threshold, the word unit budget quantity ratio of the decoding task is increased.

5. The response information generating method according to claim 1, characterized in that: Determining the word unit budget quantity of the pre-filling task and the word unit budget quantity of the decoding task according to the computing power utilization rate of the device where the pre-filling task is performed, including: Determine the word unit budget number of the pre-filling task and the word unit budget number of the decoding task according to the computing power utilization rate of the pre-filling task and the operating state parameter of the device; Wherein, the operating status parameter includes at least one of temperature, power, and speed of a cooling fan; The word unit budget quantity ratio of the decoding task is positively correlated with the operating state parameter.

6. The response information generating method according to claim 5, characterized in that: Determining the word unit budget quantity of the pre-filling task and the word unit budget quantity of the decoding task according to the computing power utilization rate of the pre-filling task and the operating state parameter of the device includes: When at least one of the following conditions is met: the computing power utilization rate of the pre-filling task is less than the computing power threshold, and the operating state parameter is greater than the operating state threshold, the word unit budget quantity ratio of the decoding task is increased.

7. The response information generating method according to any one of claims 2 to 6, characterized in that: The computing power utilization is a ratio of the actual computing power of the device when executing the pre-filling task to the peak computing power of the device.

8. The response information generating method according to claim 7, characterized in that: The actual computing power is the difference between the computing power consumed by the device to execute each word of the pre-filling task of the current batch and the computing power consumed by the device to generate the first word when executing the pre-filling task of the current batch.

9. The response information generating method according to claim 7, characterized in that: The device includes an extended memory; The actual computing power is the difference between the computing power consumed by the device when executing each word of the pre-filling task of the current batch plus the communication loss of the device when executing the pre-filling task of the current batch minus the computing power consumed by the device when generating the first word when executing the pre-filling task of the current batch.

10. The response information generating method according to claim 1, characterized in that: Inputting the corresponding sequence to be processed into the pre-trained language model for parallel reasoning calculation according to the word unit budget number includes: Execute the pre-filling task on the sequence to be processed to obtain a first sequence; Reading the model weights of the pre-trained language model from the system memory to calculate the query vector and key-value matrix corresponding to the first sequence; Storing the key-value matrix in an extended memory; The decoding task is performed according to the query vector, the key-value matrix read from the extended memory, and the first sequence of words read from the system memory.

11. The response information generating method according to claim 10, characterized in that: Determining the word unit budget quantity of the pre-filling task and the word unit budget quantity of the decoding task according to the computing power utilization rate of the device where the pre-filling task is performed, including: Determining a total word unit budget amount allocated to the inference calculation according to the overall computing power index of the device and the memory utilization rate of the extended memory; After initializing the word unit budget quantity of the decoding task, determining the word unit budget quantity of the pre-filling task according to the remaining memory resources of the device; When the computing power utilization rate of the pre-filling task is less than the first computing power threshold, increasing the word unit budget quantity ratio of the decoding task; When the computing power utilization rate of the pre-filling task is greater than or equal to the second computing power threshold, the word unit budget quantity ratio of the decoding task is reduced.

12. The response information generating method according to claim 1, characterized in that: Inputting the corresponding sequence to be processed into the pre-trained language model for parallel reasoning calculation according to the word unit budget number includes: Execute the pre-filling task in parallel on a plurality of the sequences to be processed to obtain a first sequence corresponding to the sequences to be processed; If there are the first sequences that meet the number of long sequences, the first sequences are processed in blocks; The decoding task is performed on a plurality of the first sequences in parallel.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the response information generating method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the response information generating method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the response information generating method according to any one of claims 1 to 12 are implemented.

Images