A text inference method, product, device and storage medium

Abstract

The application discloses a text reasoning method, product, equipment and storage medium, and relates to the technical field of computers, and comprises the following steps: determining whether the text length of a current text to be reasoned exceeds a preset length, if the text length exceeds the preset length, predicting the length of a text sequence output by the current text to be reasoned after reasoning by a text reasoning model through a text generation sequence length prediction model to obtain a text prediction length, and when the text prediction length is greater than or equal to a preset threshold, combining a current text reasoning request and an unprocessed text reasoning request with a shorter text prediction length, and then scheduling the combined request to a target computing node for text reasoning. In this way, multiple output length longer requests can be avoided from being distributed to the same computing node, load balancing is achieved for processing multiple text reasoning requests, the probability of occurrence of memory occupation of a reasoning framework is effectively reduced, and the performance and efficiency of text reasoning are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to a text reasoning method, product, device and storage medium. BACKGROUND

[0002] Currently, when the inference framework (such as vLLM, SGLang, etc.) of the pre-trained language model infers an ultra-long text, the memory preemption phenomenon often occurs. This phenomenon is a key mechanism triggered by the system to maintain service stability, which can ensure that the inference service will not be interrupted, but significantly affects the inference performance. This is because when a low-priority request is preempted, the corresponding KVCache (Key-Value Cache) cache will be forced to release or copied to the memory. Whether the KVCache cache is forced to release or copied to the memory, there will be a certain time consumption. Especially when the KVCache cache is forced to release, the request will be reprocessed and recalculated, and the already calculated KVCache cache will be refilled. Moreover, when the low-priority request is preempted, the corresponding request will be returned to the waiting area, and the batch processing size of the inference will be smaller, which cannot effectively utilize the computing resources, thereby reducing the computing efficiency of the text reasoning. In addition, in order to avoid the memory preemption phenomenon, the computing power nodes are increased to ensure sufficient memory, but this way increases the cost. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a text reasoning method, product, device and storage medium, which realizes the processing of multiple text reasoning requests by multiple computing node load balancing, can effectively reduce the probability of memory occupation of the inference framework, and improves the performance and efficiency of the text reasoning. The specific scheme is as follows:

[0004] In the first aspect, the present application discloses a text reasoning method, comprising:

[0005] obtaining a current text reasoning request corresponding to a current to-be-reasoned text, and counting the length of the current to-be-reasoned text to obtain a current text length;

[0006] if the current text length is greater than a first length threshold, inputting the current to-be-reasoned text into a text generation sequence length prediction model to predict the length of the text sequence output by the current to-be-reasoned text after being inferred by a text reasoning model, and obtaining a current text prediction length;

[0007] if the current text prediction length is greater than or equal to a second length threshold, combining the current text reasoning request and a to-be-processed text reasoning request before the current time to obtain a combined request; the historical text prediction length corresponding to the to-be-processed text reasoning request is less than the second length threshold;

[0008] The combined request is scheduled to the target computing node to perform text reasoning on the combined request using a text reasoning model.

[0009] Secondly, this application discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-mentioned text reasoning methods.

[0010] Thirdly, this application discloses an electronic device, including a processor and a memory; wherein the processor executes a computer program stored in the memory to implement the aforementioned text reasoning method.

[0011] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned text reasoning method.

[0012] This application first determines whether the length of the current text to be inferred exceeds a preset length. If it does, it further predicts the length of the text sequence output after text inference. When the predicted text length is greater than or equal to a preset threshold, it combines the current text inference request with unprocessed text inference requests with shorter predicted text lengths. Then, it schedules the combined request to the target computing node for text inference. In this way, it avoids distributing multiple requests with long output lengths to the same inference service (i.e., computing node), achieving load balancing for processing multiple text inference requests. This effectively reduces the probability of memory occupancy in the inference framework and improves the performance and efficiency of text inference. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0014] Figure 1 This is a flowchart of a text reasoning method disclosed in this application;

[0015] Figure 2 This application discloses a specific text reasoning flowchart;

[0016] Figure 3 This is a flowchart of a specific text reasoning method disclosed in this application. Detailed Implementation

[0017] With reference to the drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0018] It should be noted that, in the description of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0019] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0020] The embodiments of the present application disclose a text reasoning method, as shown in Figure 1 The method comprises the following steps.

[0021] Step S11: obtaining a current text reasoning request corresponding to a current text to be reasoned, and counting the length of the current text to be reasoned to obtain a current text length.

[0022] It should be noted that the text reasoning scheme proposed in the present application can be applied to an agent node (i.e. Agent node) in a distributed server cluster deployed with a target reasoning framework, wherein the agent node can be any computing node pre-specified in the cluster, and the target reasoning framework includes but is not limited to vLLM (Vectorized Large Language Model Inference), SGLang (Structured Generation Language) and Dynamo (a low-latency distributed inference framework) and other inference frameworks, or a combination of multiple inference frameworks, which can be selected according to the actual application scenario, including but not limited to reading comprehension, intelligent question answering, information extraction, machine translation, article summary and other scenarios.

[0023] Among them, the vLLM inference framework can realize efficient page management of video memory by its Paged Attention technology, support super-long sequence generation. Its dynamic batch processing mechanism and asynchronous scheduling strategy can significantly improve GPU (Graphics Processing Unit) utilization. And it reduces the memory fragmentation rate and reduces redundant calculations through dynamic shared memory. In addition, the vLLM inference framework supports multi-GPU distributed deployment and can be applied to high-concurrency online service (such as financial transactions, intelligent customer service, etc.) scenarios.

[0024] The core advantage of SGLang inference framework is its Radix Attention technology, which can achieve an ultra-high throughput of 100,000 tokens / s through shared prefix requests and efficient caching strategies. Unlike vLLM tokens / s, SGLang inference framework not only optimizes the underlying inference engine, but also provides a domain-specific language (DSL) to simplify the programming logic of complex generation tasks such as structured data extraction and multi-turn dialogue management. Its distributed scheduling capability supports cross-node load balancing, especially suitable for RAG (Retrieval-augmented Generation) and Agent (Agent) workflow scenarios. In addition, the mixed precision calculation (such as FP16 / FP32 intelligent switching) of SGLang inference framework further improves the computing efficiency.

[0025] The core of Dynamo inference framework is the decoupling of prefilling (Prefill) and decoding (Decode) stages, allowing computationally intensive tasks and memory-intensive tasks to be optimized independently. Its smart router (Smart Router) reduces repeated calculations of KVCache, while the distributed KV Cache manager supports offloading low-frequency access caches to CPU (Central Processing Unit) or SSD (Solid State Drive), significantly reducing memory usage. In addition, the Dynamo inference framework also integrates an inference transmission library, optimizing the transmission efficiency of KV Cache between multiple nodes, making it perform well in long-context inference (such as million-token scenarios).

[0026] It can be understood that different reasoning frameworks have different functions, and a new framework formed by combining multiple reasoning frameworks can further optimize the functions of a single reasoning framework. In a specific embodiment, a target reasoning framework obtained by combining a distributed reasoning framework (such as the Dynamo reasoning framework) and a vectorized large language model reasoning framework (i.e., a vLLM reasoning framework) can be used. The reasoning framework can be seamlessly integrated into mainstream reasoning frameworks such as vLLM, SGLang, Dynamo, etc., and is fully compatible with existing API (Application Programming Interface) interfaces.

[0027] In this embodiment, when a text reasoning request input by the current user is received, the current text reasoning request corresponding to the current reasoning text (i.e., the prompt sequence) is first obtained, and then the length of the current reasoning text (i.e., the prompt sequence) is counted to obtain the current text length.

[0028] The text reasoning request can be a reasoning request in different scenarios (such as a reading comprehension request, a medical question and answer request, an article information extraction request, an English-Chinese translation request, an article summary request, an intelligent question and answer request, etc.). For example, write 20 articles about spring outing, and no less than 1000 words.

[0029] Step S12: If the current text length is greater than the first length threshold, the current reasoning text is input into the text generation sequence length prediction model to predict the length of the text sequence output by the current reasoning text after reasoning by the text reasoning model, and obtain the current text prediction length.

[0030] In this embodiment, after counting the length of the current reasoning text, it can be further determined whether the current text length is greater than the first length threshold, that is, whether the length of the text reasoning request input by the current user is greater than the preset length. If the current text length is greater than the first length threshold, it indicates that the length of the current received request is relatively long. At this time, the current reasoning text (i.e., the prompt sequence) can be input into the trained text generation sequence length prediction model, so as to predict the length of the text sequence output by the current reasoning text after reasoning by the text reasoning model through the prediction model, and obtain the current text prediction length. The first length threshold can be selected according to the actual request length determination rule.

[0031] In a specific embodiment, the text generation sequence length prediction model is a model obtained by distilling a self-attention mechanism-based pre-trained language model (i.e., LLM, Large Language Model); the total model parameters of the self-attention mechanism-based pre-trained language model are less than a preset parameter amount, and a binary cross-entropy loss is used for model parameter optimization in the distillation process of the text generation sequence length prediction model. It should be noted that the pre-trained language model (such as a Bert model or a model based on a transformers structure) used by the text generation sequence length prediction model should be a lightweight model, and a single inference should be completed within a short time (such as within 10 ms), and a single request only predicts the length of a text generation sequence once, thereby ensuring that the overall inference performance of the text inference model is not affected. For example, the text generation sequence length prediction model is a model obtained by training a DistilBERT-base model (total parameter amount: 66Mib), which can be trained independently of the text inference model, so that different text inference models can reuse the text generation sequence length prediction model. In addition, it should be noted that the binary cross-entropy loss (Binary Cross-Entropy Loss) can be used to optimize the parameters of the model in the distillation process of the text generation sequence length prediction model.

[0032] Step S13: If the current text prediction length is greater than or equal to the second length threshold, the current text inference request and the unprocessed text inference request before the current time are combined to obtain a combined request; the historical text prediction length corresponding to the unprocessed text inference request is less than the second length threshold.

[0033] In this embodiment, after predicting the length of the text sequence output by the text inference model after inference on the current text to be inferred, it is determined whether the current text prediction length is greater than or equal to the second length threshold. If the current text prediction length is greater than or equal to the second length threshold, it indicates that the length of the text sequence output by the text inference model after inference on the current request to be inferred may be relatively long, and at this time, the historical text inference request (i.e., the request whose length of the text sequence output by the text inference model after inference is possibly relatively short) whose text prediction length is less than the second length threshold before the current time can be combined with the current text inference request.

[0034] It is understandable that when processing a large number of inference requests simultaneously, if a single compute node in the cluster is allocated multiple long inference requests, it may lead to insufficient KV cache space during the computation process, resulting in memory preemption and a decrease in inference performance. To solve the above problem, this application combines current inference requests with longer predicted output sequences with unprocessed inference requests with shorter predicted output sequences. This avoids allocating requests with longer output sequences to the same inference service (i.e., compute node) for text inference, thereby reducing the probability of memory preemption.

[0035] For requests with potentially excessively long output lengths, a target length threshold greater than the second length threshold can be set. If the text prediction length of a request exceeds this target length threshold, there's no need to combine the requests; instead, the request can be directly scheduled to a single computing node for inference. This avoids excessively long requests consuming the computing resources of a single computing node. For example, for inference requests requiring at least 1000 words to write a spring outing article, or those requiring deep thinking and containing thought chains, the output length of these requests can reach tens of thousands of lines, meaning the output sequence is quite long. In this case, such requests can be scheduled to a single computing node. Other inference requests can continue to be combined using the above method, and the combined requests can be scheduled to a computing node for text inference operations.

[0036] In one specific implementation, the current text inference request and the unprocessed text inference requests before the current time are combined to obtain a combined request. Specifically, this may include: obtaining the historical text prediction length corresponding to each unprocessed text inference request before the current time, and sorting the multiple unprocessed text inference requests in ascending order of historical text prediction length to obtain a first sorted inference request; combining the current text inference request and the first sorted inference request with the first preset number of unprocessed text inference requests to obtain a combined request. In this embodiment, if there is only one unprocessed text inference request before the current time (corresponding to a historical text prediction length less than the second length threshold), the current text inference request and the unprocessed text inference request can be directly combined, and then text inference is performed on the combined request; if there are multiple unprocessed text inference requests before the current time (corresponding to a historical text prediction length less than the second length threshold), the unprocessed text inference requests can be sorted in ascending order of historical text prediction length, and then the current text inference request and the first preset number (e.g., the first 3) of the sorted inference requests can be combined.

[0037] Understandably, for requests with short prompt sequences, such as those asking how high Mount Tai is or how long the Yangtze River is, the output lengths are relatively short. Therefore, these requests can be combined according to a preset number. For example, if the current text inference request is 'a', and the unprocessed text inference requests in the sorted inference requests are b, c, d, e, f, and g, then the request after combining them according to a preset number (such as the first three) is a + b + c + d. By combining requests with potentially longer output sequences with those with potentially shorter output sequences, it's possible to ensure that when the cluster processes multiple inference requests simultaneously, the length of the output sequences received by each computing node is roughly the same. This fully utilizes the computing resources of a single computing node, achieving load balancing across multiple nodes to process inference requests, thereby reducing the probability of memory preemption and ensuring the inference performance of the inference framework. Furthermore, it can further meet the needs of more diverse text inference scenarios. Based on load balancing, it rationally utilizes the computing resources of each computing node, thereby improving the response speed of inference requests and further enhancing the user experience.

[0038] In another specific implementation, the current text inference request and unprocessed text inference requests before the current time are combined to obtain a combined request. Specifically, this may include: sorting multiple unprocessed text inference requests before the current time according to request priority to obtain a second sorted inference request; and combining the current text inference request and the first preset number of unprocessed text inference requests from the second sorted inference request to obtain the combined request. In this embodiment, for application scenarios with multiple unprocessed text inference requests, multiple unprocessed text inference requests before the current time can be sorted according to request priority, and then the current text inference request and the first preset number (e.g., the first 2) of unprocessed text inference requests from the sorted inference requests can be combined. Through the above combination method, both the length of the output sequence and the priority of the inference requests are considered, thereby satisfying the inference requests of users with different priorities as much as possible on the basis of load balancing, thus improving the user experience.

[0039] In another specific embodiment, the current text reasoning request and the unprocessed text reasoning requests before the current time are combined to obtain a combined request, which can specifically include: sorting the multiple unprocessed text reasoning requests before the current time according to the chronological order of the request initiation time to obtain third sorted reasoning requests; and combining the current text reasoning request and the first preset number of unprocessed text reasoning requests in the third sorted reasoning requests to obtain the combined request. In this embodiment, for the application scenario of multiple unprocessed text reasoning requests, the multiple unprocessed text reasoning requests before the current time can be sorted according to the chronological order of the request initiation time (such as the timestamp of the request initiation), and then the current text reasoning request and the first preset number (such as the first 5) of unprocessed text reasoning requests in the sorted reasoning requests can be combined. Through the above combination method, both the length of the output sequence and the time sequence of the reasoning request are considered, so that the client that initiates the reasoning request first can be responded to preferentially on the basis of load balancing, the user that sends the request first can obtain the corresponding text reasoning result as soon as possible, and thus the user experience is improved.

[0040] In another specific embodiment, when the current text reasoning request and the unprocessed text reasoning requests before the current time are combined, the above multiple factors (such as the text prediction length, the request priority, and the chronological order of the request initiation time) can also be considered simultaneously. Through the above combination method, both the length of the output sequence and the time and user priority of the reasoning request are considered, so that the client that sends the reasoning request first and has a high priority can be responded to preferentially on the basis of load balancing, and thus the user experience is further improved. In addition, through the combination of multiple unprocessed text reasoning requests, the processing efficiency of the unprocessed text reasoning requests can be improved.

[0041] Specifically, if the current text prediction length is greater than or equal to the second length threshold, the current text reasoning request and the unprocessed text reasoning request before the current time are combined to obtain a combined request, which can include: determining whether the current text prediction length is greater than a third length threshold, if the current text prediction length is greater than the third length threshold, determining whether the current text prediction length is greater than or equal to the second length threshold; the third length threshold is used for selecting the node separation strategy or the node fusion strategy, and the third length threshold is less than the second length threshold; if the current text prediction length is greater than or equal to the second length threshold, the current text reasoning request and the unprocessed text reasoning request before the current time are combined to obtain a combined request. In this embodiment, before the current text reasoning request and the unprocessed text reasoning request are combined, the node separation strategy or the node fusion strategy can be selected first. Specifically, it can be determined whether the current text prediction length is greater than the third length threshold, if it is greater (the node separation strategy is selected in subsequent text reasoning), it is further determined whether the current text prediction length is greater than or equal to the second length threshold (that is, it is determined whether the output sequence of the current text reasoning request is long). The third length threshold can be selected according to the actual node separation strategy and the node fusion strategy, but it needs to satisfy that the third length threshold is less than the second length threshold.

[0042] It should be noted that before text reasoning, specifically, it further includes: deploying a text reasoning model on each computing node in the distributed server cluster, and initializing each computing node in the distributed server cluster; creating a key-value cache space in the acceleration card of each initialized computing node. In this embodiment, before text reasoning, the text reasoning model needs to be deployed on each computing node of the distributed server cluster, and each computing node in the cluster is initialized, and then a key-value cache space (i.e., a KV Cache cache space) of a preset size is created in the AI (Artificial Intelligence) acceleration card of each initialized computing node, that is, a certain cache space is reserved in advance. It should be noted that the target computing node for text reasoning of the current combined request is any node in the distributed server cluster.

[0043] Specifically, the distributed server cluster adopts a target inference framework; the target inference framework is a framework obtained by combining a distributed inference framework (such as a Dynamo inference framework) and a vectorized large language model inference framework (that is, a vllm inference framework); and after creating a key-value cache space in the acceleration card of each computing node after initialization, the computing nodes can also be divided into prefill nodes and decode nodes; wherein the number of decode nodes is greater than the number of prefill nodes. That is, the computing nodes in the cluster are divided into two types of nodes, namely prefill nodes (Prefill nodes) and decode nodes (Decode nodes), and the number of Decode nodes is greater than that of Prefill nodes. Specifically, the proxy node can be any one node selected from a plurality of Prefill nodes.

[0044] Among them, the Prefill node is used to receive the user's request, that is, the prompt (token sequence), and perform parallel calculation on the entire prompt to generate embedding (embedding) and attention (attention mechanism) information for each token (word), and build a key context state, that is, KV Cache, and the output of the Prefill node is KV Cache.

[0045] The Decode node is used to receive the KV Cache output by the Prefill node, and predict the next token based on the KV Cache, then add the newly generated token to the sequence, and update the KV Cache, repeat the process until the complete reply is generated or the preset stopping condition is reached. It can be seen that the Decode node needs to frequently read, write and update the KV Cache.

[0046] It can be understood that the text inference is performed in a PD separation manner using prefill nodes (Prefill nodes) and decode nodes (Decode nodes), that is, a PD separation manner, which includes two stages, namely a prefill (Prefill) stage and a decode (Decode) stage. In the prefill (Prefill) stage, the KV Cache of the prompt is mainly calculated, the calculation amount is large, and it belongs to a compute-intensive task; the decode (Decode) stage is to calculate the KV Cache of each generated sequence step by step, and the number of calls is large, which belongs to a memory-intensive task. When performing long text inference, PD separation will have higher performance benefits, but when inferring short text, parameter communication will cause performance degradation. When the inference service receives an inference task and performs sequence inference, it is not clear how many sequences to generate, and the sequence length is determined during the generation process. If the length of the text inference is predicted when the inference service receives the inference task (the manner adopted by the present application), then the access frequency to the KV Cache can be reduced through differentiated scheduling.

[0047] For example, the input to the user question "How tall is Mount Tai?" is 5 words. The calculation is performed in the prefill node, and the model infers once. The corresponding answer is "1545 meters". The output has 6 words plus a terminator. The model infers 7 times. The decode node outputs 1 word for each inference. The decode node is executed a total of 7 times.

[0048] In another specific implementation, the target reasoning framework can also be a framework obtained by combining the Dynamo reasoning framework and the SGlang reasoning framework.

[0049] For distributed server clusters, predicting the length of the generated text sequence can reduce the hyperparameter of KV Cache memory usage for some services. This reduces the amount of GPU memory occupied by the KV Cache, freeing up more GPU memory for model loading and thus reducing the number of AI accelerator cards required for model deployment. Furthermore, as the number of AI accelerator cards decreases, the number of model communication parameters also decreases, thereby improving model performance. Additionally, reducing the number of AI accelerator cards also lowers the cost of model deployment.

[0050] In this embodiment, the combined request is scheduled to the target computing node to perform text reasoning on the combined request using a text reasoning model. Specifically, this may include: scheduling the combined request to the target computing node in the distributed server cluster through a load balancer, adopting a node separation strategy and using a text reasoning model to perform text reasoning on the combined request. See also: Figure 2As shown, for a text inference scenario using a combination of a distributed inference framework (such as a Dynamo inference framework) and a vectorized large language model inference framework (i.e., a vllm inference framework), if the request length corresponding to the current text inference request is greater than a first threshold value, and the predicted output length is greater than a second threshold value, the combined request can be scheduled to a target computing node (which can be any Prefill node) in the distributed server cluster through a node (worker) load balancing scheduler; when the target computing node receives the combined request, it can use a node separation strategy and utilize the pre-deployed text inference model in the target computing node to perform text inference operation on the combined request. Among them, the node separation strategy is to use the inference engine in the pre-filled node (i.e., Prefill node) in the distributed server cluster to calculate the key-value cache (i.e., KV Cache) of each word (token) in the text to be inferred, and then receive the key-value cache (i.e., KV Cache) data output by the Prefill node through the decoding node (Decode node), and then calculate the KV Cache through the inference engine in the Decode node based on the key-value cache data (i.e., KV Cache data) generated by the Prefill node to predict the next token, that is, the KV Cache calculated by the Prefill node is passed to the Decode node to perform text sequence generation; when the prediction of all tokens is completed, the final text inference result can be obtained.

[0051] In another specific embodiment, it can also include: if the current text length is not greater than the first length threshold or the current text prediction length is not greater than the third length threshold, the current text inference request is scheduled to a target computing node in the distributed server cluster through a load balancing scheduler to use a node fusion strategy and utilize a text inference model to perform text inference on the current text inference request; wherein the node fusion strategy is to use the pre-filled node in the distributed server cluster to calculate the key-value cache of each word in the text to be inferred, and to predict the text sequence based on the generated key-value cache data. In this embodiment, referring to Figure 2 As shown, if the request length of the current text inference request is not greater than the first threshold value (i.e., the length of the current text inference request is longer), or the predicted output length is not greater than the second threshold value (i.e., the length of the current text inference request is shorter, but the length of the output sequence can be longer), the current text inference request is scheduled to a target computing node (which can be any Prefill node) in the distributed server cluster through a node (worker) load balancing scheduler, so as to perform text inference on the current text inference request through the target computing node and using a node fusion strategy and a text inference model.

[0052] It's important to note that the node fusion strategy refers to utilizing prefill nodes in a distributed server cluster to compute the key-value cache (KV Cache) of each word (token) in the text to be inferred, and then predicting the text sequence based on the generated KV Cache data. In other words, if the text inference request length is short, or the predicted output length is short, the KV Cache operations of the Prefill and Decode phases are performed on the same prefill node. Thus, in the Decode phase, the prefill node is equivalent to the Decode node performing sequence prediction. This allows for more efficient use of the computing resources of the cluster's nodes for text inference, even when the output sequence is not particularly long, thereby further improving the efficiency of text inference.

[0053] It is evident that when the request length is short or the request itself is long but the output sequence is short, using a node fusion strategy (i.e., treating the Prefill node as a Decode node for KV Cache calculation) for text inference can reduce the number of KV Cache parameter transmissions, thereby improving the response time when the inference model outputs a short sequence, which in turn improves the speed of text inference.

[0054] Further, see Figure 2 As shown, it may also include: determining whether the text inference model has completed the inference operation for the combined request; if the text inference model has completed the inference operation for the combined request, then updating the state of the load balancer. That is, after completing the entire text inference operation for the combined request, the state of the load balancer is updated to indicate that the current inference has been completed.

[0055] Step S14: Schedule the combined request to the target computing node to perform text reasoning on the combined request using the text reasoning model.

[0056] In this embodiment, after combining the current text inference request with a longer text prediction length and the unprocessed text inference request with a shorter text prediction length, the combined request can be scheduled to the target computing node in the distributed server cluster by the load balancer. The text inference model deployed on the target computing node can then be used to perform text inference on the combined request to obtain the text inference result for the combined request. The text inference result is then sent to the corresponding user terminal to complete the response to the corresponding inference request.

[0057] The text inference model is a model obtained by training a pre-trained language model (LLM) using a historical text data set. For example, a multimodal large language model (MLLM) is trained using a historical medical question and answer data set to obtain an inference model for answering different medical questions.

[0058] It can be seen that the embodiment of the present application first determines whether the text length of the current text to be inferred exceeds the preset length. If it exceeds, the length of the text sequence output after the current text to be inferred is inferred is further predicted, and when the text prediction length is greater than or equal to the preset threshold, the current text inference request and the unprocessed text inference request with a shorter text prediction length are combined, and then the combined request is dispatched to the target computing node for text inference. In this way, multiple requests with a longer output length can be avoided from being allocated to the same inference service (i.e., computing node), thereby achieving load balancing for processing multiple text inference requests, effectively reducing the probability of occurrence of inference framework video memory occupation, and improving the performance and efficiency of text inference.

[0059] The embodiment of the present application discloses a specific text inference method applied to a proxy node in a distributed server cluster using a target inference framework, as shown in Figure 3 The method comprises the following steps:

[0060] Step S21: obtaining a current text to be inferred corresponding to a current text inference request, and counting the length of the current text to be inferred to obtain a current text length.

[0061] Step S22: if the current text length is greater than a first length threshold, inputting the current text to be inferred into a text generation sequence length prediction model to predict the length of the text sequence output after the current text to be inferred is inferred by a text inference model, and obtaining a current text prediction length.

[0062] Step S23: if the current text prediction length is greater than or equal to a second length threshold, combining the current text inference request and an unprocessed text inference request at a current time to obtain a combined request, and the historical text prediction length corresponding to the unprocessed text inference request is less than the second length threshold.

[0063] Step S24: dispatching the combined request to a target computing node to infer the combined request using a text inference model, and in the inference process, monitoring the length of the output sequence generated by the text inference model in the inference process in real time to obtain a real-time inference sequence length.

[0064] In the embodiment, in the process of performing text inference on the combined request by the text inference model deployed in the target computing node, the length of the output sequence generated by the current text inference model in the inference process is counted in real time to obtain a real-time inference sequence length.

[0065] Step S25: Determine whether the real-time inference sequence length is equal to the sum of the multiple text prediction lengths corresponding to the combined request.

[0066] In the embodiment, after counting the real-time inference sequence length, the real-time inference sequence length is compared with the multiple text prediction lengths corresponding to the combined request, and it is determined whether the real-time inference sequence length reaches the sum of the multiple text prediction lengths corresponding to the combined request.

[0067] For example, when the combined request is a + b (where a is the current text inference request and b is the unprocessed text inference request before the current time), the corresponding two text prediction lengths are a' and b', and the sum of the text prediction lengths is a' + b'. In the inference process, it can be determined in real time whether the inference sequence length is equal to a' + b'.

[0068] For another example, when the combined request is a + b + c + d (where a is the current text inference request and b, c, and d are the unprocessed text inference requests before the current time), the corresponding four text prediction lengths are a', b', c', and d', and the sum of the text prediction lengths is a' + b' + c' + d'. In the inference process, it can be determined in real time whether the inference sequence length is equal to a' + b' + c' + d'.

[0069] Step S26: If the real-time inference sequence length is equal to the sum of the multiple text prediction lengths corresponding to the combined request, a termination symbol is generated and the current inference process is ended to prohibit the text inference model from repeatedly outputting.

[0070] In the embodiment, considering that the traditional long thought chain prediction is prone to repeated output, which is a general problem of large models, especially in the reinforcement learning process, if there is repeated output, it will cause the trained model to diverge. In order to solve the problem that the model repeatedly outputs and cannot normally terminate when performing inference and output on a long thought chain, a termination symbol can be forced to be generated when the two length values (i.e., ) are equal (i.e., the generated text length is equal to the predicted text length), thereby ending the current text inference process. In this way, the text inference model can be effectively prevented from repeatedly outputting.

[0071] The more specific processing procedures of the above steps S21 to S23 can be referred to the corresponding contents disclosed in the foregoing embodiments, which will not be described here in detail.

[0072] It can be seen that, in view of the problem that the long thinking chain model output is repeatedly output and cannot be normally terminated, the traditional strategy is to continuously optimize the prompt word during training, and the training cost of this way is high, while the application uses a text generation sequence length prediction model to predict the output length of the reasoning model, and forces the generation of a termination symbol when the generated sequence length reaches the predicted sequence length, thereby ending the reasoning process, so that the rationality of the model output result can be ensured, and the repetition phenomenon can be effectively reduced.

[0073] Correspondingly, the embodiment of the application also discloses a text reasoning device, which comprises:

[0074] a text acquisition module, configured to acquire a current to-be-reasoned text corresponding to a current text reasoning request;

[0075] a statistics module, configured to count the length of the current to-be-reasoned text to obtain a current text length;

[0076] a length prediction module, configured to input the current to-be-reasoned text into a text generation sequence length prediction model if the current text length is greater than a first length threshold, so as to predict the length of a text sequence output by the current to-be-reasoned text after reasoning by a text reasoning model, and obtain a current text predicted length;

[0077] a combination module, configured to combine the current text reasoning request and an unprocessed text reasoning request before the current time to obtain a combined request if the current text predicted length is greater than or equal to a second length threshold; the unprocessed text reasoning request corresponds to a historical text predicted length less than the second length threshold;

[0078] a request scheduling module, configured to schedule the combined request to a target computing node, so as to perform text reasoning on the combined request by using the text reasoning model.

[0079] The specific working procedures of the above-mentioned various modules can refer to the corresponding content disclosed in the foregoing embodiments, and will not be described here in detail.

[0080] The embodiment of the application also provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above text reasoning method embodiments.

[0081] The embodiment of the application also provides a computer readable storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above text reasoning method embodiments when running.

[0082] In an example embodiment, the computer readable storage medium described above can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0083] Embodiments of the present application also provide a computer program product, which comprises a computer program. The computer program is executed by a processor to implement the steps in any of the text inference method embodiments described above.

[0084] Embodiments of the present application also provide another computer program product, which comprises a non-volatile computer readable storage medium. The non-volatile computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps in any of the text inference method embodiments described above.

[0085] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0086] The above describes in detail a text inference method, product, device and storage medium provided by the present application. The principles and implementation modes of the present application are described by applying specific examples. The above description of the examples is only to help understand the method and core idea of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application. These improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A text-based reasoning method, characterized in that, include: Obtain the current text to be inferred corresponding to the current text inference request, and calculate the length of the current text to be inferred to obtain the current text length; If the current text length is greater than the first length threshold, the current text to be inferred is input into the text generation sequence length prediction model to predict the length of the text sequence output by the text inference model after the current text to be inferred, and thus obtain the current text prediction length. If the current text prediction length is greater than or equal to the second length threshold, then the current text inference request and the unprocessed text inference request before the current time are combined to obtain a combined request; the historical text prediction length corresponding to the unprocessed text inference request is less than the second length threshold. The combined request is scheduled to the target computing node to perform text reasoning on the combined request using a text reasoning model.

2. The text reasoning method according to claim 1, characterized in that, The step of combining the current text reasoning request and the unprocessed text reasoning requests before the current time to obtain the combined request includes: Obtain the historical text prediction length corresponding to each unprocessed text inference request before the current time, and sort the multiple unprocessed text inference requests in ascending order of the historical text prediction length to obtain the first sorted inference request; The current text reasoning request and the first preset number of unprocessed text reasoning requests in the first sorted reasoning request are combined to obtain a combined request.

3. The text reasoning method according to claim 1, characterized in that, The step of combining the current text reasoning request and the unprocessed text reasoning requests before the current time to obtain the combined request includes: The inference requests for the second time are sorted according to their priority to obtain the inference requests after the second sort. The current text reasoning request and the first preset number of unprocessed text reasoning requests in the second sorted reasoning request are combined to obtain a combined request.

4. The text reasoning method according to claim 1, characterized in that, The step of combining the current text reasoning request and the unprocessed text reasoning requests before the current time to obtain the combined request includes: The inference requests are sorted according to the order in which they were initiated, and the third sorted inference requests are obtained. The current text reasoning request and the first preset number of unprocessed text reasoning requests in the third sorted reasoning request are combined to obtain a combined request.

5. The text reasoning method according to claim 1, characterized in that, Also includes: Deploy the text inference model on each computing node in the distributed server cluster, and initialize each computing node in the distributed server cluster; The target computing node is any node in the distributed server cluster; Create key-value cache space in the accelerator cards of each compute node after initialization.

6. The text reasoning method according to claim 5, characterized in that, The distributed server cluster employs a target inference framework. The target reasoning framework is a framework obtained by combining a distributed reasoning framework and a vectorized large-scale language model reasoning framework. After creating key-value cache spaces in the accelerator cards of each computing node after initialization, the process also includes: The computing nodes are divided into pre-filled nodes and decoding nodes; wherein the number of decoding nodes is greater than the number of pre-filled nodes.

7. The text reasoning method according to claim 6, characterized in that, If the current text prediction length is greater than or equal to the second length threshold, then the current text inference request and the unprocessed text inference request before the current time are combined to obtain a combined request, including: Determine whether the current text prediction length is greater than a third length threshold. If the current text prediction length is greater than the third length threshold, then determine whether the current text prediction length is greater than or equal to a second length threshold. The third length threshold is used to select a node separation strategy or a node fusion strategy, and the third length threshold is less than the second length threshold. If the current text prediction length is greater than or equal to the second length threshold, then the current text inference request and the unprocessed text inference request before the current time are combined to obtain a combined request.

8. The text reasoning method according to claim 7, characterized in that, The step of scheduling the combined request to the target computing node to perform text reasoning on the combined request using a text reasoning model includes: The combined request is scheduled to the target computing node in the distributed server cluster by the load balancer, so as to adopt the node separation strategy and use the text reasoning model to perform text reasoning on the combined request. The node separation strategy involves using pre-filled nodes in the distributed server cluster to calculate the key-value cache of each word in the text to be inferred, and then using decoding nodes to predict the text sequence based on the key-value cache data generated by the pre-filled nodes.

9. The text reasoning method according to claim 7, characterized in that, Also includes: If the current text length is not greater than the first length threshold or the current text prediction length is not greater than the third length threshold, then the current text inference request is scheduled to the target computing node in the distributed server cluster by the load balancer, so as to use the node fusion strategy and the text inference model to perform text inference on the current text inference request. The node fusion strategy involves using pre-filled nodes in the distributed server cluster to calculate the key-value cache of each word in the text to be inferred, and then predicting the text sequence based on the generated key-value cache data.

10. The text reasoning method according to claim 8, characterized in that, Also includes: Determine whether the text reasoning model has completed the reasoning operation for the combined request; If the text reasoning model has completed the reasoning operation for the combined request, then the state of the load balancer is updated.

11. The text reasoning method according to claim 1, characterized in that, The text generation sequence length prediction model is a model obtained by distilling a pre-trained language model based on a self-attention mechanism. The total number of model parameters of the pre-trained language model based on the self-attention mechanism is less than the preset number of parameters, and the binary cross-entropy loss is used to optimize the model parameters during the distillation process of the text generation sequence length prediction model.

12. The text reasoning method according to any one of claims 1 to 11, characterized in that, The process of performing text reasoning on the combined request using a text reasoning model also includes: The length of the output sequence generated by the text reasoning model during the reasoning process is monitored in real time to obtain the real-time reasoning sequence length. Determine whether the length of the real-time inference sequence is equal to the sum of the lengths of the multiple text predictions corresponding to the combined request; If the length of the real-time inference sequence is equal to the sum of the lengths of the multiple text predictions corresponding to the combined request, a terminator is generated and the current inference process ends to prevent the text inference model from repeatedly outputting.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the text reasoning method as described in any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the text reasoning method as described in any one of claims 1 to 12.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the text reasoning method as described in any one of claims 1 to 12.

Images