Inference request processing methods, electronic devices, storage media, and computer program products

Abstract

This disclosure provides an inference request processing method, an electronic device, a storage medium, and a computer program product. Belonging to the field of computer technology, the inference request processing method includes: receiving an inference request; determining the request type of the inference request; and classifying the inference request as either a first type of inference request or a second type of inference request. In the model pre-filling stage, the first type of inference request is processed using a linear layer with quantized computation and a sparse attention mechanism to determine a first key-value cache set, or the second type of inference request is fully computed to determine a second key-value cache set. In the model decoding stage, the inference result of the first type of inference request is determined based on the target key-value cache in the first key-value cache set determined by the attention scoring function, or the inference result of the second type of inference request is determined based on all key-value caches in the second key-value cache set.

Description

Technical Field

[0001] This disclosure belongs to the field of computer technology, and in particular relates to a reasoning request processing method, electronic device, storage medium and computer program product. Background Technology

[0002] In large-scale model inference scenarios, as users' demands for real-time performance and accuracy become more diverse, existing technologies face the challenge of balancing processing efficiency with result accuracy. On the one hand, low-precision inference requests (such as real-time interaction and rapid response scenarios) are latency-sensitive, but general-purpose inference frameworks often employ a uniform full-scale computation model without optimization for these characteristics, leading to wasted computing resources and response delays. On the other hand, high-precision inference requests (such as medical diagnosis and financial analysis scenarios) have extremely high requirements for result accuracy, while traditional key-value caching reuse mechanisms may reduce accuracy due to accumulated errors (such as feature offset and cache aging), failing to meet stringent requirements. Furthermore, existing key-value caching technologies mostly reuse data indiscriminately, lacking differentiated designs for requests of varying precision, making it difficult to adapt to the efficiency requirements of low-precision scenarios while also failing to guarantee the accuracy integrity of high-precision scenarios. Summary of the Invention

[0003] This disclosure provides a reasoning request processing method, an electronic device, a storage medium, and a computer program product.

[0004] According to one aspect of this disclosure, a method for processing inference requests is provided. The inference request is processed through a model. The method includes: receiving an inference request; determining the request type of the inference request; classifying the inference request as a first type of inference request or a second type of inference request, wherein the first type of inference request is a low-precision inference request and the second type of inference request is a high-precision inference request. In the model pre-filling stage, the first type of inference request is processed using a linear layer of quantized computation and a sparse attention mechanism to determine a first key-value cache set corresponding to the first type of inference request, or a full computation is performed on the second type of inference request to determine a second key-value cache set corresponding to the second type of inference request. In the model decoding stage, the inference result of the first type of inference request is determined based on a target key-value cache in the first key-value cache set determined by an attention scoring function, or the inference result of the second type of inference request is determined based on all key-value caches in the second key-value cache set.

[0005] The inference request processing method according to this embodiment can intelligently differentiate between low-precision first-type inference requests and high-precision second-type inference requests based on request type, significantly improving resource utilization efficiency while ensuring service quality and user experience. In the pre-filling stage, a linear layer of quantized computation and a sparse attention mechanism are used for first-type inference requests to reduce computational load and memory consumption, determining a lightweight first key-value cache set. For second-type inference requests, full computation is performed to ensure maximum accuracy, determining a comprehensive second key-value cache set. Upon entering the decoding stage, the target key-value cache is selected from the first key-value cache set based on an attention scoring function, quickly generating the result for the first-type inference request. The second-type inference request is processed directly based on the complete second key-value cache set, ensuring accuracy. This flexible scheduling and optimization strategy not only enables a single large language model instance to support multiple request levels but also reduces overall computational resource consumption, simplifies system deployment and maintenance processes, and ultimately provides a more efficient, accurate, and responsive service experience.

[0006] According to at least one embodiment of the inference request processing method of this disclosure, in the pre-filling stage of the model, the first type of inference request is processed using a quantized linear layer and a sparse attention mechanism, including: converting the weights and activation values ​​of the linear layer into low-precision values ​​based on a target quantization precision; performing forward propagation computation on the first type of inference request through the quantized linear layer; and performing matrix operations on the first type of inference request using a sparse attention mechanism.

[0007] According to the technical solution of this embodiment, by converting the weights and activation values ​​of the linear layer into low-precision values ​​and using these quantized parameters for calculation during forward propagation, the computational complexity and memory consumption of the model are significantly reduced. Furthermore, by performing sparse matrix operations on the input sequence through a sparse attention mechanism, redundant computations are further reduced.

[0008] According to at least one embodiment of the inference request processing method of this disclosure, determining the inference result of the first type of inference request based on a target key-value cache in the first key-value cache set determined by an attention scoring function includes: during the decoding phase of the model, determining an attention score of a first key-value cache in the first key-value cache set based on the attention scoring function; selecting a first key-value cache set whose attention score is greater than or equal to a target score as a target key-value cache; and determining the inference result of the first type of inference request based on the target key-value cache.

[0009] According to the technical solution of this embodiment, the key context information in low-precision inference tasks can be efficiently identified and utilized. While significantly reducing the amount of computation and memory access, the most critical information for the current inference task can be retained, thereby improving decoding efficiency and inference speed.

[0010] According to at least one embodiment of the inference request processing method of this disclosure, in the decoding stage of the model, determining the attention score of a first key-value cache in the first key-value cache set based on an attention scoring function includes: determining a query vector based on context information of the first type of inference request; processing the first key-value cache in the first key-value cache set based on a linear layer and a self-attention mechanism in the decoding stage to determine key-value cache entries for intermediate state information of the first key-value cache; the key-value cache entries include key vectors and corresponding value vectors; and comparing the similarity between the query vector and the key vector based on the attention scoring function to determine the attention score of the first key-value cache in the first key-value cache set.

[0011] According to the technical solution of this embodiment, the key-value cache most relevant to the current decoding task can be efficiently selected.

[0012] According to at least one embodiment of the inference request processing method of this disclosure, a full computation is performed on the second type of inference request to determine a second key-value cache set corresponding to the second type of inference request, including: performing a linear transformation on the second type of inference request through a target weight; performing attention calculation on the linearly transformed second type of inference request to determine a second key-value cache set for the second type of inference request.

[0013] According to the technical solution of this embodiment, the semantic information and relationships of the input context can be fully preserved without introducing any loss of accuracy, thus ensuring higher accuracy and stability when processing complex tasks.

[0014] The inference request processing method according to at least one embodiment of this disclosure further includes: processing the first type of inference request or the second type of inference request in a single batch during a pre-filling stage to determine the corresponding first key-value cache set or second key-value cache set; and processing the first key-value cache set or the second key-value cache set in multiple batches during a decoding stage to determine the corresponding inference result.

[0015] According to the technical solution of this embodiment, the initial computational overhead is effectively controlled in the pre-filling stage, and the throughput and response flexibility of the system are improved through multi-batch processing in the decoding stage, thereby improving the efficiency and scalability of the overall inference system while ensuring service quality.

[0016] According to at least one embodiment of the inference request processing method of this disclosure, the GPU memory utilization and CPU queue depth are monitored in real time, and the batch size is adjusted through a sliding window mechanism, including: increasing the batch size based on the sliding window mechanism when the GPU memory utilization is less than a first target threshold and the CPU queue depth is greater than or equal to a second target threshold; and decreasing the batch size based on the sliding window mechanism when the GPU memory utilization is greater than or equal to a third target threshold or the CPU queue depth is less than a fourth target threshold.

[0017] According to the technical solution of this embodiment, when GPU resources are sufficient and CPU tasks are plentiful, the batch size is automatically increased to improve throughput and GPU utilization. Conversely, when GPU memory is nearing saturation or CPU tasks are few, the batch size is reduced to avoid resource overload and reduce task latency. This effectively balances the relationship between system throughput and response speed, improving the stability of the inference service under high concurrency and fluctuating load scenarios.

[0018] The inference request processing method according to at least one embodiment of this disclosure further includes: performing static analysis on the operators in the model to determine control-intensive operators and parallel-intensive operators; allocating control-intensive operators to the CPU for execution and allocating parallel-intensive operators to the GPU for execution.

[0019] According to the technical solution of this embodiment, the advantages of CPU in processing logic control tasks and the high performance characteristics of GPU in large-scale parallel computing are brought into play, thereby effectively improving the overall inference efficiency and reducing inference latency.

[0020] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform a reasoning request processing method according to any embodiment of this disclosure.

[0021] According to another aspect of this disclosure, a storage medium is provided that stores execution instructions, which, when executed by a processor, are used to implement the inference request processing method of any embodiment of this disclosure.

[0022] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the inference request processing method of any embodiment of this disclosure.

[0023] The beneficial effects of the technical solution according to this application are at least as follows: it can intelligently differentiate between low-precision first-type inference requests and high-precision second-type inference requests based on request type, significantly improving resource utilization efficiency while ensuring service quality and user experience. In the pre-filling stage, a linear layer of quantized computation and a sparse attention mechanism are used for first-type inference requests to reduce computational load and memory consumption, determining a lightweight first key-value cache set. For second-type inference requests, full computation is performed to ensure maximum accuracy, determining a comprehensive second key-value cache set. Upon entering the decoding stage, the target key-value cache is selected from the first key-value cache set based on the attention scoring function, quickly generating the result for the first-type inference request, while the second-type inference request is directly processed based on the complete second key-value cache set, ensuring accuracy. This flexible scheduling and optimization strategy not only enables a single large language model instance to support multiple request levels but also reduces overall computational resource consumption, simplifies system deployment and maintenance processes, and ultimately provides a more efficient, accurate, and responsive service experience. Attached Figure Description

[0024] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0025] Figure 1 This is a schematic diagram of the overall flow of a reasoning request processing method according to one embodiment of the present disclosure.

[0026] Figure 2 This is a flowchart illustrating the processing of a first type of reasoning request in the pre-filling stage of a reasoning request processing method according to one embodiment of the present disclosure.

[0027] Figure 3 This is a schematic diagram of the process for determining a second key-value cache set in a reasoning request processing method according to one embodiment of the present disclosure.

[0028] Figure 4 This is a flowchart illustrating the process of determining the reasoning result of a first type of reasoning request in a reasoning request processing method according to one embodiment of the present disclosure.

[0029] Figure 5 This is a flowchart illustrating the process of determining the attention score of the first key-value cache in an inference request processing method according to one embodiment of the present disclosure.

[0030] Figure 6 This is a schematic block diagram of the structure of a reasoning request processing apparatus according to one embodiment of the present disclosure.

[0031] Figure 7This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0032] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0033] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] In practical applications, users may need to handle both simple tasks with extremely high response time requirements (such as instant chatbots) and complex analytical tasks with strict accuracy requirements (such as professional document translation or deep data analysis). Current technical solutions typically require deploying multiple model instances with different configurations to handle these needs separately. This not only increases the consumption of computing resources and management costs, but also limits the flexibility and scalability of the system.

[0035] To address this, this disclosure proposes the following technical solution: receiving an inference request, determining the request type of the inference request, and classifying the inference request as either a first type of inference request or a second type of inference request, wherein the first type of inference request is a low-precision inference request and the second type of inference request is a high-precision inference request. In the model pre-filling stage, a linear layer with quantized computation and a sparse attention mechanism are used to process the first type of inference request to determine a first key-value cache set corresponding to the first type of inference request, or a full computation is performed on the second type of inference request to determine a second key-value cache set corresponding to the second type of inference request. And in the model decoding stage, the inference result of the first type of inference request is determined based on the target key-value cache in the first key-value cache set determined by the attention scoring function, or the inference result of the second type of inference request is determined based on all key-value caches in the second key-value cache set.

[0036] In the technical solution of this application, low-precision first-type inference requests and high-precision second-type inference requests can be intelligently differentiated according to the request type, significantly improving resource utilization efficiency while ensuring service quality and user experience. In the pre-filling stage, a linear layer of quantized computation and a sparse attention mechanism are used for first-type inference requests to reduce computational load and memory consumption, determining a lightweight first key-value cache set; while for second-type inference requests, full computation is performed to ensure the highest accuracy, determining a comprehensive second key-value cache set. Upon entering the decoding stage, the target key-value cache is selected from the first key-value cache set based on the attention scoring function, quickly generating the result for the first-type inference request, while the second-type inference request is directly processed based on the complete second key-value cache set, ensuring accuracy. This flexible scheduling and optimization strategy not only enables a single large language model instance to support multiple request levels but also reduces overall computational resource consumption, simplifies system deployment and maintenance processes, and ultimately provides a more efficient, accurate, and responsive service experience.

[0037] To facilitate description and make the technical solutions of this disclosure easier to understand, the terminology of this disclosure will be explained before describing the technical solutions of this disclosure.

[0038] The prefill phase occurs after the AI ​​model receives the complete input request, but before it begins generating the first output token. The main tasks of this phase are processing the input request, computing the context representation of all input tokens, and initializing the key-value cache required for the subsequent decoding phase.

[0039] The decoding phase begins after the pre-filling phase is complete. In this phase, the model generates output tokens one by one in an autoregressive manner. Each generated token is added to the already generated sequence and used as input for the next generation.

[0040] A linear layer, also known as a fully connected layer or a dense layer, is one of the most fundamental layers in a neural network. Its main function is to perform linear transformations on the input data through matrix multiplication and the addition of a bias vector.

[0041] Attention mechanisms are a data processing method in machine learning, widely used in various machine learning tasks such as natural language processing, image recognition, and speech recognition. The degree of attention (importance) given to different pieces of information in an attention mechanism is represented by weights. An attention mechanism can be viewed as a query matrix, a key, and a weighted average, forming a multilayer perceptron (MLP).

[0042] An attention scoring function measures the relevance or matching degree between a query vector and a key vector. Its core function is to determine which parts of the input sequence are most relevant to the current output when the model is processing the current task. By calculating the similarity scores between the query vector and each key vector, the attention scoring function provides the basis for subsequent Softmax normalization and weighted summation of values, thereby enabling the focus and selective utilization of key information. Common attention scoring methods include dot product, scaled dot product, and additive attention, which affect how the model pays attention to contextual information and its computational efficiency in different scenarios.

[0043] In server environments, by dynamically adjusting computing strategies and resource allocation, simple queries requiring rapid responses or complex analytical tasks demanding high precision can be processed efficiently. For example, in cloud-based intelligent customer service systems, appropriate model configurations are automatically selected based on the complexity of user questions, ensuring immediate responses while maintaining a high level of service quality. Furthermore, for resource-constrained terminal devices, such as smartphones or IoT devices, quantization and sparsity techniques can optimize a single model, enabling it to run locally without sacrificing excessive performance, providing services such as personalized recommendations and real-time speech recognition, reducing reliance on network connectivity and protecting user privacy. Simultaneously, this flexible scheduling mechanism can also be applied to edge computing scenarios, allowing edge devices closer to the data source to intelligently decide when and how to execute inference tasks, thereby reducing latency and improving overall inference efficiency.

[0044] Figure 1 A schematic diagram illustrating the overall flow of a reasoning request processing method according to one embodiment of this disclosure is shown. Figure 1 The inference request in method M100 is processed through a model, and method M100 includes steps S110 to S130. This method can be executed by a server.

[0045] In step S110, an inference request is received, and the request type of the inference request is determined. The inference request is determined to be either a first type of inference request or a second type of inference request. The first type of inference request is a low-precision inference request, and the second type of inference request is a high-precision inference request.

[0046] Upon receiving an inference request, the request type is determined based on the request type marker in the request or by analyzing the information carried in the request. Tasks with high response time requirements but relatively low accuracy requirements are marked as Type I inference requests; while complex tasks requiring high accuracy are marked as Type II inference requests. Dynamically adjusting the processing strategy based on the request type maximizes resource utilization and reduces overall computational costs while ensuring service quality.

[0047] The aforementioned inference requests include various modalities of data such as text sequences, images, audio, video, or tables.

[0048] The above request types can be determined based on the categories selected by the user, or by analyzing the information carried in the inference request through the identification module.

[0049] The aforementioned models include the trained GPT and BERT models.

[0050] In one specific embodiment, a reasoning request is received from a client or server. This request includes data such as input text, task description, user identity, and / or device information. The information contained in the reasoning request is parsed, and key information is extracted based on semantic features, such as the length of the input text (short or long sentences), the semantics of the request content (simple question-and-answer, complex reasoning, or professional consultation, etc.), user identity identifier, device information (mobile, edge device, or server), and tags or priorities carried in the request (high real-time requirements or accuracy priority). The extracted key information is transformed into structured features that can be used for classification. The structured features are analyzed and judged based on target rules or classification models to obtain the final classification result.

[0051] In step S120, during the model pre-filling stage, a linear layer of quantized computation and a sparse attention mechanism are used to process the first type of inference request to determine the first key-value cache set corresponding to the first type of inference request, or a full computation is performed on the second type of inference request to determine the second key-value cache set corresponding to the second type of inference request.

[0052] During the pre-filling phase of the AI ​​model, different computational strategies are selected based on the request type. To accelerate processing and reduce resource consumption, a linear layer with quantized computation and a sparse attention mechanism are applied to process the first type of inference requests. This reduces the precision of the model parameters and selects the most representative attention connections to determine the first key-value cache set. To ensure the highest accuracy, no compression or simplification strategies are employed; instead, a full computation method is used to determine the second key-value cache set for the second type of inference requests. This ensures that all necessary intermediate state information is fully preserved to provide the most accurate results during the decoding phase.

[0053] Optionally, during the model pre-filling stage, a linear layer with quantized computation and a sparse attention mechanism are used to process the first type of inference request. In some embodiments of this disclosure, this may include, for example... Figure 2 Steps S210 to S230 are shown.

[0054] In step S210, the weights and activation values ​​of the linear layer are converted into low-precision values ​​based on the target quantization precision.

[0055] In AI model inference, linear layers typically handle a large number of matrix operations, making them a major source of computational resource consumption. To improve inference efficiency and reduce resource consumption, the weights and activation values ​​of linear layers are quantized, mapping the original high-precision floating-point numbers to low-precision values ​​according to a preset target quantization precision. This process can be completed during model deployment or pre-inference processing, supporting static quantization (e.g., post-training quantization) or dynamic quantization (e.g., dynamically adjusting precision based on input at runtime), and can be seamlessly integrated with subsequent inference processes, providing fundamental support for lightweight inference. The target quantization precision is determined based on the model configuration; preferably, the target quantization precision is 8 bits.

[0056] In step S220, the first type of inference request is forward propagated through the quantized linear layer.

[0057] In the AI ​​model inference process, the first type of inference request is forward-propagated through a quantized linear layer. The weights and activation values ​​of this linear layer have been converted from high precision to low precision to support the efficient execution of low-precision computation instructions. This significantly improves inference speed, reduces memory bandwidth requirements, and optimizes overall resource utilization while ensuring basic semantic understanding capabilities of the task.

[0058] In step S230, matrix operations are performed on the first type of inference request using a sparse attention mechanism.

[0059] When processing the first type of inference request, a sparse attention mechanism is employed to optimize the attention matrix. Specifically, after calculating the attention score for the first type of inference request, only the key-value pairs with higher scores are retained, while redundant information with lower scores is ignored, thereby reducing the computational load and memory access overhead of subsequent matrix operations. This significantly improves inference efficiency without significantly impacting the task's semantic understanding capabilities.

[0060] Optionally, a full computation is performed on the second type of inference request to determine the second key-value cache set corresponding to the second type of inference request. In some embodiments of this disclosure, this may include, for example: Figure 3 Steps S310 to S320 are shown.

[0061] In step S310, the second type of inference request is linearly transformed by the target weight.

[0062] When processing the second type of inference request, a linear transformation of the input is performed using target weights, i.e., an affine transformation of the input data is performed using a high-precision weight matrix. This transformation process does not quantize or compress the weights or activation values ​​to ensure that the model can fully utilize the expressive power of the original model during inference, thereby outputting high-quality results.

[0063] In step S320, attention calculation is performed on the second type of inference request after linear transformation to determine the second key-value cache set of the second type of inference request.

[0064] After the linear transformation of the second type of inference request is completed, attention calculation is performed to extract the correlation between positions in the input sequence of the second type of inference request. By calculating the attention score between the query vector and the key vector, key contextual information is filtered out, and the corresponding key vector and value vector are cached to form a second key-value cache set. The second key-value cache set will be reused in the subsequent decoding stage to improve generation efficiency and semantic coherence.

[0065] In step S130, during the model decoding stage, the inference result of the first type of inference request is determined based on the target key-value cache in the first key-value cache set determined by the attention scoring function, or the inference result of the second type of inference request is determined based on all key-value caches in the second key-value cache set.

[0066] In the decoding phase of the AI ​​model, for the first type of inference request, an attention scoring function is used to evaluate the relevance of each key-value cache entry in the first key-value cache set, selecting the target key-value cache with the highest attention score. This reduces computational load while determining the inference result that meets the requirement of fast response. For the second type of inference request, the complete second key-value cache set is used directly for inference to ensure high accuracy and completeness of the output result. This demonstrates intelligent scheduling capabilities for resource utilization, meeting both the efficiency requirements of low-latency scenarios and ensuring the accuracy of high-precision tasks.

[0067] Optionally, the inference result of the first type of inference request is determined based on the target key-value cache in the first key-value cache set determined by the attention scoring function. In some embodiments of this disclosure, this may include, for example: Figure 4 Steps S410 to S430 are shown.

[0068] In step S410, during the model decoding phase, the attention score of the first key-value cache in the first key-value cache set is determined based on the attention scoring function.

[0069] The relevance of key-value cache entries in the first key-value cache set is evaluated using an attention scoring function to determine the context information that should be prioritized at the current decoding moment. This lays the foundation for dynamic context modeling in the decoding process using the attention mechanism.

[0070] Optionally, during the model decoding stage, the attention score of the first key-value cache in the first key-value cache set is determined based on the attention scoring function. In some embodiments of this disclosure, this may include, for example: Figure 5 Steps S510 to S530 are shown.

[0071] In step S510, the query vector is determined based on the context information of the first type of reasoning request.

[0072] When processing Type I inference requests, a query vector for the current time step or the current task is generated based on the contextual information of the input. This query vector is typically generated by a linear transformation of the input representation and serves to represent "what the current task is looking for" in the attention mechanism, used for relevance matching with the key vector.

[0073] In step S520, the first key-value cache in the first key-value cache set is processed based on the linear layer and self-attention mechanism of the decoding stage to determine the key-value cache entries of the intermediate state information of the first key-value cache; the key-value cache entries include key vectors and corresponding value vectors.

[0074] In the decoding process of artificial intelligence models, to improve inference efficiency and resource utilization, the key-value cache of the input sequence is usually pre-computed and cached for reuse in subsequent decoding stages. In this step, based on the linear layer output and self-attention mechanism of the current decoding stage, the key-value cache entries in the first key-value cache set are further updated to reflect the contextual understanding state of the model at the current decoding time step.

[0075] In step S530, the attention score of the first key-value cache in the first key-value cache set is determined by comparing the similarity between the query vector and the key vector based on the attention scoring function.

[0076] During the decoding phase of the AI ​​model, a query vector is generated based on the current decoding state, and cached key vectors are read from the first key-value cache set. An attention scoring function is used to calculate the similarity score, or attention score, between the query vector and each key vector. This attention score reflects the semantic relevance between the current decoding position and each position in the input sequence.

[0077] In step S420, the first set of key-value caches with attention scores greater than or equal to the target score is selected as the target key-value cache. During model decoding, key-value cache entries in the first set of key-value caches are filtered based on the attention score between the query vector and the key vector. By setting a target score, key-value cache entries with attention scores greater than or equal to the target score are retained to form the target key-value cache. The target key-value cache is considered to be the part most relevant to the current decoding state and with the greatest semantic contribution.

[0078] Preferably, the target score is 0.9.

[0079] In step S430, the inference result of the first type of inference request is determined based on the target key-value cache.

[0080] When processing the first type of inference request, attention-weighted summation is performed based on the target key-value cache to obtain the context vector for the current decoding time step. This context vector is then processed through modules such as the decoding layer, activation function, and output layer to finally generate the inference result. This approach can significantly reduce computational resource consumption and inference latency without sacrificing the core semantic understanding capabilities of the task.

[0081] Optionally, in the pre-filling stage, model inference is typically computationally intensive. Therefore, in the pre-filling stage, either the first type of inference request or the second type of inference request is processed in a single batch to determine the corresponding first key-value cache set or second key-value cache set. In the decoding stage, model inference typically incurs high communication overhead. Therefore, in the decoding stage, either the first key-value cache set or the second key-value cache set is processed in multiple batches to determine the corresponding inference result.

[0082] Optionally, when the GPU memory utilization is less than a first target threshold and the CPU queue depth is greater than or equal to a second target threshold, the batch size is increased based on a sliding window mechanism. When the GPU memory utilization is greater than or equal to a third target threshold, or the CPU queue depth is less than a fourth target threshold, the batch size is decreased based on a sliding window mechanism.

[0083] Specifically, the system monitors GPU memory utilization and CPU queue depth in real time, and calculates the average or current value of these metrics within a sliding window (e.g., data from the past few seconds) to assess the system's current resource usage. If GPU memory utilization is below a first target threshold and CPU queue depth is greater than or equal to a second target threshold, the batch size is increased based on the data trend within the sliding window to improve throughput. Conversely, if GPU memory utilization is greater than or equal to a third target threshold, or CPU queue depth is less than a fourth target threshold, the batch size is decreased to reduce resource consumption and prevent overload. Throughout the process, the sliding window mechanism smoothly adjusts batch size changes, avoiding frequent adjustments due to instantaneous fluctuations and ensuring system stability and responsiveness.

[0084] Optionally, during the deployment of AI model inference, different types of operators exhibit significant differences in computational characteristics, data flow control, and parallelism. Static analysis is performed on the operators in the model to identify control-intensive and parallelism-intensive operators. Control-intensive operators are assigned to the CPU for execution, parallelism-intensive operators are assigned to the GPU, or both control-intensive and parallelism-intensive operators are executed by the GPU.

[0085] Specifically, control-intensive operators are characterized by complex control flow, low parallelism, and reliance on CPU instruction flow control. These operators include conditional statements, branch control, and dynamic routing. Parallel-intensive operators, on the other hand, are characterized by high parallelism and suitability for GPU parallel computing. These operators include matrix multiplication, convolution, and attention computation.

[0086] Optionally, a mathematical optimization method is introduced into the attention computation during the pre-filling and decoding phases of the artificial intelligence model. This mathematical optimization method reduces the number of accesses to the high-bandwidth memory (HBM) by rearranging the computation order.

[0087] The aforementioned mathematical optimization methods include the FlashAttention method. Specifically, it divides the input query vector, key vector, and value vector into multiple small blocks of fixed size, and processes these blocks in parallel using a GPU. It also optimizes the computation order to avoid unnecessary redundant calculations, reducing computational overhead. Furthermore, it adjusts memory access patterns to reduce the number of accesses to the HBM, lowering communication overhead. In addition, knowledge distillation can be used to transfer knowledge from a large, well-trained model (such as GPT or BERT) to a smaller model, allowing inference to be performed using this smaller model. This approach maintains high accuracy while reducing computational resource consumption.

[0088] The inference request processing method provided in this embodiment can distinguish between first-type and second-type inference requests based on their type. During the pre-filling stage, it generates corresponding key-value caches using quantization and sparse attention mechanisms, or full computation, respectively. In the decoding stage, it further filters key information based on an attention scoring function or directly uses the complete cache for inference. This achieves adaptive processing of tasks with different precision levels under the same model architecture, improving the inference speed and resource utilization of low-precision requests while ensuring the output quality of high-precision requests. Therefore, it optimizes inference efficiency overall, reduces computational resource consumption, and enhances the system's flexibility and adaptability in diverse application scenarios.

[0089] The following section uses an intelligent customer service application scenario as an example to further illustrate the technical solution of this disclosure. Those skilled in the art should understand that it can also be applied to other application scenarios besides intelligent customer service.

[0090] When a user initiates a reasoning request to the customer service system via mobile phone or web browser, the system first determines the request type based on the complexity of the request content and the user's identity tag (such as a regular user or a VIP user), classifying the request into either Type I or Type II reasoning requests. For example, a user asking about "business hours" is a simple question-and-answer task and is identified as a Type I reasoning request; while a user submitting a request for "contract terms interpretation" is a more complex task requiring specialized knowledge and is identified as a Type II reasoning request.

[0091] Subsequently, during the pre-filling phase of the AI ​​model, different processing strategies are selected based on the request type. For the first type of inference request, a linear layer with quantized computation and a sparse attention mechanism are enabled to perform lightweight processing on the input content. This significantly reduces computational resource consumption while ensuring basic semantic understanding, and generates a first key-value cache set. For the second type of inference request, a full computation approach is adopted without any compression or sparsification operations to ensure that all contextual information is fully preserved, generating a second key-value cache set to guarantee high accuracy of the output results.

[0092] During the decoding phase of the AI ​​model, different decoding strategies are selected based on the request type. For the first type of inference request, an attention scoring function is used to evaluate the entries in the first key-value cache set, selecting those with attention scores greater than the target score to form the target key-value cache. Response results are then quickly generated based on these key entries, shortening response time. For the second type of inference request, the complete second key-value cache set is directly used for decoding, ensuring the comprehensiveness and accuracy of the output content.

[0093] Finally, the generated inference results are returned to the user, completing the entire inference process. This inference process flexibly supports inference tasks with different levels of precision within the same model architecture, improving resource utilization efficiency while ensuring service quality.

[0094] Based on any of the above embodiments, this disclosure also provides a reasoning request processing apparatus.

[0095] Figure 6 This is a schematic block diagram of the structure of a reasoning request processing apparatus according to one embodiment of the present disclosure.

[0096] like Figure 6 As shown, the inference request processing device includes:

[0097] The inference request receiving module 610 receives inference request information input by the user;

[0098] The inference request identification module 620 performs intent identification or identifier identification on the received inference request information, obtains the identification result, and determines the request type of the inference request information based on the identification result. The request type is either a low-precision type inference request or a high-precision type inference request.

[0099] For AI model 630, if the request type is a low-precision inference request, the AI ​​model 630 uses a linear layer of quantized computation and a sparse attention mechanism in the pre-filling stage of the inference process, and determines the target key-value cache based on the attention scoring function in the decoding stage, thereby determining the corresponding inference result. If the request type is a high-precision inference request, the AI ​​model 630 uses full computation in the pre-filling stage of the inference process, and uses the full key-value cache in the decoding stage to determine the corresponding inference result.

[0100] The aforementioned inference request processing device can be in the form of computer software, and each module of the aforementioned inference request processing device can be implemented through computer software modules.

[0101] The specific implementation process of the functions and roles of each module in the above-mentioned reasoning request processing device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0102] This disclosure also provides an electronic device 1000 (corresponding to the inference request processing method). Figure 7 A schematic diagram of the hardware implementation using the processing system is shown.

[0103] The hardware architecture of electronic devices can be implemented using a bus architecture. A bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripherals, voltage regulators, power management circuits, external antennas, etc. Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not indicate that there is only one bus or one type of bus.

[0104] For ease of explanation, certain steps of the above method are described in relation to modules. It should be understood that the corresponding module performing one or more steps of the above method may be one or more hardware modules specifically configured to perform the corresponding step, or implemented by a processor configured to perform the corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination thereof.

[0105] This disclosure also provides a storage medium storing a computer program, which, when executed by a processor, is used to implement the methods described above. "Storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use with or in conjunction with an instruction execution system, apparatus, or device. More specific examples of storage media include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0106] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0107] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0108] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, electronic devices, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0113] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method for processing inference requests, wherein the inference requests are processed through a model, characterized in that, include: Receive an inference request, determine the request type of the inference request, and determine the inference request as a first type of inference request or a second type of inference request. The first type of inference request is a low-precision inference request, and the second type of inference request is a high-precision inference request. In the pre-filling stage of the model, a linear layer with quantized computation and a sparse attention mechanism are used to process the first type of inference request to determine the first key-value cache set corresponding to the first type of inference request, or a full computation is performed on the second type of inference request to determine the second key-value cache set corresponding to the second type of inference request; and During the decoding phase of the model, the inference result of the first type of inference request is determined based on the target key-value cache in the first key-value cache set determined by the attention scoring function, or the inference result of the second type of inference request is determined based on all key-value caches in the second key-value cache set. Specifically, determining the inference result of the first type of inference request based on the target key-value cache in the first key-value cache set determined by the attention scoring function includes: in the decoding stage of the model, determining the attention score of the first key-value cache in the first key-value cache set based on the attention scoring function; taking the first key-value cache set whose attention score is greater than or equal to the target score as the target key-value cache; and determining the inference result of the first type of inference request based on the target key-value cache. Specifically, in the decoding phase of the model, determining the attention score of the first key-value cache in the first key-value cache set based on the attention scoring function includes: determining a query vector based on the context information of the first type of inference request; processing the first key-value cache in the first key-value cache set based on the linear layer and self-attention mechanism of the decoding phase to determine the key-value cache entries of the intermediate state information of the first key-value cache; the key-value cache entries include key vectors and corresponding value vectors; and comparing the similarity between the query vector and the key vector based on the attention scoring function to determine the attention score of the first key-value cache in the first key-value cache set.

2. The reasoning request processing method as described in claim 1, characterized in that, During the pre-filling stage of the model, the first type of inference request is processed using a linear layer with quantized computation and a sparse attention mechanism, including: The weights and activation values ​​of the linear layer are converted to low-precision values ​​based on the target quantization precision. The first type of inference request is forward-propagated through a quantized linear layer; Matrix operations are performed on the first type of inference request using a sparse attention mechanism.

3. The reasoning request processing method as described in claim 1, characterized in that, Perform a full computation on the second type of inference request to determine the second key-value cache set corresponding to the second type of inference request, including: The second type of inference request is linearly transformed using the target weight; Attention is calculated on the second type of inference request after linear transformation to determine the second key-value cache set for the second type of inference request.

4. The reasoning request processing method as described in claim 1, characterized in that, Also includes: During the pre-filling phase, the first type of inference request or the second type of inference request is processed in a single batch to determine the corresponding first key-value cache set or second key-value cache set. During the decoding phase, the first key-value cache set or the second key-value cache set is processed in multiple batches to determine the corresponding inference result.

5. The reasoning request processing method as described in claim 4, characterized in that, Also includes: When the GPU memory usage is less than the first target threshold and the CPU queue depth is greater than or equal to the second target threshold, the batch size is increased based on the sliding window mechanism. When the GPU memory utilization rate is greater than or equal to the third target threshold, or the CPU queue depth is less than the fourth target threshold, the batch size is reduced based on the sliding window mechanism.

6. The reasoning request processing method as described in claim 1, characterized in that, Also includes: Static analysis is performed on the operators in the model to determine control-intensive operators and parallel-intensive operators; Assign control-intensive operators to the CPU for execution, and assign parallel-intensive operators to the GPU for execution.

7. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the inference request processing method according to any one of claims 1 to 6.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the reasoning request processing method according to any one of claims 1 to 6.

Images