Method and system for processing user input information for model inference phase

Abstract

The application discloses a user input information processing method and system for a model inference stage, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a plurality of complete prompts input by a user; regularly acquiring hardware resource information and current task information; determining hardware resource allocation information corresponding to the complete prompts; performing a pre-execution parallel computing operation on the complete prompts according to the hardware resource allocation information in a pre-filling stage; alternately circulating the first type of computing device with the computing power reaching a preset computing power threshold to read the intermediate features from a preset resource pool for decoding and storage operation in a feedforward network stage according to the hardware resource allocation information, and circulating the second type of computing device with the memory access performance reaching a preset memory access performance condition to read data from the preset resource pool for attention computing and storage operation in an attention stage; obtaining a model inference result and outputting the model inference result. The technical problem of waste of hardware resources is solved, and the technical effect of improving resource utilization and inference efficiency is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a user input information processing method and system for a model inference stage. BACKGROUND

[0002] In the model inference stage of artificial intelligence, such as the prefill stage, the attention stage and the feedforward network (FFN) stage, the calculation characteristics of each stage are significantly different. In related technologies, the calculation modules of the attention and FFN are bound to the single hardware (such as a graphics processing unit, GPU) with the display memory resource, which causes the calculation resource and the memory resource to be unable to be dynamically allocated as needed. For example, in the prefill stage, the high calculation capability of the GPU can quickly process a large amount of input data, but the subsequent decoding stage is limited by the display memory bandwidth due to frequent access to the KV Cache, which causes the GPU utilization to decrease. In addition, in a multi-user concurrent scene (such as a chat robot and a document summary generation), the calculation requirements of different tasks are significantly different, and the traditional architecture is difficult to balance the resource allocation of the calculation-intensive and memory-intensive tasks, which causes the waste of hardware resources or the performance bottleneck.

[0003] Therefore, there is an urgent need for a user input information processing method for a model inference stage to improve the above problems. SUMMARY

[0004] The present application provides a user input information processing method and system for a model inference stage to at least solve the problems of low resource utilization and low inference efficiency in related technologies.

[0005] The present application provides a user input information processing method for a model inference stage, the model inference stage including a prefill stage, an attention stage and a feedforward network stage, and the method includes the following steps.

[0006] Obtaining a plurality of complete prompts input by a user, the complete prompt including a plurality of word pieces;

[0007] Periodically obtaining hardware resource information and current task information, and determining hardware resource allocation information corresponding to the complete prompt according to the hardware resource information and the current task information;

[0008] In the prefill stage, performing a parallel calculation operation on the complete prompt according to the hardware resource allocation information to obtain intermediate features corresponding to each word piece and a key-value cache, and storing the intermediate features and the key-value cache in a preset resource pool;

[0009] According to the hardware resource allocation information, the first type of computing device whose computing power reaches the preset computing power threshold is alternately called in the feedforward network stage to read the intermediate features from the preset resource pool for decoding and storage operations, and the second type of computing device whose memory access performance reaches the preset memory access performance condition is called in the attention stage to read data from the preset resource pool for attention calculation and storage operations, until the preset decoding stop condition is triggered, the model inference result is obtained and output.

[0010] The application also provides a user input information processing system for a model inference stage, which comprises: an attention calculation resource pool, a preset resource pool and a feedforward network calculation resource pool.

[0011] The attention calculation resource pool comprises a plurality of second type of computing devices whose memory access performance reaches a preset memory access performance condition, a plurality of third type of computing devices whose memory read-write performance reaches a preset read-write performance condition, and a controller. The second type of computing device reads data from the preset resource pool for attention calculation and storage operations. The third type of computing device is used to simultaneously perform self-attention calculation operations on a plurality of word tokens to obtain intermediate features corresponding to each word token and key-value caches, and synchronously stores the intermediate features corresponding to each word token and the key-value caches into the preset resource pool in a preset memory expansion protocol fragmentation format. The controller is used to implement the steps of the user input information processing method for the model inference stage when executing a computer program.

[0012] The preset resource pool comprises a plurality of computing fast link devices. The computing fast link device is used to maintain cache consistency with the attention calculation resource pool and the feedforward network calculation resource pool through a memory expansion protocol.

[0013] The feedforward network calculation resource pool comprises at least one first type of computing device whose computing power reaches a preset computing power threshold. The first type of computing device is used to read intermediate features from the preset resource pool for decoding and storage operations.

[0014] According to the application, the hardware resource allocation information corresponding to the complete prompt of each task is determined first, and then the hardware resources are allocated in different stages of the model inference stage to complete different calculation processes, thereby improving the resource utilization rate and inference efficiency in different calculation scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 The related technical architecture diagram provided by the embodiments of the present application is shown.

[0017] Figure 2 A flowchart illustrating a user input information processing method for the model inference stage, provided in an embodiment of this application;

[0018] Figure 3 A schematic diagram of a transmission frame format for a preset memory expansion protocol fragmentation format provided in an embodiment of this application;

[0019] Figure 4 A schematic diagram of a user input information processing device for the model inference stage provided in this application embodiment;

[0020] Figure 5 A schematic diagram of a user input information processing system architecture for the model inference stage is provided in an embodiment of this application;

[0021] Figure 6 This is a schematic diagram of the structure of a controller provided in this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0023] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0024] Figure 1 A schematic diagram of the architecture of the related technologies provided in the embodiments of this application.

[0025] like Figure 1 As shown, each computational module in the model inference stage includes computational modules for attention and feedforward networks, and these modules are inseparable from the hardware. Figure 1The dashed box in the image represents a complete hardware entity upon which video memory and computation are based. This hardware entity can be a graphics processing unit (GPU) or other computing devices with similar functionality. As mentioned above, the significant difference in GPU utilization between Attention and FFN in related technologies confirms the differences in their computational characteristics. Due to memory-intensive operations, the computation of Attention is not significantly affected by changes in size. This leads to problems of low computational resource utilization and low processing efficiency in the inference process based on user input.

[0026] To address this issue, this invention provides the following inventive concept: by designing a dedicated resource pool as a relay, multimodal tasks are decoupled and deployed to their respective independent hardware modules, thus solving the problem of balancing the computational needs between modalities in a unified resource pool. For example, the processor in the image processing resource pool can efficiently perform convolution calculations, while the processor in the text processing resource pool can focus on high-bandwidth memory access during the Attention phase. Through modal-specific hardware configuration, the processing efficiency of multimodal tasks is significantly improved, while cross-modal communication overhead is reduced, providing a more flexible resource allocation scheme for complex task scenarios (such as text generation and video analysis).

[0027] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Figure 2 This is a flowchart illustrating a user input information processing method for the model inference stage, provided in an embodiment of this application.

[0029] like Figure 2 As shown, embodiments of this application provide a method for processing user input information during the model inference stage. The execution entity of this method can be a server or other hardware device with the same function. Based on Figure 2 The method is described in detail below. In this embodiment, the model inference stage includes a pre-filling stage, an attention stage, and a feedforward network stage. The method includes the following steps:

[0030] S201: Obtain multiple complete suggestions from user input, where each complete suggestion includes multiple lexical units.

[0031] In this embodiment, the user can simultaneously compute input prompt information in large batches using multiple computing devices to create a complete prompt. In this embodiment, the complete prompt can be a user-input instruction, a question, or text information used to guide the model in generating content. Each complete prompt includes multiple tokens. A token refers to the smallest unit into which the complete prompt is segmented, such as a word, phrase, or character.

[0032] S202: Periodically obtain hardware resource information and current task information, and determine the hardware resource allocation information corresponding to the complete prompt based on the hardware resource information and current task information.

[0033] In this embodiment, hardware resource information can be the utilization rate of currently idle computing hardware resources, such as the graphics processing unit (GPU), and current task information can be dynamic information about the task characteristics of the computing task to be executed and the priority information of each task to be executed, etc., which are related to task execution. Based on the hardware resource information and the current task information, the hardware resource allocation strategy can be generated through the algorithm deployed in the controller to determine the hardware resource allocation information corresponding to the complete prompt.

[0034] In an optional embodiment of this application, step S202 includes:

[0035] S202a: Determine the task queue backlog and average task sequence length based on the current task information.

[0036] In this embodiment, the task queue backlog represents the task waiting status at each stage of the inference phase, determining the resource allocation direction. For example, if the number of tasks waiting in the feedforward network stage is less than the number of tasks in the attention stage, then the resource allocation direction is to increase the number of calls to computing devices with high memory access performance. The average task sequence length represents the impact on memory usage and computational intensity, and is used to guide the computational task allocation strategy.

[0037] S202b: Determine hardware resource utilization and memory pool access latency based on hardware resource information.

[0038] In this embodiment, hardware resource information may include the number of idle hardware units and the number of active hardware units. Based on these two types of data, hardware resource utilization can be calculated. Memory access latency can be data inherent in the hardware monitoring data. Hardware resource utilization is used to assess hardware resource availability and avoid overload or idleness. Memory access latency is used to assess cross-node data transfer efficiency and optimize task scheduling priorities.

[0039] In an optional embodiment of this application, the method further includes: periodically obtaining a list of connected computing devices and their memory capacity through the interconnection switch interface of the computing fast link at a preset period, and performing feature extraction operations to obtain task queues, computing device utilization, and network latency as hardware resource information and current task information.

[0040] In this embodiment, hardware resource information can be obtained by acquiring the list of connected computing devices and their memory capacity through the CXL switch interface API. Then, feature extraction is performed, for example, by collecting data such as task queues, computing device utilization (e.g., GPU utilization), and network latency at a preset period of 100ms.

[0041] S202c: Determine the resource request skew coefficient corresponding to the complete prompt based on the task queue backlog, average task sequence length, hardware resource utilization, memory pool access latency, preset task backlog penalty coefficient, and preset custom weight.

[0042] In this embodiment, the corresponding resource request tilt coefficient is calculated using a specific calculation formula based on the task queue backlog, average task sequence length, hardware resource utilization, memory pool access latency, preset task backlog penalty coefficient, and preset custom weight.

[0043] In an optional embodiment of this application, the calculation formula for step S202c is as follows:

[0044]

[0045] In the formula, W decode This represents the resource request skew coefficient corresponding to the complete prompt, and γ represents the task backlog penalty coefficient. δ represents a preset custom weight, Gutil represents hardware resource utilization, Avg_decode_len represents the average sequence length of the task, Mem_pool_lat represents the memory pool access latency, and Task_decode represents the task queue backlog.

[0046] S202d: Determine hardware resource allocation information based on the resource request skew coefficient and hardware resource information corresponding to the complete prompt.

[0047] In this embodiment, the allocation ratio of each computing device is dynamically adjusted based on the resource request skew coefficient and hardware resource information such as GPU computing power during the current task, i.e., the processing of the complete prompt.

[0048] Specifically, in an optional embodiment of this application, step S202d includes:

[0049] Step d1: Normalize the calculation based on the resource request skew coefficient corresponding to each complete prompt to obtain the hardware resource ratio corresponding to each complete prompt.

[0050] Step d2: Determine hardware allocation information based on hardware resource ratio and hardware resource information.

[0051] In this embodiment, normalization refers to a rule that scales multiple resource request skew coefficients or weights proportionally, so that each value falls within a specific numerical range, such as the (0,1) range. In this embodiment, determining hardware allocation information can involve normalizing the resource request skew coefficients of multiple task requests to determine the computing device allocation ratio, and then dynamically adjusting it based on GPU computing power to balance throughput and latency.

[0052] In an optional embodiment of this application, step S202d further includes:

[0053] Step d3: Based on the hardware resource information and the current task information, perform iterative calculations using reinforcement learning or a pre-initialized Q-table to obtain the hardware resource allocation information corresponding to the complete prompt.

[0054] In this embodiment, reinforcement learning is an algorithm that allows an agent (which can be understood here as a "weight optimization module") to learn the optimal behavior as much as possible through "trial and error" in an environment (here, a "large model inference system")—the agent adjusts weights by performing actions. By observing environmental feedback (such as whether inference latency decreases) and gradually learning the weight values ​​that maximize the "reward" (meeting performance goals), the algorithm learns to learn the "state-action value function Q(s,a)" (i.e., "the long-term reward obtained by performing action a in state s") and finds the optimal action (here, the action is "choosing") by continuously updating the Q value. The specific value of δ). This step aims to automatically optimize custom weights through reinforcement learning or pre-initialized Q-table Q-Learning. The value of δ allows the weights to better adapt to the performance goals of large model inference (such as reducing latency and improving the reuse rate of key-value cache).

[0055] S203: During the pre-filling stage, based on the hardware resource allocation information, the complete prompt is pre-executed through parallel computing operations to obtain the intermediate features and key-value caches corresponding to each word and store them in the preset resource pool.

[0056] In this embodiment, the prefill stage can utilize multiple computing devices, such as graphics processing units (GPUs), to process the computational operations of multiple tokens in parallel, thereby outputting the intermediate hidden feature and cache key value (KVcache) corresponding to each token. Then, the intermediate hidden feature and cache key value (KVcache) can be stored in a preset resource pool by the corresponding computation fast link format flow control unit (CXL) flit data frames.

[0057] In this embodiment, the preset resource pool can be a hardware memory resource pool for storing data, which is constructed by a third type of computing device (type3) based on computing fast link, consisting of multiple computing fast link devices. The type3 device can maintain cache consistency with the feedforward network resource pool through the computing fast link protocol (CXL.mem protocol) and the attention resource pool.

[0058] Specifically, in an optional embodiment of this application, step S203 includes:

[0059] S203a: In the pre-filling stage, the complete prompt is broken down to obtain multiple tokens.

[0060] In this embodiment, the decomposition process refers to decomposing a complete prompt containing multiple word elements into multiple word elements according to a pre-specified word element decomposition rule, such as decomposing it into multiple instruction word elements according to the instruction type.

[0061] S203b: Based on hardware resource allocation information, multiple third-class computing devices with memory read / write performance reaching preset read / write performance conditions are invoked to simultaneously perform self-attention calculation operations on multiple lexical units, thereby obtaining intermediate features and key-value caches corresponding to each lexical unit.

[0062] In this embodiment, the hardware resource allocation information may include grouping multiple tokens into a fixed number and calling the content of a corresponding number of third-type computing devices. Multiple third-type computing devices simultaneously perform self-attention calculation operations on multiple tokens in each group to obtain intermediate features and key-value caches. For example, if there were originally 20 tokens grouped into sets of 5, they are allocated to 4 graphics processing units (GPUs) according to the hardware resource allocation information: GPU1 processes tokens 1-5, GPU2 processes tokens 6-10, GPU3 processes tokens 11-15, and GPU4 processes tokens 16-20.

[0063] S203c: The intermediate features and key-value caches corresponding to each word are synchronously stored into a preset resource pool by multiple third-class computing devices according to a preset memory expansion protocol sharding format.

[0064] In this embodiment, the preset memory expansion protocol fragmentation format can be the standard 256B CXL Flit transmission frame format.

[0065] In an optional embodiment of this application, the preset memory expansion protocol fragmentation format is to divide the standard byte-sized data load into three first-byte-sized load fragments and one second-byte-sized load fragment, wherein the first-byte-sized data load is greater than the second-byte-sized data load, and the sum of the three first-byte-sized data load fragments and the second-byte-sized data load is equal to the standard byte-sized data load.

[0066] Figure 3 This is a schematic diagram of a transmission frame format for a preset memory expansion protocol fragmentation format provided in an embodiment of this application.

[0067] like Figure 3As shown, in this embodiment, the transmission frame format remains compatible with the standard 256B CXL Flit transmission frame format. However, the difference lies in that the middle 240B data is divided into four blocks: three 64B payload blocks are used for transmission with the Attention ATTN, and the last 48B payload block is used for transmitting the token vector and intermediate features. This is because key-value cache multiplexing transmission has higher requirements for bytes and latency than intermediate features and token vectors. Generally, intermediate features and token vectors are transmitted in a single transmission, while key-value cache includes reading and reusing historical data as well as writing operations to newly generated key-value caches.

[0068] S204: Based on the hardware resource allocation information, the first type of computing device with computing power reaching the preset computing power threshold is alternately called in the feedforward network stage to read intermediate features from the preset resource pool for decoding and storage operations, and the second type of computing device with memory access performance reaching the preset memory access performance condition is called in the attention stage to read data from the preset resource pool for attention computing and storage operations, until the preset decoding stop condition is triggered, and the model inference result is obtained and output.

[0069] In this embodiment, hardware resource allocation information may include allocating corresponding computing device resources according to hardware usage requirements at different inference stages. Generally, resource allocation is performed by the controller dynamically sensing task information and hardware resource monitoring data.

[0070] In the feedforward network stage, a first-type computing device, such as a high-performance graphics processing unit (GPU) with computing power reaching a preset computing power threshold, performs non-linear transformations on the intermediate features from the pre-filling stage to output decoded tokens. The preset computing power threshold can be a user-defined, relatively high-performance processor value within the industry. In the attention stage, a second-type computing device with high memory access performance reaching a preset memory access performance condition calculates attention weights by combining the previously generated key-value cache and newly generated tokens to capture contextual relationships. This process alternates between the feedforward network stage and the attention stage until a preset decoding stop condition is triggered. In this embodiment, the preset decoding stop condition can be that the number of generated tokens reaches an upper limit, or that the last decoded token is a special punctuation mark at the end of the text, such as a period.

[0071] Based on the above embodiments, in an optional embodiment of this application, step S204 includes:

[0072] S204a: Based on the hardware resource allocation information, in the feedforward network stage, the first type of computing device with computing power reaching the preset computing power threshold reads intermediate features from the preset resource pool and performs decoding calculations to obtain the first decoded word, and saves the first decoded word to the preset resource pool.

[0073] In this embodiment, intermediate features from the pre-filling stage are read from the preset resource pool during the feedforward network stage for calculation to obtain the first decoded token output. In this embodiment, saving the first decoded token to the preset resource pool can be done by writing the first decoded token to a specific storage address in the preset resource pool in vector form. For example, the GPU5 graphics processor in the feedforward network FFN resource pool reads the complete 20×1024-dimensional hidden feature matrix from addresses 0x0001~0x0008 in the preset resource pool via the CXL.mem protocol. Then, matrix multiplication (GEMM) is performed, first multiplying the hidden feature matrix with the weight matrix W1 (1024×4096 dimensions), then performing the ReLU activation function, and finally multiplying it with the weight matrix W2 (4096×1024 dimensions) to obtain the transformed 20×1024-dimensional features. A Softmax operation is then performed on the transformed features, and the token with the highest probability is selected as the first generated result, denoted as Token21. Then, by using a CXL flit frame (64B fragment, occupying 1 fragment), the 1024-dimensional vector of Token21 is written to address 0x0017 of the memory resource pool, while recording the metadata of Token21 (generation time, corresponding input Prompt correlation, etc.).

[0074] S204b: Based on the hardware resource allocation information, during the attention phase, the second type of computing device whose memory access performance meets the preset memory access performance conditions is called to read the first decoded word and key-value cache from the preset resource pool and perform attention calculation, obtain the attention result and the updated key-value cache and store them in the preset resource pool.

[0075] In this embodiment, the main process of reading the first decoded token and key-value cache from the preset resource pool and performing attention calculation to obtain the attention result and the updated key-value cache is as follows: First, the attention stage calls a processor with high memory access performance, such as GPU1. The specific call can be allocated by the controller or scheduler. Through the CXL.mem protocol, two types of data are read from the preset resource pool: the newly generated token, such as Token21, and the key-value cache corresponding to the updated newly generated token, as well as the key-value cache corresponding to the first 20 tokens generated in the pre-filling stage. The process of generating the query vector Q vector, calculating similarity, weight normalization, and calculating the output is as follows.

[0076] For example, the vector of Token21 is multiplied by the query weight matrix WQ (1024×1024 dimensions) of the Attention layer to obtain a 1×1024-dimensional query vector Q. The Q vector is multiplied by the transpose of the key matrix K (1024×20 dimensions) to obtain a 1×20-dimensional similarity matrix (representing the association strength between Token21 and the input Tokens 1~20). A Softmax operation is performed on the similarity matrix to obtain the normalized attention weights (e.g., the weight of Token21 with Token5 is 0.18, and the weight with Token8 is 0.23). Finally, the attention weights are multiplied by the numerical matrix V to obtain a 1×1024-dimensional attention result (incorporating input context information).

[0077] The process of storing the attention results and the updated key-value cache into the preset resource pool can be as follows: the query vector of the new token is used as the new key K vector, and the intermediate value of the attention result is used as the new numerical V vector (both are 1024-dimensional). GPU1 writes the 1×1024-dimensional attention result to memory address 0x0018 through CXL flit frames, and appends the newly generated key-value cache (K / V vector) to KVcache (addresses 0x0019~0x0020). At this time, the total number of KVcache is updated to 21 groups (corresponding to Token1~21).

[0078] S204c: Based on the hardware resource allocation information, in the feedforward network stage, the first type of computing device with computing power reaching the preset computing power threshold is called to read the attention result from the preset resource pool for decoding calculation, obtain the subsequent decoded words and store them in the preset resource pool.

[0079] S204d: Based on the hardware resource allocation information, during the attention phase, the second type of computing device whose memory access performance reaches the preset memory access performance condition is called to read the subsequent decoded words and the updated key-value cache from the preset resource pool for attention calculation, and jumps to the step of obtaining the attention result and the updated key-value cache and storing them in the preset resource pool, until the preset decoding stop condition is triggered, and the model inference result is obtained and output.

[0080] In this embodiment, during the feedforward network stage, a first type of computing device, different from that in step S204a, can be invoked to read attention results from a preset resource pool for decoding calculation. The decoding principle and specific decoding example are similar to those in step S204a. The process of storing the subsequently decoded tokens into the preset resource pool after decoding calculation is also similar to that in step S204a, so it will not be repeated here in this embodiment. The subsequent decoded tokens can generate the query vector of the 22nd token, and the similarity and normalized weights are calculated with the previous 21 sets of key vectors, and finally the attention results of the context of tokens 1 to 22 are integrated. The principle and process of step S204d are similar to those of step S204b, so it will not be repeated here in this embodiment either. The above steps are repeated in a loop, with different computing devices invoked at different stages to ensure that the calculations of steps S204a, S204c and steps S204b, S204d are parallel and efficient. This improves resource utilization and data processing efficiency.

[0081] In summary, the user input information processing method provided in this application for the model inference stage first determines the hardware resource allocation information corresponding to the complete prompts of each task, and then allocates hardware resources at different stages of the model inference stage to complete different calculation processes, thereby improving resource utilization and inference efficiency in different computing scenarios.

[0082] Furthermore, a dynamic scheduling algorithm is used to optimize the calculation of resource request skew coefficients based on current task information and hardware resource allocation information, thereby dynamically adjusting hardware resource allocation information. This further improves the accuracy and applicability of resource usage and allocation.

[0083] At the same time, a unified pre-set resource pool memory architecture is used to solve the communication overhead between different stages in the model inference phase.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0085] Figure 4 This is a schematic diagram of a user input information processing device for the model inference stage, provided as an embodiment of this application. Figure 4 As shown, embodiments of this application also provide a user input information processing device for the model inference stage, the device comprising: an acquisition module 41, a parallel computing module 42, and a dynamic processing module 43.

[0086] The acquisition module 41 is used to acquire multiple complete prompts input by the user, and the complete prompts include multiple word elements.

[0087] The acquisition module 41 is also used to periodically acquire hardware resource information and current task information, and determine the hardware resource allocation information corresponding to the complete prompt based on the hardware resource information and current task information.

[0088] The parallel computing module 42 is used to perform parallel computing operations on the complete prompt during the pre-filling stage according to the hardware resource allocation information, obtain the intermediate features and key-value cache corresponding to each word, and store them in the preset resource pool.

[0089] The dynamic processing module 43 is used to alternately call the first type of computing device with computing power reaching the preset computing power threshold to read intermediate features from the preset resource pool for decoding and storage operations in the feedforward network stage according to the hardware resource allocation information, and call the second type of computing device with memory access performance reaching the preset memory access performance condition to read data from the preset resource pool for attention computing and storage operations in the attention stage, until the preset decoding stop condition is triggered, so as to obtain the model inference result and output it.

[0090] In an optional embodiment of this application, the parallel computing module 42 is specifically used for: decomposing the complete prompt during the pre-filling stage to obtain multiple lexical units; calling multiple third-type computing devices with memory read / write performance reaching preset read / write performance conditions to simultaneously perform self-attention calculation operations on the multiple lexical units according to hardware resource allocation information, to obtain intermediate features and key-value caches corresponding to each lexical unit; and synchronously storing the intermediate features and key-value caches corresponding to each lexical unit into a preset resource pool through multiple third-type computing devices according to a preset memory expansion protocol sharding format.

[0091] In an optional embodiment of this application, the dynamic processing module 43 is specifically configured to: In the feedforward network stage, based on hardware resource allocation information, call a first type of computing device whose computing power reaches a preset computing power threshold to read intermediate features from a preset resource pool and perform decoding calculations to obtain the first decoded term, and save the first decoded term to the preset resource pool; In the attention stage, based on hardware resource allocation information, call a second type of computing device whose memory access performance reaches a preset memory access performance condition to read the first decoded term and key-value cache from the preset resource pool and perform attention calculations to obtain the attention result and the updated key-value cache, and store them in the preset resource pool; In the feedforward network stage, based on hardware resource allocation information, call a first type of computing device whose computing power reaches a preset computing power threshold to read the attention result from the preset resource pool and perform decoding calculations to obtain subsequent decoded terms and store them in the preset resource pool; In the attention stage, based on hardware resource allocation information, call a second type of computing device whose memory access performance reaches a preset memory access performance condition to read subsequent decoded terms and the updated new key-value cache from the preset resource pool to perform attention calculations, and jump to the step of obtaining the attention result and the updated key-value cache and storing them in the preset resource pool, until a preset decoding stop condition is triggered, to obtain the model inference result and output it.

[0092] In an optional embodiment of this application, the acquisition module 41 is further specifically used to: determine the task queue backlog and the average task sequence length based on the current task information; determine the hardware resource utilization and memory pool access latency based on the hardware resource information; determine the resource request skew coefficient corresponding to the complete prompt based on the task queue backlog, the average task sequence length, the hardware resource utilization, the memory pool access latency, the preset task backlog penalty coefficient, and the preset custom weight; and determine the hardware resource allocation information based on the resource request skew coefficient corresponding to the complete prompt and the hardware resource information.

[0093] In an optional embodiment of this application, the acquisition module 41 is specifically used to determine the calculation formula for the resource request skew coefficient corresponding to each complete prompt based on the task queue backlog, average task sequence length, hardware resource utilization, memory pool access latency, preset task backlog penalty coefficient, and preset custom weight.

[0094]

[0095] In the formula, W decode This represents the resource request skew coefficient corresponding to the complete prompt, and γ represents the task backlog penalty coefficient. δ represents a preset custom weight, Gutil represents hardware resource utilization, Avg_decode_len represents the average sequence length of the task, Mem_pool_lat represents the memory pool access latency, and Task_decode represents the task queue backlog.

[0096] In an optional embodiment of this application, the acquisition module 41 is further configured to: perform normalization calculation based on the resource request skew coefficient corresponding to each complete prompt to obtain the hardware resource ratio corresponding to each complete prompt; and determine the hardware allocation information based on the hardware resource ratio and hardware resource information.

[0097] In an optional embodiment of this application, the acquisition module 41 is further configured to: perform iterative calculation processing based on hardware resource information and current task information through reinforcement learning or a preset initialized Q-table to obtain the hardware resource allocation information corresponding to the complete prompt.

[0098] The description of the features in the embodiment of the user input information processing device for the model inference stage provided in this application can be referred to the relevant description of the embodiment of the user input information processing method for the model inference stage described above, and will not be repeated here.

[0099] Figure 5 This is a schematic diagram of a user input information processing system architecture for the model inference stage, provided in an embodiment of this application.

[0100] like Figure 5As shown, the system includes an attention computing resource pool 51, a preset resource pool 52, and a feedforward network computing resource pool 53. The attention computing resource pool 51 includes multiple second-type computing devices with memory access performance meeting preset memory access performance conditions, multiple third-type computing devices with memory read / write performance meeting preset read / write performance conditions, and a controller. The second-type computing devices read data from the preset resource pool 52 for attention computing and storage operations. The third-type computing devices are used to simultaneously perform self-attention computing operations on multiple lexical units, obtaining intermediate features and key-value caches corresponding to each lexical unit, and synchronously storing the intermediate features and key-value caches corresponding to each lexical unit into the preset resource pool 52 according to a preset memory expansion protocol sharding format. The controller is used to implement the steps of the user input information processing method for the model inference stage provided in the above method embodiment when executing computer programs.

[0101] The preset resource pool 52 includes multiple fast computing link devices, which are used to maintain cache consistency with the attention computing resource pool 51 and the feedforward network computing resource pool 53 through the memory expansion protocol.

[0102] The feedforward network computing resource pool 53 includes at least one first type of computing device with computing power reaching a preset computing power threshold. The first type of computing device is used to read intermediate features from the preset resource pool for decoding and storage operations.

[0103] Figure 6 This is a schematic diagram of the structure of a controller provided in this application. Figure 6 As shown, the controller 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the controller 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus.

[0104] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to execute the steps of the user input information processing method for the model inference stage provided in the above-described method embodiments.

[0105] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0106] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0107] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0108] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0109] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute, at runtime, the steps of any of the user input information processing methods for the model inference phase provided in the above method embodiments.

[0110] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0111] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps of any of the user input information processing methods for the model inference stage provided in the above-described method embodiments.

[0112] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of any of the user input information processing methods for the model inference stage provided in the above method embodiments.

[0113] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0114] The steps of a user input information processing method for the model inference stage provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for processing user input information during the model inference stage, characterized in that, The model inference stage includes a pre-filling stage, an attention stage, and a feedforward network stage, and the method includes: Obtain multiple complete suggestions input by the user, wherein the complete suggestions include multiple word elements; Regularly obtain hardware resource information and current task information; Based on the current task information, determine the task queue backlog and the average task sequence length; Based on the hardware resource information, determine the hardware resource utilization and memory pool access latency; The resource request skew coefficient corresponding to the complete prompt is determined based on the task queue backlog, the average task sequence length, the hardware resource utilization, the memory pool access latency, the preset task backlog penalty coefficient, and the preset custom weight. Based on the resource request skew coefficient and hardware resource information corresponding to the complete prompt, determine the hardware resource allocation information; In the pre-filling stage, based on the hardware resource allocation information, the complete prompt is pre-executed through parallel computing operations to obtain intermediate features and key-value caches corresponding to each word and store them in a preset resource pool. According to the hardware resource allocation information, in the feedforward network stage, a first type of computing device with computing power reaching a preset computing power threshold is alternately called to read the intermediate features from the preset resource pool for decoding and storage operations. In the attention stage, a second type of computing device with memory access performance reaching a preset memory access performance condition is called to read data from the preset resource pool for attention computing and storage operations until a preset decoding stop condition is triggered, and the model inference result is obtained and output.

2. The method according to claim 1, characterized in that, The step of the pre-filling stage, based on the hardware resource allocation information, involves performing parallel computation operations on the complete prompt to obtain intermediate features and key-value caches corresponding to each lexical unit and storing them in a preset resource pool, including: In the pre-filling stage, the complete prompt is broken down into multiple word units; Based on the hardware resource allocation information, multiple third-type computing devices with memory read and write performance reaching the preset read and write performance conditions are invoked to simultaneously perform self-attention calculation operations on multiple words to obtain intermediate features and key-value caches corresponding to each word. The intermediate features and key-value caches corresponding to each word are synchronously stored into the preset resource pool by the multiple third-class computing devices according to the preset memory expansion protocol sharding format.

3. The method according to claim 2, characterized in that, The preset memory expansion protocol sharding format divides the standard byte-sized data load into three load shards of first byte size and one load shard of second byte size, wherein the first byte size is greater than the second byte size, and the sum of the three first byte sizes and the second byte size is equal to the standard byte size.

4. The method according to claim 1, characterized in that, The process involves alternatingly calling a first type of computing device with computing power reaching a preset computing power threshold during the feedforward network phase to read intermediate features from the preset resource pool for decoding and storage operations, and calling a second type of computing device with memory access performance reaching a preset memory access performance condition during the attention phase to read data from the preset resource pool for attention computing and storage operations, until a preset decoding stop condition is triggered, to obtain and output the model inference result, including: According to the hardware resource allocation information, in the feedforward network stage, the first type of computing device with computing power reaching the preset computing power threshold is called to read the intermediate features from the preset resource pool and perform decoding calculation to obtain the first decoded word, and the first decoded word is saved to the preset resource pool; According to the hardware resource allocation information, during the attention phase, a second type of computing device whose memory access performance reaches the preset memory access performance condition is invoked to read the first decoded word and the key-value cache from the preset resource pool and perform attention calculation to obtain the attention result and the updated key-value cache and store them in the preset resource pool. Based on the hardware resource allocation information, in the feedforward network stage, the first type of computing device with computing power reaching the preset computing power threshold is called to read the attention result from the preset resource pool for decoding calculation, and the subsequent decoded words are obtained and stored in the preset resource pool. According to the hardware resource allocation information, during the attention phase, a second type of computing device whose memory access performance reaches the preset memory access performance condition is invoked to read the subsequent decoded lexical and the updated key-value cache from the preset resource pool for attention calculation. Then, the process jumps to the step of obtaining the attention result and the updated key-value cache and storing them in the preset resource pool until the preset decoding stop condition is triggered, and the model inference result is obtained and output.

5. The method according to claim 1, characterized in that, The formula for determining the resource request skew coefficient corresponding to each complete prompt based on the task queue backlog, the average task sequence length, the hardware resource utilization, the memory pool access latency, the preset task backlog penalty coefficient, and the preset custom weight is as follows: In the formula, W decode γ represents the resource request skew coefficient corresponding to the complete prompt, and γ represents the task backlog penalty coefficient. δ represents the preset custom weight, Gutil represents the hardware resource utilization rate, Avg_decode_len represents the average sequence length of the task, Mem_pool_lat represents the memory pool access latency, and Task_decode represents the task queue backlog.

6. The method according to claim 1, characterized in that, The step of determining hardware resource allocation information based on the resource request skew coefficient and hardware resource information corresponding to the complete prompt includes: The hardware resource ratio corresponding to each complete prompt is obtained by normalizing the resource request skew coefficients corresponding to the complete prompts. Hardware allocation information is determined based on the hardware resource ratio and the hardware resource information.

7. The method according to claim 1, characterized in that, The periodic acquisition of hardware resource information and current task information includes: According to a preset cycle, the interconnection switch interface of the fast link periodically obtains the list of connected computing devices and their memory capacity, and performs feature extraction operations to obtain task queue, computing device utilization, network latency as hardware resource information and current task information.

8. The method according to any one of claims 1 to 7, characterized in that, When determining the hardware resource allocation information corresponding to the complete prompt based on the hardware resource information and the current task information, the method further includes: Based on the hardware resource information and the current task information, iterative calculations are performed using reinforcement learning or a preset initialized Q-table to obtain the hardware resource allocation information corresponding to the complete prompt.

9. A user input information processing system for the model inference stage, characterized in that, include: Attention computing resource pool, preset resource pool and feedforward network computing resource pool; The attention computing resource pool includes multiple second-type computing devices with memory access performance reaching preset memory access performance conditions, multiple third-type computing devices with memory read / write performance reaching preset read / write performance conditions, and a controller. The second-type computing devices read data from the preset resource pool to perform attention computing and storage operations. The third-type computing devices are used to simultaneously perform self-attention computing operations on multiple lexical units to obtain intermediate features and key-value caches corresponding to each lexical unit, and synchronously store the intermediate features and key-value caches corresponding to each lexical unit into the preset resource pool according to a preset memory expansion protocol sharding format. The controller is used to implement the steps of the user input information processing method for the model inference stage as described in any one of claims 1 to 8 when executing a computer program. The preset resource pool includes multiple fast computing link devices, which are used to maintain cache consistency with the attention computing resource pool and the feedforward network computing resource pool through a memory expansion protocol. The feedforward network computing resource pool includes at least one first type of computing device with computing power reaching a preset computing power threshold. The first type of computing device is used to read the intermediate features from the preset resource pool for decoding and storage operations.

Images