User input information processing method and system for model reasoning stage

Abstract

The invention discloses a user input information processing method and system for a model reasoning stage, and belongs to the technical field of artificial intelligence, and the method comprises the steps: obtaining a plurality of complete prompts inputted by a user; periodically acquiring hardware resource information and current task information; determining hardware resource allocation information corresponding to the complete prompt; pre-executing parallel computing operation on the complete prompt according to the hardware resource allocation information in the pre-filling stage; according to the hardware resource allocation information, calling the first type of computing equipment with the computing power reaching a preset computing power threshold value in the feed-forward network stage to read the intermediate characteristics from the preset resource pool for decoding and storage operation, wherein the computing power of the first type of computing equipment reaches the preset computing power threshold value; and calling the second type of computing equipment with the memory access performance meeting the preset memory access performance condition in the attention stage to read data from the preset resource pool to perform attention computing storage operation, obtaining a model reasoning result and outputting the model reasoning result, thereby solving the technical problem of hardware resource waste, and achieving the technical effects of improving the resource utilization rate and reasoning efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a user input information processing method and system for a model inference stage. BACKGROUND

[0002] In the model inference stage of artificial intelligence, such as the prefill stage, the attention stage and the feedforward network (FFN) stage, the calculation characteristics of each stage are significantly different. In related technologies, the calculation modules of the attention and FFN are bound to the single hardware (such as a graphics processing unit, GPU) with the display memory resource, which causes the calculation resource and the memory resource to be unable to be dynamically allocated as needed. For example, in the prefill stage, the high calculation capability of the GPU can quickly process a large amount of input data, but the subsequent decoding stage is limited by the display memory bandwidth due to frequent access to the KV Cache, which causes the GPU utilization to decrease. In addition, in a multi-user concurrent scene (such as a chat robot and a document summary generation), the calculation requirements of different tasks are significantly different, and the traditional architecture is difficult to balance the resource allocation of the calculation-intensive and memory-intensive tasks, which causes the waste of hardware resources or the performance bottleneck.

[0003] Therefore, there is an urgent need for a user input information processing method for a model inference stage to improve the above problems. SUMMARY

[0004] The present application provides a user input information processing method and system for a model inference stage to at least solve the problems of low resource utilization and low inference efficiency in related technologies.

[0005] The present application provides a user input information processing method for a model inference stage, the model inference stage including a prefill stage, an attention stage and a feedforward network stage, and the method includes the following steps.

[0006] Obtaining a plurality of complete prompts input by a user, the complete prompt including a plurality of word pieces;

[0007] Periodically obtaining hardware resource information and current task information, and determining hardware resource allocation information corresponding to the complete prompt according to the hardware resource information and the current task information;

[0008] In the prefill stage, performing a parallel calculation operation on the complete prompt according to the hardware resource allocation information to obtain intermediate features corresponding to each word piece and a key-value cache, and storing the intermediate features and the key-value cache in a preset resource pool;

[0009] According to the hardware resource allocation information, the first type of computing device whose computing power reaches the preset computing power threshold is alternately called in the feedforward network stage to read the intermediate features from the preset resource pool for decoding and storage operations, and the second type of computing device whose memory access performance reaches the preset memory access performance condition is called in the attention stage to read data from the preset resource pool for attention calculation and storage operations, until the preset decoding stop condition is triggered, the model inference result is obtained and output.

[0010] The application also provides a user input information processing system for a model inference stage, which comprises: an attention calculation resource pool, a preset resource pool and a feedforward network calculation resource pool.

[0011] The attention calculation resource pool comprises a plurality of second type of computing devices whose memory access performance reaches a preset memory access performance condition, a plurality of third type of computing devices whose memory read-write performance reaches a preset read-write performance condition and a controller, the second type of computing device reads data from the preset resource pool for attention calculation and storage operations, the third type of computing device is used to simultaneously perform self-attention calculation operations on a plurality of word elements to obtain intermediate features corresponding to each word element and key-value caches, and the intermediate features corresponding to each word element and the key-value caches are synchronously stored into the preset resource pool in a preset memory expansion protocol fragmentation format, and the controller is used to implement the steps of the user input information processing method for the model inference stage when executing a computer program.

[0012] The preset resource pool comprises a plurality of computing fast link devices, and the computing fast link device is used to keep cache consistent with the attention calculation resource pool and the feedforward network calculation resource pool through a memory expansion protocol.

[0013] The feedforward network calculation resource pool comprises at least one first type of computing device whose computing power reaches a preset computing power threshold, and the first type of computing device is used to read intermediate features from the preset resource pool for decoding and storage operations.

[0014] Through the application, the hardware resource allocation information corresponding to the complete prompt of each task is determined first, and then the hardware resources are allocated in different stages of the model inference stage to complete different calculation processes, thereby improving the resource utilization rate and inference efficiency in different calculation scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 The related technical architecture diagram of the embodiments of the present application is provided.

[0017] Figure 2 A flowchart illustrating a user input information processing method for the model inference stage, provided in an embodiment of this application;

[0018] Figure 3 A schematic diagram of a transmission frame format for a preset memory expansion protocol fragmentation format provided in an embodiment of this application;

[0019] Figure 4 A schematic diagram of a user input information processing device for the model inference stage provided in this application embodiment;

[0020] Figure 5 A schematic diagram of a user input information processing system architecture for the model inference stage is provided in an embodiment of this application;

[0021] Figure 6 This is a schematic diagram of the structure of a controller provided in this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0023] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0024] Figure 1 A schematic diagram of the architecture of the related technologies provided in the embodiments of this application.

[0025] like Figure 1 As shown, each computational module in the model inference stage includes computational modules for attention and feedforward networks, and these modules are inseparable from the hardware. Figure 1The dashed box in the image represents a complete hardware entity upon which video memory and computation are based. This hardware entity can be a graphics processing unit (GPU) or other computing devices with similar functionality. As mentioned above, the significant difference in GPU utilization between Attention and FFN in related technologies confirms the differences in their computational characteristics. Due to memory-intensive operations, the computation of Attention is not significantly affected by changes in size. This leads to problems of low computational resource utilization and low processing efficiency in the inference process based on user input.

[0026] To address this issue, this invention provides the following inventive concept: by designing a dedicated resource pool as a relay, multimodal tasks are decoupled and deployed to their respective independent hardware modules, thus solving the problem of balancing the computational needs between modalities in a unified resource pool. For example, the processor in the image processing resource pool can efficiently perform convolution calculations, while the processor in the text processing resource pool can focus on high-bandwidth memory access during the Attention phase. Through modal-specific hardware configuration, the processing efficiency of multimodal tasks is significantly improved, while cross-modal communication overhead is reduced, providing a more flexible resource allocation scheme for complex task scenarios (such as text generation and video analysis).

[0027] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Figure 2 This is a flowchart illustrating a user input information processing method for the model inference stage, provided in an embodiment of this application.

[0029] like Figure 2 As shown, embodiments of this application provide a method for processing user input information during the model inference stage. The execution entity of this method can be a server or other hardware device with the same function. Based on Figure 2 The method is described in detail below. In this embodiment, the model inference stage includes a pre-filling stage, an attention stage, and a feedforward network stage. The method includes the following steps:

[0030] S201: Obtain multiple complete suggestions from user input, where each complete suggestion includes multiple lexical units.

[0031] In this embodiment, the user can simultaneously compute input prompt information in large batches using multiple computing devices to create a complete prompt. In this embodiment, the complete prompt can be a user-input instruction, a question, or text information used to guide the model in generating content. Each complete prompt includes multiple tokens. A token refers to the smallest unit into which the complete prompt is segmented, such as a word, phrase, or character.

[0032] S202: Periodically obtain hardware resource information and current task information, and determine the hardware resource allocation information corresponding to the complete prompt based on the hardware resource information and current task information.

[0033] In this embodiment, hardware resource information can be the utilization rate of currently idle computing hardware resources, such as the graphics processing unit (GPU), and current task information can be dynamic information about the task characteristics of the computing task to be executed and the priority information of each task to be executed, etc., which are related to task execution. Based on the hardware resource information and the current task information, the hardware resource allocation strategy can be generated through the algorithm deployed in the controller to determine the hardware resource allocation information corresponding to the complete prompt.

[0034] In an optional embodiment of this application, step S202 includes:

[0035] S202a: Determine the task queue backlog and average task sequence length based on the current task information.

[0036] In this embodiment, the task queue backlog represents the task waiting status at each stage of the inference phase, determining the resource allocation direction. For example, if the number of tasks waiting in the feedforward network stage is less than the number of tasks in the attention stage, then the resource allocation direction is to increase the number of calls to computing devices with high memory access performance. The average task sequence length represents the impact on memory usage and computational intensity, and is used to guide the computational task allocation strategy.

[0037] S202b: Determine hardware resource utilization and memory pool access latency based on hardware resource information.

[0038] In this embodiment, hardware resource information may include the number of idle hardware units and the number of active hardware units. Based on these two types of data, hardware resource utilization can be calculated. Memory access latency can be data inherent in the hardware monitoring data. Hardware resource utilization is used to assess hardware resource availability and avoid overload or idleness. Memory access latency is used to assess cross-node data transfer efficiency and optimize task scheduling priorities.

[0039] In an optional embodiment of this application, the method further includes: periodically obtaining a list of connected computing devices and their memory capacity through the interconnection switch interface of the computing fast link at a preset period, and performing feature extraction operations to obtain task queues, computing device utilization, and network latency as hardware resource information and current task information.

[0040] In this embodiment, hardware resource information can be obtained by acquiring the list of connected computing devices and their memory capacity through the CXL switch interface API. Then, feature extraction is performed, for example, by collecting data such as task queues, computing device utilization (e.g., GPU utilization), and network latency at a preset period of 100ms.

[0041] S202c: determining the resource request tilt coefficient corresponding to the complete prompt according to the task queue backlog, the average task sequence length, the hardware resource utilization rate, the memory pool access delay, the preset task backlog penalty coefficient and the preset custom weight.

[0042] In this embodiment, the corresponding resource request tilt coefficient is calculated based on the task queue backlog, the average task sequence length, the hardware resource utilization rate, the memory pool access delay, the preset task backlog penalty coefficient and the preset custom weight through a specific calculation formula.

[0043] In an optional embodiment of the present application, the calculation formula of step S202c is:

[0044]

[0045] In the formula, W decode represents the resource request tilt coefficient corresponding to the complete prompt, γ represents the task backlog penalty coefficient, and δ are preset custom weights, Gutil represents the hardware resource utilization rate, Avg_decode_len represents the average task sequence length, Mem_pool_lat represents the memory pool access delay, and Task_decode represents the task queue backlog.

[0046] S202d: determining the hardware resource allocation information according to the resource request tilt coefficient corresponding to the complete prompt and the hardware resource information.

[0047] In this embodiment, the allocation proportion of each computing device is dynamically adjusted according to the resource request tilt coefficient in the processing process of the current task, i.e., the complete prompt, and the hardware resource information such as GPU computing power.

[0048] Specifically, in an optional embodiment of the present application, step S202d includes:

[0049] d1: performing normalization calculation according to the resource request tilt coefficient corresponding to the complete prompt to obtain the hardware resource proportion corresponding to each complete prompt.

[0050] d2: determining the hardware allocation information according to the hardware resource proportion and the hardware resource information.

[0051] In this embodiment, normalization refers to a scaling rule for multiple resource request tilt coefficients or weights, so that each value is in a specific numerical range interval, such as the interval (0, 1). In this embodiment, the hardware allocation information can be determined by normalizing the resource request tilt coefficients of multiple task requests to determine the allocation proportion of the computing device, and then dynamically adjusting according to the GPU computing power to balance the throughput and delay.

[0052] In an optional embodiment of the present application, step S202d is further performed when:

[0053] Step d3: performing iterative calculation processing according to the hardware resource information and the current task information through reinforcement learning or preset initialization of a Q table to obtain hardware resource allocation information corresponding to the complete prompt.

[0054] In this embodiment, reinforcement learning is an algorithm that allows an agent (which can be understood as a "weight optimization module") to learn the optimal behavior as much as possible in an environment (which is a "large model inference system") through "trial and error" - the agent gradually learns the weight value that can maximize the "reward" (comply with the performance target) by performing actions (adjusting weights , δ), observing environmental feedback (such as whether the inference delay is reduced), and gradually learning the weight value that can maximize the "reward" (comply with the performance target). Preset initialization of a Q table for iterative calculation processing is one of the algorithms of reinforcement learning, which learns the "state-action value function Q (s, a)" (i.e., the long-term reward obtained by performing action a in state s), and finds the optimal action (here, the action is "selecting , δ specific values") by constantly updating the Q value. This step aims to automatically optimize the values of the custom weights , δ through reinforcement learning or preset initialization of a Q table Q-Learning, so that the weights can better adapt to the performance target of large model inference (such as reducing delay and improving key-value cache reuse rate).

[0055] S203: In the pre-filling stage, perform pre-execution parallel computing operations on the complete prompt according to the hardware resource allocation information to obtain intermediate features corresponding to each token and key-value caches, and store them in a preset resource pool.

[0056] In this embodiment, the pre-filling Prefill stage can use multiple computing devices such as graphics processing units GPU to perform parallel processing of multiple token computing operations, and then output intermediate features Hidden feature and cache key-value KVcache corresponding to each token. After that, the intermediate features Hidden feature and cache key-value KVcache can be stored in the preset resource pool through the corresponding computing fast link format flit data frame.

[0057] In this embodiment, the preset resource pool can be a hardware memory resource pool for storing data constructed by a third type of computing fast link device type3 device based on a computing fast link, where the type3 device can maintain cache consistency with the feedforward network resource pool through the computing fast link protocol (CXL.mem protocol) and attention resource pool.

[0058] Specifically, in an optional embodiment of the present application, step S203 comprises:

[0059] S203a: disassembling the complete prompt to obtain a plurality of word pieces in the pre-filling stage.

[0060] In this embodiment, the disassembling process refers to disassembling the complete prompt containing a plurality of word pieces into a plurality of word pieces according to the pre-specified word piece disassembling rule, such as disassembling into a plurality of instruction word pieces according to the instruction type.

[0061] S203b: According to the hardware resource allocation information, a plurality of third type computing devices with memory read-write performance reaching the preset read-write performance condition are called to simultaneously perform self-attention computing operation on the plurality of word pieces to obtain the intermediate features and key-value caches corresponding to each word piece.

[0062] In this embodiment, the hardware resource allocation information can include the content of grouping the plurality of word pieces by a fixed number to call a corresponding number of third type computing devices, and the plurality of third type computing devices simultaneously perform self-attention computing operation on the plurality of word pieces in each group to obtain the intermediate features and key-value caches. For example: there are originally 20 word pieces Token, which are grouped by every 5 Token, and according to the hardware resource allocation information, 4 GPUs are allocated, GPU1 processes Token1-5, GPU2 processes Token6-10, GPU3 processes Token11-15, and GPU4 processes Token16-20.

[0063] S203c: The intermediate features and key-value caches corresponding to each word piece are synchronously stored into the preset resource pool by the plurality of third type computing devices according to the preset memory expansion protocol fragmentation format.

[0064] In this embodiment, the preset memory expansion protocol fragmentation format can be a standard 256B CXL Flit transmission frame format.

[0065] In an optional embodiment of the present application, the preset memory expansion protocol fragmentation format is to divide the data load of a standard byte amount into three first byte amount load fragments and one second byte amount load fragment, wherein the first byte amount is greater than the second byte amount, and the sum of the byte amounts of the three first byte amounts and the second byte amount is equal to the standard byte amount.

[0066] Figure 3 A transmission frame format diagram of a preset memory expansion protocol fragmentation format provided in an embodiment of the present application.

[0067] As Figure 3As shown, in this embodiment, the transmission frame format remains compatible with the standard 256B CXL Flit transmission frame format, but the difference is that the 240B data in the middle is divided into 4 blocks, three of which are 64B payloads for transmission with attention ATTN, and the last one is 48B payload for transmission of Token vector and intermediate features. The reason is that the key-value cache multiplexing transmission has higher requirements for bytes and latency than the intermediate features and word vector, and generally the intermediate features and word vector are single transmission, while the key-value cache contains reading multiplexing of historical data and writing operation of newly generated key-value cache.

[0068] S204: According to the hardware resource allocation information, alternately cycle the first type of computing device whose computing power reaches the preset computing power threshold in the feedforward network stage to read the intermediate features from the preset resource pool for decoding and storage operation, and the second type of computing device whose memory access performance reaches the preset memory access performance condition in the attention stage to read data from the preset resource pool for attention calculation and storage operation, until the preset decoding stop condition is triggered, to obtain the model inference result and output.

[0069] In this embodiment, the hardware resource allocation information can include the allocation of corresponding computing device resources according to hardware usage requirements in different inference stages. Generally, the controller dynamically senses task information and hardware resource monitoring data to allocate resources.

[0070] In the feedforward network stage, the first type of computing device such as high-performance graphics processor whose computing power reaches the preset computing power threshold is used to make nonlinear changes to the intermediate features in the pre-padding stage to output decoded words, wherein the preset computing power threshold can be a large processor computing power value in the industry preset by a user. In the attention stage, the second type of computing device with high memory access performance is used to calculate attention weight in combination with the previous key-value cache and newly generated words to capture context association. Thus, the alternately cycling operation is performed in the feedforward network stage and the attention stage. Until the preset decoding stop condition is triggered. In this embodiment, the preset decoding stop condition can be that the number of generated words reaches an upper limit value, or that the last word obtained by decoding is a special punctuation symbol such as a period at the end of the text, etc.

[0071] Based on the above embodiment, in an optional embodiment of the present application, step S204 includes:

[0072] S204a: According to the hardware resource allocation information, in the feedforward network stage, the first type of computing device whose computing power reaches the preset computing power threshold is used to read the intermediate features from the preset resource pool and perform decoding calculation to obtain the first decoded word, and the first decoded word is saved to the preset resource pool.

[0073] In this embodiment, the first decoding token output is obtained by reading the intermediate features of the pre-filling stage from the preset resource pool in the feedforward network stage. In this embodiment, saving the first decoding token to the preset resource pool can be writing the first decoding token to a specific storage address in the preset resource pool in the form of a vector. For example: the GPU 5 of the feedforward network FFN resource pool reads the complete 20x1024-dimensional Hidden feature matrix from the preset resource pool address 0x0001~0x0008 through the CXL.mem protocol. And perform matrix multiplication GEMM, first multiply the Hidden feature matrix with the weight matrix W1 (1024x4096-dimensional), then pass through the activation function ReLU, and then multiply it with the weight matrix W2 (4096x1024-dimensional), to get the 20x1024-dimensional transformed feature, and perform Softmax operation on the transformed feature, select the Token with the highest probability as the first generated result, denoted as Token21. Then through the CXL flit frame (64B fragmentation, occupying 1 fragment), write the 1024-dimensional vector of Token21 into the address 0x0017 of the memory resource pool, and record the metadata (generation time, corresponding input Prompt correlation degree, etc.) of Token21.

[0074] S204b: According to the hardware resource allocation information, the second type of computing device with memory access performance reaching the preset memory access performance condition reads the first decoding token and key value cache from the preset resource pool in the attention stage and performs attention calculation to obtain the attention result and the updated key value cache and stores them in the preset resource pool.

[0075] In this embodiment, the main process of reading the first decoding token and the key value cache from the preset resource pool and performing attention calculation to obtain the attention result and the updated key value cache is as follows: first, the attention stage calls a high memory performance processor such as GPU1, which can be allocated by a controller or a scheduler, reads two types of data from the preset resource pool through the CXL.mem protocol, the newly generated token such as Token21 and the updated key value cache corresponding to the newly generated token, and the key value cache corresponding to the first 20 tokens generated in the pre-filling stage. The process of generating a query vector Q vector, calculating similarity, weight normalization and calculating output.

[0076] Exemplarily, the vector of Token21 is multiplied by the query weight matrix WQ (1024x1024 dimensions) of the attention layer to obtain a 1x1024-dimensional query Q vector. The Q vector is multiplied by the transpose of the key K matrix (1024x20 dimensions) to obtain a 1x20-dimensional similarity matrix (indicating the association strength of Token21 with input Tokens 1~20), a Softmax operation is performed on the similarity matrix to obtain normalized attention weights (such as Token21 and Token5 weight 0.18, and Token8 weight 0.23), and finally the attention weights are multiplied by the value V matrix to obtain a 1x1024-dimensional attention result (fusing input context information).

[0077] The process of storing the attention result and the updated key-value cache in the preset resource pool can be to take the query vector of the new token as a new key K vector and the intermediate value of the attention result as a new value V vector (both 1024 dimensions). GPU1 writes the 1x1024-dimensional attention result to the memory address 0x0018 through the CXL flit frame, and appends the newly generated key-value cache (K / V vector) to the KVcache (addresses 0x0019~0x0020), at which time the total number of KVcache is updated to 21 groups (corresponding to Tokens 1~21).

[0078] S204c: According to the hardware resource allocation information, the first type of computing device whose computing power reaches the preset computing power threshold in the feedforward network stage reads the attention result from the preset resource pool for decoding calculation to obtain subsequent decoding tokens and store them in the preset resource pool.

[0079] S204d: According to the hardware resource allocation information, the second type of computing device whose memory access performance reaches the preset memory access performance condition in the attention stage reads the subsequent decoding tokens and the updated new key-value cache from the preset resource pool for attention calculation, and jumps to the step of obtaining the attention result and the updated key-value cache and storing them in the preset resource pool until the preset decoding stop condition is triggered, obtaining the model inference result and outputting.

[0080] In this embodiment, in the feedforward network stage, a first type of computing device different from step S204a is called to read the attention result from the preset resource pool for decoding calculation. The principle and specific decoding example of decoding are similar to step S204a. The process of storing the subsequent decoding word units in the preset resource pool after decoding calculation is also similar to step S204a. Therefore, this embodiment will not be described here. The subsequent decoding word units can generate the query vector of the 22nd word unit. Similar to the calculation of the similarity and normalized weight of the previous 21 groups of key vectors, the attention result of the context of word units 1 to 22 is finally fused. The principle and process of step S204d are similar to step S204b. Therefore, this embodiment will not be described here. The above steps are repeatedly and alternately executed. Different stages call different computing devices to ensure that steps S204a and S204c are calculated in parallel and efficiently with steps S204b and S204d. The resource utilization rate is improved, and the data calculation processing efficiency is also improved.

[0081] In summary, the method for processing user input information in the model inference stage provided by the embodiments of the present application first determines the hardware resource allocation information corresponding to the complete prompts of each task, and then allocates hardware resources in different stages of the model inference stage to complete different calculation processes, thereby improving the resource utilization rate and inference efficiency in different calculation scenarios.

[0082] Meanwhile, the dynamic scheduling algorithm is used to optimize the resource request skew coefficient according to the current task information and the hardware resource allocation information, and then the hardware resource allocation information is dynamically adjusted. The accuracy and applicability of resource use allocation are further improved.

[0083] Meanwhile, the unified preset resource pool memory architecture is used to solve the communication overhead between stages in the model inference stage.

[0084] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0085] Figure 4 A structural schematic diagram of a device for processing user input information in the model inference stage provided by the embodiments of the present application is shown in FIG. 4. As shown in FIG. 4, the embodiments of the present application also provide a device for processing user input information in the model inference stage, which comprises an acquisition module 41, a parallel computing module 42 and a dynamic processing module 43. Figure 4

[0086] The acquisition module 41 is used to acquire a plurality of complete prompts input by a user, and the complete prompts comprise a plurality of word units.

[0087] ​The acquisition module 41 is further configured to periodically acquire the hardware resource information and the current task information, and determine the complete prompt corresponding hardware resource allocation information according to the hardware resource information and the current task information.

[0088] The parallel computing module 42 is configured to perform a pre-execution parallel computing operation on the complete prompt according to the hardware resource allocation information in a pre-filling stage, to obtain the intermediate features corresponding to each token and the key-value cache, and store them in a preset resource pool.

[0089] The dynamic processing module 43 is configured to alternately cycle, according to the hardware resource allocation information, a decoding storage operation of reading the intermediate features from the preset resource pool by the first type of computing device whose computing power reaches a preset computing power threshold in a feedforward network stage, and an attention computing storage operation of reading data from the preset resource pool by the second type of computing device whose memory access performance reaches a preset memory access performance condition in an attention stage, until a preset decoding stop condition is triggered, to obtain a model inference result and output.

[0090] In an optional embodiment of the present application, the parallel computing module 42 is specifically configured to: in the pre-filling stage, disassemble the complete prompt to obtain a plurality of tokens; according to the hardware resource allocation information, simultaneously perform a self-attention computing operation on the plurality of tokens by the third type of computing device whose memory read-write performance reaches a preset read-write performance condition, to obtain the intermediate features corresponding to each token and the key-value cache; and store the intermediate features corresponding to each token and the key-value cache in the preset resource pool in a preset memory expansion protocol fragmentation format through the plurality of third type of computing devices.

[0091] In an optional embodiment of the present application, the dynamic processing module 43 is specifically configured to: according to the hardware resource allocation information, in the feedforward network stage, read the intermediate features from the preset resource pool by the first type of computing device whose computing power reaches the preset computing power threshold, and perform decoding calculation to obtain a first decoded token, and save the first decoded token to the preset resource pool; according to the hardware resource allocation information, in the attention stage, read the first decoded token and the key-value cache from the preset resource pool by the second type of computing device whose memory access performance reaches the preset memory access performance condition, and perform attention calculation to obtain an attention result and an updated key-value cache and store them in the preset resource pool; according to the hardware resource allocation information, in the feedforward network stage, read the attention result from the preset resource pool by the first type of computing device whose computing power reaches the preset computing power threshold, and perform decoding calculation to obtain a subsequent decoded token and store it in the preset resource pool; according to the hardware resource allocation information, in the attention stage, read the subsequent decoded token and the updated new key-value cache from the preset resource pool by the second type of computing device whose memory access performance reaches the preset memory access performance condition, and perform attention calculation, and jump to the step of obtaining the attention result and the updated key-value cache and storing them in the preset resource pool, until the preset decoding stop condition is triggered, to obtain a model inference result and output.

[0092] In an optional embodiment of this application, the acquisition module 41 is further specifically used to: determine the task queue backlog and the average task sequence length based on the current task information; determine the hardware resource utilization and memory pool access latency based on the hardware resource information; determine the resource request skew coefficient corresponding to the complete prompt based on the task queue backlog, the average task sequence length, the hardware resource utilization, the memory pool access latency, the preset task backlog penalty coefficient, and the preset custom weight; and determine the hardware resource allocation information based on the resource request skew coefficient corresponding to the complete prompt and the hardware resource information.

[0093] In an optional embodiment of this application, the acquisition module 41 is specifically used to determine the calculation formula for the resource request skew coefficient corresponding to each complete prompt based on the task queue backlog, average task sequence length, hardware resource utilization, memory pool access latency, preset task backlog penalty coefficient, and preset custom weight.

[0094]

[0095] In the formula, W decode This represents the resource request skew coefficient corresponding to the complete prompt, and γ represents the task backlog penalty coefficient. δ represents a preset custom weight, Gutil represents hardware resource utilization, Avg_decode_len represents the average sequence length of the task, Mem_pool_lat represents the memory pool access latency, and Task_decode represents the task queue backlog.

[0096] In an optional embodiment of this application, the acquisition module 41 is further configured to: perform normalization calculation based on the resource request skew coefficient corresponding to each complete prompt to obtain the hardware resource ratio corresponding to each complete prompt; and determine the hardware allocation information based on the hardware resource ratio and hardware resource information.

[0097] In an optional embodiment of this application, the acquisition module 41 is further configured to: perform iterative calculation processing based on hardware resource information and current task information through reinforcement learning or a preset initialized Q-table to obtain the hardware resource allocation information corresponding to the complete prompt.

[0098] The description of the features in the embodiment of the user input information processing device for the model inference stage provided in this application can be referred to the relevant description of the embodiment of the user input information processing method for the model inference stage described above, and will not be repeated here.

[0099] Figure 5 This is a schematic diagram of a user input information processing system architecture for the model inference stage, provided in an embodiment of this application.

[0100] like Figure 5As shown, the system includes an attention computing resource pool 51, a preset resource pool 52, and a feedforward network computing resource pool 53. The attention computing resource pool 51 includes multiple second-type computing devices with memory access performance meeting preset memory access performance conditions, multiple third-type computing devices with memory read / write performance meeting preset read / write performance conditions, and a controller. The second-type computing devices read data from the preset resource pool 52 for attention computing and storage operations. The third-type computing devices are used to simultaneously perform self-attention computing operations on multiple lexical units, obtaining intermediate features and key-value caches corresponding to each lexical unit, and synchronously storing the intermediate features and key-value caches corresponding to each lexical unit into the preset resource pool 52 according to a preset memory expansion protocol sharding format. The controller is used to implement the steps of the user input information processing method for the model inference stage provided in the above method embodiment when executing computer programs.

[0101] The preset resource pool 52 includes multiple fast computing link devices, which are used to maintain cache consistency with the attention computing resource pool 51 and the feedforward network computing resource pool 53 through the memory expansion protocol.

[0102] The feedforward network computing resource pool 53 includes at least one first type of computing device with computing power reaching a preset computing power threshold. The first type of computing device is used to read intermediate features from the preset resource pool for decoding and storage operations.

[0103] Figure 6 This is a schematic diagram of the structure of a controller provided in this application. Figure 6 As shown, the controller 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the controller 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus.

[0104] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to execute the steps of the user input information processing method for the model inference stage provided in the above-described method embodiments.

[0105] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0106] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. The steps of the method disclosed in combination with the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0107] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0108] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0109] Embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps of the user input information processing method for the model inference stage provided by any of the above method embodiments when running.

[0110] In an exemplary embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0111] Embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps of the user input information processing method for the model inference stage provided by any of the above method embodiments.

[0112] Embodiments of the present application further provide another computer program product comprising a non-transitory computer-readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps of the method for processing user input information in a model inference phase according to any of the above method embodiments.

[0113] Those skilled in the art will further appreciate that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or both, and that the implementation decisions are within the skill of an expert in the art to make based on available technology and design constraints. The examples have been described in general terms above in order to illustrate the functionality of the components and steps, and the choice between hardware and software implementation is a design choice that can be made by a skilled person for a particular application. The described functionality can be implemented in different ways for each particular application, but such implementation should not be considered to be outside the scope of the present application.

[0114] The above describes in detail the steps of the method for processing user input information in a model inference phase according to the present application. The principles and implementation modes of the present application are described herein by applying specific examples, and the above description of the examples is only to help understand the method of the present application and its core idea. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A method for processing user input information during the model inference stage, characterized in that, The model inference stage includes a pre-filling stage, an attention stage, and a feedforward network stage, and the method includes: Obtain multiple complete suggestions input by the user, wherein the complete suggestions include multiple word elements; Periodically acquire hardware resource information and current task information, and determine the hardware resource allocation information corresponding to the complete prompt based on the hardware resource information and the current task information; In the pre-filling stage, based on the hardware resource allocation information, the complete prompt is pre-executed through parallel computing operations to obtain intermediate features and key-value caches corresponding to each word and store them in a preset resource pool. According to the hardware resource allocation information, in the feedforward network stage, a first type of computing device with computing power reaching a preset computing power threshold is alternately called to read the intermediate features from the preset resource pool for decoding and storage operations. In the attention stage, a second type of computing device with memory access performance reaching a preset memory access performance condition is called to read data from the preset resource pool for attention computing and storage operations until a preset decoding stop condition is triggered, and the model inference result is obtained and output.

2. The method of claim 1, wherein, The step of the pre-filling stage, based on the hardware resource allocation information, involves performing parallel computation operations on the complete prompt to obtain intermediate features and key-value caches corresponding to each word and storing them in a preset resource pool, including: In the pre-filling stage, the complete prompt is broken down into multiple word units; Based on the hardware resource allocation information, multiple third-type computing devices with memory read and write performance reaching the preset read and write performance conditions are invoked to simultaneously perform self-attention calculation operations on multiple words to obtain intermediate features and key-value caches corresponding to each word. The intermediate features and key-value caches corresponding to each word are synchronously stored into the preset resource pool by multiple third-class computing devices according to a preset memory expansion protocol sharding format.

3. The method of claim 2, wherein, The preset memory expansion protocol sharding format divides the standard byte-sized data load into three load shards of first byte size and one load shard of second byte size, wherein the first byte size is greater than the second byte size, and the sum of the three first byte sizes and the second byte size is equal to the standard byte size.

4. The method of claim 1, wherein, The process involves alternatingly calling a first type of computing device with computing power reaching a preset computing power threshold during the feedforward network phase to read intermediate features from the preset resource pool for decoding and storage operations, and calling a second type of computing device with memory access performance reaching a preset memory access performance condition during the attention phase to read data from the preset resource pool for attention computing and storage operations, until a preset decoding stop condition is triggered, to obtain and output the model inference result, including: According to the hardware resource allocation information, in the feedforward network stage, the first type of computing device with computing power reaching the preset computing power threshold is called to read the intermediate features from the preset resource pool and perform decoding calculation to obtain the first decoded word, and the first decoded word is saved to the preset resource pool; According to the hardware resource allocation information, a second type of computing device with memory access performance reaching a preset memory access performance condition is called in the attention stage to read the first decoding token and the key-value cache from the preset resource pool and perform attention calculation, obtain an attention result and an updated key-value cache, and store them in the preset resource pool; According to the hardware resource allocation information, a first type of computing device with computing power reaching a preset computing power threshold is called in the feedforward network stage to read the attention result from the preset resource pool to perform decoding calculation, obtain a subsequent decoding token, and store it in the preset resource pool; According to the hardware resource allocation information, a second type of computing device with memory access performance reaching a preset memory access performance condition is called in the attention stage to read the subsequent decoding token and the updated new key-value cache for attention calculation, and jump to the step of obtaining the attention result and the updated key-value cache and storing them in the preset resource pool until a preset decoding stop condition is triggered, obtaining a model inference result, and outputting.

5. The method of claim 1, wherein, The determining of the hardware resource allocation information corresponding to the complete prompt according to the hardware resource information and the current task information comprises: determining a task queue backlog and a task average sequence length according to the current task information; determining a hardware resource utilization rate and a memory pool access delay according to the hardware resource information; determining a resource request tilt coefficient corresponding to the complete prompt according to the task queue backlog, the task average sequence length, the hardware resource utilization rate, the memory pool access delay, a preset task backlog penalty coefficient, and a preset custom weight; determining the hardware resource allocation information according to the resource request tilt coefficient corresponding to the complete prompt and the hardware resource information.

6. The method of claim 5, wherein, The calculation formula for determining the resource request tilt coefficient corresponding to each complete prompt according to the task queue backlog, the task average sequence length, the hardware resource utilization rate, the memory pool access delay, the preset task backlog penalty coefficient, and the preset custom weight is: In the formula, W decode represents the resource request inclination coefficient corresponding to the complete prompt, γ represents the task backlog penalty coefficient, and δ are the preset custom weights, Gutil represents the hardware resource utilization rate, Avg_decode_len represents the average sequence length of the task, Mem_pool_lat represents the memory pool access delay, and Task_decode represents the task queue backlog.

7. The method of claim 5, wherein, The determining of the hardware resource allocation information according to the resource request tilt coefficient corresponding to the complete prompt and the hardware resource information comprises: performing normalization calculation on the resource request tilt coefficient corresponding to the complete prompt to obtain a hardware resource proportion corresponding to each complete prompt; determining the hardware allocation information according to the hardware resource proportion and the hardware resource information.

8. The method of claim 1, wherein, The periodic acquisition of hardware resource information and current task information comprises: acquiring a connected computing device list and a video memory capacity through a computing fast link interconnection switch interface periodically according to a preset period, and performing feature extraction to obtain a task queue, a computing device utilization rate, and a network delay as the hardware resource information and the current task information.

9. The method according to any one of claims 1 to 8, characterized in that, When the hardware resource allocation information corresponding to the complete prompt is determined according to the hardware resource information and the current task information, the method further comprises: performing iterative calculation and processing on the hardware resource allocation information corresponding to the complete prompt according to the hardware resource information and the current task information through reinforcement learning or a preset initialized Q table.

10. A user input information processing system for the model inference stage, characterized in that, comprises: The attention computing resource pool, the preset resource pool, and the feedforward network computing resource pool; The attention computing resource pool includes a plurality of second-type computing devices with memory access performance reaching a preset memory access performance condition, a plurality of third-type computing devices with memory read-write performance reaching a preset read-write performance condition, and a controller. The second-type computing devices read data from the preset resource pool for attention computing and storage operation. The third-type computing devices are used to simultaneously perform self-attention computing operation on a plurality of word units to obtain intermediate features and key-value caches corresponding to each word unit, and store the intermediate features and key-value caches corresponding to each word unit into the preset resource pool in a preset memory expansion protocol fragmentation format. The controller is used to implement the steps of the method for processing user input information in the model inference stage according to any one of claims 1 to 9 when executing a computer program. The preset resource pool includes a plurality of computing fast link devices, which are used to maintain cache consistency with the attention computing resource pool and the feedforward network computing resource pool through a memory expansion protocol. The feedforward network computing resource pool includes at least one first-type computing device with computing power reaching a preset computing power threshold, which is used to read the intermediate features from the preset resource pool for decoding and storage operation.

Images