Model inference method and device, computer device, and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a model inference method and device, computer equipment and a storage medium, the method comprising the following steps: receiving a plurality of inference requests, wherein the inference request comprises a prompt word; allocating a local filtering cache space for each inference request, wherein the local filtering cache space is used for storing a local filtering cache value; obtaining the local filtering cache value in the local filtering cache space of each inference request, splicing the local filtering cache value to obtain a spliced sequence; performing multiple times of inference based on the prompt word and the spliced sequence of each inference request by using a large language model to obtain an inference result of each inference request; and updating the local filtering cache value stored in the local filtering cache space of each inference request after each inference is completed, so that the next inference is performed based on the updated local filtering cache value. In a high-concurrency scene, the application can accurately perform inference on the inference request, and ensures the accuracy of the inference result.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to model reasoning methods, apparatus, computer equipment, and storage media. Background Technology

[0002] The Source 2.0 large language model receives prompts from the user and generates an inference result after multiple inferences. Each inference generates a new token, which serves as the input for the next inference. During each inference process, the key-value pairs generated by each transformer block and the local filter cache value (lf_cache value) are cached for use in the next inference.

[0003] In related technologies, cache space is allocated for local filtering cache values ​​according to a fixed-size strategy. For any inference request, a region is partitioned from the pre-allocated cache space to store the local filtering cache value for that inference request.

[0004] However, this method of caching local filter cache values ​​may fail to clear the local filter cache value of an inference request in a timely manner when the model receives multiple concurrent inference requests and any one of the inference requests is processed prematurely. As a result, the model uses the uncleared local filter cache value when inferring other inference requests, making it unable to accurately infer other inference requests and obtaining inaccurate inference results. Summary of the Invention

[0005] In view of this, the present invention provides a model inference method to solve the problem that the method of caching local filter cache values ​​in related technologies may cause the model to use local filter cache values ​​that have not been cleared in time during the inference process of other inference requests, thus failing to accurately infer other inference requests and obtaining inaccurate inference results.

[0006] In a first aspect, the present invention provides a model reasoning method, the method comprising:

[0007] Receive multiple inference requests input by the user, the inference requests including prompt words;

[0008] Allocate a local filter cache space for each of the inference requests, the local filter cache space being used to store local filter cache values, and the local filter cache space being associated with the inference request;

[0009] Obtain the local filter cache value in the local filter cache space of each inference request, and concatenate the local filter cache values ​​to obtain a concatenation sequence;

[0010] The large language model is used to perform multiple inferences based on the prompts for each inference request and the concatenated sequence to obtain the inference results for each inference request;

[0011] Specifically, after each inference is completed, the local filter cache value stored in the local filter cache space of each inference request is updated so that the next inference can be performed based on the updated local filter cache value.

[0012] The model inference method provided in this embodiment receives multiple inference requests input by the user, allocates a local filtering cache space for each inference request, obtains the local filtering cache values ​​in the local filtering cache space of each inference request, concatenates the local filtering cache values ​​to obtain a concatenated sequence, and uses a large language model to perform multiple inferences based on the prompt words of each inference request and the concatenated sequence to obtain the inference result of each inference request. After each inference, the local filtering cache values ​​stored in the local filtering cache space of each inference request are updated so that the next inference is based on the updated local filtering cache values. By allocating a unique local filtering cache space for each inference request to store the local filtering cache values ​​updated by the model during the inference process, the large language model will not use incorrect local filtering cache values ​​in high-concurrency scenarios, and can perform efficient and accurate inference for all inference requests, ensuring the accuracy of the inference results.

[0013] In one optional implementation, the step of using a large language model to perform multiple inferences based on the prompts for each inference request and the concatenated sequence to obtain the inference result for each inference request includes:

[0014] When performing the first reasoning on any reasoning request using a large language model, the prompt words of the reasoning request are converted into a vector sequence, and the vector sequence is input into a local filtering attention layer so that the local filtering attention layer performs forward propagation calculation based on the vector sequence and the concatenated sequence.

[0015] When using a large language model to perform a second and subsequent inference for any inference request, the token generated by the previous inference of the inference request is converted into a vector, and the vector is input into a local filtering attention layer so that the local filtering attention layer performs forward propagation calculation based on the vector and the updated concatenated sequence.

[0016] For any reasoning request, when the reasoning request meets the preset reasoning termination condition, obtain the tag generated for each reasoning of the reasoning request;

[0017] Based on the tags generated in each inference, the inference result of the inference request is determined.

[0018] The model inference method provided in this embodiment converts the prompt words into a vector sequence during the first inference, and inputs the vector sequence into a local filtering attention layer for forward propagation calculation. During the second and subsequent inferences, the markers generated in the previous inference are introduced, converted into vectors, and input into the local filtering attention layer for forward propagation calculation. By gradually accumulating and updating contextual information through multiple inferences, a more accurate and efficient inference process is achieved.

[0019] In an optional implementation, the method further includes:

[0020] The spliced ​​sequence is updated during each inference process;

[0021] After each inference iteration, the local filter cache value stored in the local filter cache space of each inference request is updated so that the next inference iteration is based on the updated local filter cache value. This includes:

[0022] After each inference, the markers generated by this inference and the concatenated sequence of this inference update for each inference request are obtained. Based on the concatenated sequence of this inference update, the local filter cache values ​​stored in the local filter cache space of each inference request are updated to obtain the updated local filter cache values ​​in the local filter cache space of each inference request. The updated local filter cache values ​​are concatenated to obtain the updated concatenated sequence. The next inference is performed using the large language model based on the updated concatenated sequence and the markers generated by this inference.

[0023] The model inference method provided in this embodiment updates the concatenated sequence during each inference process. After each inference, the local filter cache value stored in the local filter cache space of each inference request is updated based on the concatenated sequence updated in this inference. The next inference is then performed based on the updated concatenated sequence obtained by concatenating the updated local filter cache value. This ensures that the local filter cache value corresponding to the inference request is accurately obtained. The next inference is then performed based on the obtained local filter cache value, thus guaranteeing the accuracy of the inference.

[0024] In one optional implementation, the large language model includes multiple transformation layers, each transformation layer including a local filtering attention layer, and updating the concatenated sequence during each inference process includes:

[0025] When performing the first inference on any inference request using a large language model, after passing through the local filtering attention layer in the first conversion layer, the first local filtering cache value of the first conversion layer corresponding to the inference request is obtained from the first concatenation sequence. The first local filtering cache value of the first conversion layer and the vector sequence corresponding to the inference request are concatenated to obtain the first concatenation result. The first local filtering cache value of the first conversion layer corresponding to the inference request in the first concatenation sequence is updated to the vector sequence corresponding to the inference request.

[0026] Based on the first convolutional layer, a convolution operation is performed on the first concatenation result to obtain a first convolution result. The second local filter cache value of the first transformation layer corresponding to the inference request is obtained from the second concatenation sequence. The second local filter cache value of the first transformation layer and the first convolution result are concatenated to obtain a second concatenation result. A convolution operation is performed on the second concatenation result to obtain a second convolution result. The second convolution result is input into the attention layer for forward propagation calculation. The second local filter cache value of the first transformation layer corresponding to the inference request in the second concatenation sequence is updated to the first convolution result.

[0027] When the inference request is inferred for the first time using a large language model, and after passing through the local filtering attention layer of any of the other conversion layers besides the first conversion layer, the first local filtering cache value of the conversion layer corresponding to the inference request is obtained from the first concatenation sequence. The first local filtering cache value of the conversion layer and the output result of the previous conversion layer are concatenated to obtain the third concatenation result. The first local filtering cache value of the conversion layer corresponding to the inference request in the first concatenation sequence is updated to the output result of the previous conversion layer.

[0028] Based on the first convolutional layer, a convolution operation is performed on the third concatenation result to obtain a third convolution result. The second local filter cache value of the transformation layer corresponding to the inference request is obtained from the second concatenation sequence. The second local filter cache value of the transformation layer and the third convolution result are concatenated to obtain a fourth concatenation result. A convolution operation is performed on the fourth concatenation result to obtain a fourth convolution result. The fourth convolution result is input into the attention layer for forward propagation calculation. The second local filter cache value of the transformation layer corresponding to the inference request in the second concatenation sequence is updated to the third convolution result.

[0029] The local filtering cache space includes a first local filtering cache space and a second local filtering cache space. The concatenation sequence includes a first concatenation sequence and a second concatenation sequence. The first local filtering cache value in the first local filtering cache space of each inference request is concatenated to obtain the first concatenation sequence. The second local filtering cache value in the second local filtering cache space of each inference request is concatenated to obtain the second concatenation sequence.

[0030] The model inference method provided in this embodiment dynamically updates the local filter cache value corresponding to each inference request in the concatenation sequence during each inference process. After each inference, the local filter cache value in the local filter cache space corresponding to each inference request is updated based on the updated concatenation sequence, ensuring that the concatenation sequence used in the next inference is concatenated with the updated local filter cache value, thereby ensuring the accuracy of model inference.

[0031] In an optional implementation, the method further includes:

[0032] When performing a second and subsequent inference on any inference request using a large language model, after passing through the local filtering attention layer in the first conversion layer, the first local filtering cache value of the first conversion layer corresponding to the inference request is obtained from the first concatenation sequence. The first local filtering cache value of the first conversion layer and the vector corresponding to the inference request are concatenated to obtain the fifth concatenation result. The first local filtering cache value of the first conversion layer corresponding to the inference request in the first concatenation sequence is updated to the vector corresponding to the inference request.

[0033] A convolution operation is performed on the fifth concatenation result based on the first convolutional layer to obtain a fifth convolution result. The second local filter cache value of the first transformation layer corresponding to the inference request is obtained from the second concatenation sequence. The second local filter cache value of the first transformation layer and the fifth convolution result are concatenated to obtain a sixth concatenation result. A convolution operation is performed on the sixth concatenation result to obtain a sixth convolution result. The sixth convolution result is input into the attention layer for forward propagation calculation. The second local filter cache value of the first transformation layer corresponding to the inference request in the second concatenation sequence is updated to the fifth convolution result.

[0034] The model inference method provided in this embodiment dynamically updates the local filter cache value corresponding to each inference request in the concatenation sequence during each inference process. After each inference, the local filter cache value in the local filter cache space corresponding to each inference request is updated based on the updated concatenation sequence, ensuring that the concatenation sequence used in the next inference is concatenated with the updated local filter cache value, thereby ensuring the accuracy of model inference.

[0035] In an optional implementation, the method further includes:

[0036] For any inference request, when the inference request meets the preset inference termination condition, the local filter cache space corresponding to the inference request is released;

[0037] The steps include: obtaining a new inference request, allocating local filter cache space for the new inference request, returning to the steps of obtaining the local filter cache values ​​in the local filter cache space of each inference request, concatenating the local filter cache values, and obtaining a concatenated sequence.

[0038] The model inference method provided in this embodiment releases the local filter cache space corresponding to the inference request when the inference of any inference request ends, that is, releases the local filter cache value corresponding to the inference request. If a new inference request is available at this time, the new inference request is obtained and processed simultaneously with the inference request that is currently inference. Local filter cache space is allocated for the new inference request. During the inference process of the new inference request, it will not obtain the local filter cache value that has not been cleared by other inference requests, thus ensuring the correct inference of all inference requests and the accuracy of all inference results.

[0039] In an optional implementation, the method further includes:

[0040] During the reasoning process using a large language model, the current number of inferences for each inference request is obtained;

[0041] Based on the current number of inferences for each inference request, multiple inference requests are classified to distinguish between first-time inference requests and non-first-time inference requests;

[0042] Based on the large language model, each initial inference request is responded to in a serial manner, and each non-initial inference request is responded to in a parallel manner.

[0043] The model reasoning method provided in this embodiment ensures the accuracy and efficiency of the model reasoning process by using different reasoning methods for reasoning requests at different reasoning stages.

[0044] In a second aspect, the present invention provides a model reasoning apparatus, the apparatus comprising:

[0045] A receiving module is used to receive multiple reasoning requests input by the user, wherein the reasoning requests include prompt words;

[0046] An allocation module is used to allocate local filter cache space for each of the inference requests. The local filter cache space is used to store local filter cache values ​​and is associated with the inference request.

[0047] The first acquisition module is used to acquire the local filter cache value in the local filter cache space of each inference request, and concatenate the local filter cache value to obtain a concatenation sequence.

[0048] The reasoning module is used to perform multiple inferences based on the prompts for each reasoning request and the concatenated sequence using a large language model, in order to obtain the reasoning results for each reasoning request.

[0049] Specifically, after each inference is completed, the local filter cache value stored in the local filter cache space of each inference request is updated so that the next inference can be performed based on the updated local filter cache value.

[0050] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the model reasoning method of the first aspect or any corresponding embodiment described above.

[0051] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the model reasoning method of the first aspect or any corresponding embodiment thereof.

[0052] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the model reasoning method of the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of the present invention, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the structure of the Source 2.0 large language model according to an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the structure of the local filtering attention layer and the self-attention layer according to an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the calculation of the local filtering attention layer according to an embodiment of the present invention;

[0057] Figure 4This is a flowchart illustrating the model reasoning method according to an embodiment of the present invention;

[0058] Figure 5 This is a flowchart illustrating another model reasoning method according to an embodiment of the present invention;

[0059] Figure 6 This is a flowchart illustrating another model reasoning method according to an embodiment of the present invention;

[0060] Figure 7 This is a structural block diagram of a model reasoning device according to an embodiment of the present invention;

[0061] Figure 8 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] The Source 2.0 large language model employs a novel model structure with local filtering attention layers, enabling it to perform text generation, knowledge-based question answering, mathematical reasoning, programming, and complex logic comprehension. This type of model can perform various natural language processing tasks and is designed to simulate human language understanding and generation capabilities.

[0064] The Source 2.0 large language model is built on the classic transformer structure, and innovatively extended on the classic transformer structure by adding a Local Filtered Attention (LFA) layer. Figure 1 This is a schematic diagram of the structure of the Source 2.0 large language model according to an embodiment of the present invention. Figure 1 As shown, the Source 2.0 large language model includes a first embedding layer, N transformer blocks, a first normalization layer (RMSNorm layer), and a second embedding layer.

[0065] Each transformer block includes a second normalization layer, a local filtering attention layer, a self-attention layer (not shown in the figure), a third normalization layer, and a feed-forward network (FFW) layer.

[0066] Figure 2 This is a schematic diagram of the structure of the local filtering attention layer and the self-attention layer according to an embodiment of the present invention. Figure 2 As shown, the local filtering attention layer consists of two convolutional layers that have undergone padding. The local filtering attention layer includes a first convolutional layer, a second convolutional layer, and a fourth normalization layer connected in sequence. Both the first and second convolutional layers are two-dimensional convolutional layers (Conv2d). The first convolutional layer has an input tensor size (indim) of H, an output tensor size (outdim) of H / 2, padding of 1, a stride of 1, and a kernel size of (2, 1). The second convolutional layer has an input tensor size (indim) of H / 2, an output tensor size (outdim) of H, padding of 1, a stride of 1, and a kernel size of (2, 1).

[0067] The self-attention layer takes the labels input to the source 2.0 large language model as linearized values ​​(Linear: V), obtains the linearized keywords (Linear: K) and linearized query statements (Linear: Q) output from the fourth normalization layer, concatenates the obtained linearized keywords and query statements with the previously cached linearized keywords and query statements to obtain the target linearized keywords and target linearized query statements, and inputs the linearized values, target linearized keywords, and target linearized query statements into the activation function layer (Softmax layer), using the activation function Softmax(QK). T / d s V processes the linearized value, the target linearization keyword, and the target linearization query statement to obtain the linearized output (Linear:Output) and output it.

[0068] Users input inference requests into the Source 2.0 Large Language Model, and the Source 2.0 Large Language Model's inference mechanism based on these requests follows a clear and defined process. Specifically, during the first inference, the Source 2.0 Large Language Model retrieves the prompt words from the user's input inference request and then begins forward propagation based on these prompt words. Through a series of inference processes across the various layers of the model structure, the model generates a new token. A crucial caching step exists throughout the inference process. Specifically, during the intermediate stages of model inference, the key-value pairs and lf_cache values ​​generated by each transformer block are cached. When handling complex inference tasks, these caching operations ensure timely data storage and reuse, improving the efficiency and accuracy of inference.

[0069] In the second and subsequent inference processes, the newly generated token from the previous inference is input into the model. Then, the newly generated key-value pair is concatenated with the cached key-value pair. After concatenation, the self-attention layer performs its computation. It's important to note that the concatenated key and value pairs are cached promptly for use by the self-attention layer when retrieving information from previous context during the next inference. This caching mechanism significantly improves the coherence and accuracy of inference. For example, in handling a multi-turn dialogue inference task, concatenating and caching the key-value pair generated in the previous round with the newly generated key-value pair allows the next round of inference to fully utilize previous information and make more accurate and logical judgments.

[0070] Simultaneously, the local filtering attention layer is calculated based on the lf1_cache and lf2_cache values ​​from the previous inference cache, and the cache is updated using the newly generated lf1_cache and lf2_cache values ​​for use in the next inference. This achieves effective filtering and processing of local information, further optimizing the inference effect and quality. This carefully designed process and data processing method provides a strong guarantee for the model's efficient and accurate inference.

[0071] For example, if the user inputs the prompt "explore the mysteries of the universe", the Source 2.0 model will process it according to the above process, generate a token and cache the key-value value and lf_cache value of each layer. Through concatenation and inference, the amount of computation is effectively reduced, thereby quickly generating inference results, improving the model's inference performance, and ensuring that the model runs more efficiently and stably.

[0072] It should be noted that each local filtering attention layer caches lf1_cache and lf2_cache values, i.e., lf_cache includes lf1_cache and lf2_cache values, which are used for the calculation of the first convolutional layer and the second convolutional layer, respectively.

[0073] Figure 3 This is a schematic diagram illustrating the calculation of a local filtering attention layer according to an embodiment of the present invention. Figure 3As shown, "1" represents the input data of the local filtering attention layer. Conv2d_1 and Conv2d_2 are two convolutional layers. The first convolutional layer of the local filtering attention layer receives the input data, concatenates it with the cached lf1_cache value (prepending the cached lf1_cache value before the input data), performs convolution on the concatenated data, and obtains the first convolution output. This first convolution output is then input into the second convolutional layer. The second convolutional layer concatenates the first convolution output with the cached lf2_cache value (prepending the cached lf2_cache value before the first convolution output), performs convolution on the concatenated data, and obtains the second convolution output. This second convolution output is then output so that the fourth normalization layer can normalize it.

[0074] It should be noted that after concatenating the input data with the cached lf1_cache value, the lf1_cache value needs to be updated using the input data. That is, the input data is copied to the cache area of ​​the lf1_cache value, and the previously cached lf1_cache value is updated with the input data.

[0075] After concatenating the first convolution output with the cached lf2_cache value, the lf2_cache value needs to be updated using the first convolution output. This means copying the first convolution output to the cache of the lf2_cache value and updating the previously cached lf2_cache value with the first convolution output.

[0076] In related technologies, the cache space for lf_cache values ​​is allocated using a fixed allocation method. Specifically, during the service initialization phase, a batch of lf_cache cache space is pre-allocated. The size of this batch is consistent with the maximum number of requests that the inference service can handle, that is, the number of inference requests that the Source 2.0 large language model can process simultaneously. This approach ensures the orderly storage and timely updating of data during inference, providing an accurate and reliable data foundation for subsequent inference steps.

[0077] For any inference request, an lf_cacheblock is allocated from the lf_cache cache space in the order of use to store the lf_cache value of the inference request. In each subsequent inference operation, the lf_cache value of the lf_cache block is read for inference and the lf_cache value in the lf_cache block is updated.

[0078] However, in high-concurrency scenarios, since each inference will overwrite the previous lf_cache value, this method of partially filtering cached values ​​may fail to clear the lf_cache value of the inference request cache in time after the inference request is processed. This will cause other inference requests to use the lf_cache value of the inference request cache, resulting in inaccurate inference results.

[0079] For example, when processing multiple similar but slightly different requests in succession, residual information from the previous inference request may interfere with the inference of the current inference request, resulting in inaccurate inference results.

[0080] Furthermore, this method of partially filtering cached values ​​cannot effectively support scenarios with high concurrency inference requests. When a new inference request is inserted before an existing one has finished, the lack of a scheme for reallocating the lf_cache cache means that the lf_cache value corresponding to the new inference request may not be accurately obtained, leading to the use of an incorrect lf_cache value and consequently, incorrect inference results.

[0081] This inability to effectively support high-concurrency inference requests severely impacts the service quality and efficiency of the Source 2.0 large language model in complex and ever-changing real-world business environments. It also significantly affects the effective application and performance of the Source 2.0 large language model, and may cause system performance degradation, corresponding latency, or even service terminal issues.

[0082] This invention provides a model inference method. It receives multiple inference requests from a user, allocates a local filtering cache space for each request, obtains the local filtering cache values ​​in each request's local filtering cache space, concatenates these values ​​to obtain a concatenated sequence, and uses a large language model to perform multiple inferences based on the prompts and concatenated sequences of each request to obtain the inference result for each request. After each inference, the local filtering cache values ​​stored in the local filtering cache space of each request are updated, and the next inference is performed based on the updated local filtering cache values. This ensures that the large language model can accurately infer all inference requests in high-concurrency scenarios, guaranteeing the accuracy of the inference results.

[0083] According to an embodiment of the present invention, a model reasoning method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0084] This embodiment provides a model inference method that can be used in mobile terminals, such as servers, central processing units, etc. Figure 4 This is a flowchart of a model reasoning method according to an embodiment of the present invention, such as... Figure 4 As shown, the process includes the following steps:

[0085] Step S401: Receive multiple reasoning requests input by the user, the reasoning requests including prompt words.

[0086] It should be noted that this embodiment is aimed at model inference in a high-concurrency scenario. When a large number of inference requests are received simultaneously, the number of inference requests corresponding to the maximum number of inference requests that the large language model can process at the same time is selected from the large number of inference requests in the order of receipt, and the selected inference requests are used as the current inference requests to be processed.

[0087] The system receives multiple inference requests from the user to obtain the current inference request to be processed.

[0088] Step S402: Allocate local filter cache space for each inference request. The local filter cache space is used to store local filter cache values ​​and is associated with the inference request.

[0089] For each inference request received, a unique local filter cache space is first created specifically for that inference request. This unique local filter cache space is used to cache the local filter cache values ​​generated during the inference process of that inference request.

[0090] It is understandable that the local filter cache values ​​are the aforementioned lf1_cache and lf2_cache values.

[0091] It should be noted that the size of this local filter cache space does not depend on the length of the prompt in the inference request, but is a fixed size. This fixed-size local filter cache space is composed of tensors with all zero values, that is, the local filter cache values ​​in this local filter cache space are initially tensors with all zero values.

[0092] In other words, regardless of whether the input prompts are short sentences or lengthy text paragraphs, the size of the local filter cache space will not change. It will always maintain a fixed size and exist in the form of a zero-value tensor, providing stable and independent cache support for the subsequent inference process, thus ensuring the efficiency and accuracy of the inference process.

[0093] Step S403: Obtain the local filter cache value in the local filter cache space of each inference request, and concatenate the local filter cache values ​​to obtain the concatenated sequence.

[0094] Specifically, after allocating local filter cache space for each inference request, local filter cache values ​​are obtained from the local filter cache space of each inference request, and the obtained multiple local filter cache values ​​are concatenated according to a preset order rule to obtain a concatenated sequence.

[0095] Understandably, the preset ordering rule could be the order in which inference requests are received. Each inference request is associated with its local filtered cache value in the concatenated sequence.

[0096] Step S404: Use the large language model to perform multiple inferences based on the prompts and concatenation sequences of each inference request to obtain the inference results of each inference request.

[0097] Specifically, after each inference, the local filter cache value stored in the local filter cache space of each inference request is updated so that the next inference can be performed based on the updated local filter cache value.

[0098] After obtaining the concatenated sequence, the prompt words for each reasoning request are input into the large language model, so that the large language model can perform multiple inferences based on the prompt words and concatenated sequence for each reasoning request, and obtain the reasoning results for each reasoning request.

[0099] It should be noted that the large language model in this embodiment is the source 2.0 large language model.

[0100] It should be further explained that after each inference request completes one inference, the end of one inference is determined, and the local filter cache value stored in the local filter cache space of each inference request is updated so that the next inference of each inference request is based on the updated local filter cache value.

[0101] The model inference method provided in this embodiment receives multiple inference requests input by the user, allocates a local filtering cache space for each inference request, obtains the local filtering cache values ​​in the local filtering cache space of each inference request, concatenates the local filtering cache values ​​to obtain a concatenated sequence, and uses a large language model to perform multiple inferences based on the prompt words of each inference request and the concatenated sequence to obtain the inference result of each inference request. After each inference, the local filtering cache values ​​stored in the local filtering cache space of each inference request are updated so that the next inference is based on the updated local filtering cache values. By allocating a unique local filtering cache space for each inference request to store the local filtering cache values ​​updated by the model during the inference process, the large language model will not use incorrect local filtering cache values ​​in high-concurrency scenarios, and can perform efficient and accurate inference for all inference requests, ensuring the accuracy of the inference results.

[0102] This embodiment provides a model inference method that can be used in mobile terminals, such as servers, central processing units, etc. Figure 5 This is a flowchart of a model reasoning method according to an embodiment of the present invention, such as... Figure 5 As shown, the process includes the following steps:

[0103] Step S501: Receive multiple inference requests input by the user. Each inference request includes prompt words. See details below. Figure 4 Step S401 of the illustrated embodiment will not be described again here.

[0104] Step S502: Allocate local filter cache space for each inference request. The local filter cache space is used to store local filter cache values ​​and is associated with each inference request. For details, please refer to [link to details]. Figure 4 Step S402 of the illustrated embodiment will not be described again here.

[0105] Step S503: Obtain the local filter cache values ​​in the local filter cache space of each inference request, and concatenate the local filter cache values ​​to obtain a concatenated sequence. For details, please refer to [link to relevant documentation]. Figure 4 Step S403 of the illustrated embodiment will not be described again here.

[0106] Step S504: Use the large language model to perform multiple inferences based on the prompts and concatenation sequences of each inference request to obtain the inference results of each inference request;

[0107] Specifically, after each inference, the local filter cache value stored in the local filter cache space of each inference request is updated so that the next inference can be performed based on the updated local filter cache value.

[0108] Specifically, step S504 includes:

[0109] Step S5041: When performing the first reasoning on any reasoning request using the large language model, the prompt words of the reasoning request are converted into a vector sequence, and the vector sequence is input into the local filtering attention layer so that the local filtering attention layer performs forward propagation calculation based on the vector sequence and the concatenated sequence.

[0110] Understandably, the prompts for the reasoning request are converted into a vector sequence through the first embedding layer in the large language model. The vector sequence is then input into the local filtering attention layer for forward propagation computation. Essentially, the vector sequence is input into the first transformer block, normalized, and then input into the local filtering attention layer of the first transformer block for forward propagation computation.

[0111] In step S5042, when performing a second inference and subsequent inferences on any inference request using the large language model, the token generated by the previous inference of the inference request is converted into a vector, and the vector is input into the local filtering attention layer so that the local filtering attention layer performs forward propagation calculation based on the vector and the updated concatenated sequence.

[0112] Specifically, when the large language model performs a second inference and subsequent inferences for any inference request, the tags generated by the previous inference are converted into vectors, the vectors are input into the first transformer block, the vectors are normalized, and the normalized vectors are input into the local filtering attention layer of the first transformer block for forward propagation computation.

[0113] Step S5043: For any reasoning request, when the reasoning request meets the preset reasoning termination condition, obtain the tag generated by each reasoning of the reasoning request.

[0114] Specifically, during the reasoning process of any reasoning request in the large language model, if an end marker is generated, it is determined that the reasoning request meets the preset reasoning end condition.

[0115] It should be noted that the end marker can be a period or any marker that represents the end, which is set by the technical staff and is not specifically restricted here.

[0116] Step S5044: Determine the reasoning result of the reasoning request based on the tags generated for each reasoning step.

[0117] Specifically, the order in which the tags generated by each inference are arranged is determined according to the execution order of the inference request. The earlier the inference is executed, the earlier the tags are arranged.

[0118] The tags generated in each inference are concatenated according to their order to obtain the inference result corresponding to the inference request.

[0119] The model inference method provided in this embodiment converts the prompt words into a vector sequence during the first inference, and inputs the vector sequence into a local filtering attention layer for forward propagation calculation. During the second and subsequent inferences, the markers generated in the previous inference are introduced, converted into vectors, and input into the local filtering attention layer for forward propagation calculation. By gradually accumulating and updating contextual information through multiple inferences, a more accurate and efficient inference process is achieved.

[0120] In some alternative implementations, the model inference method further includes:

[0121] Step a1: Update the spliced ​​sequence during each inference process.

[0122] Specifically, during the reasoning process for each reasoning request, the spliced ​​sequence is updated after passing through a local filtering attention layer.

[0123] It should be noted that updating the concatenated sequence means updating the local filter cache value corresponding to each inference request in the concatenated sequence.

[0124] After each inference iteration, the local filter cache values ​​stored in the local filter cache space for each inference request are updated so that the next inference iteration is based on the updated local filter cache values. This includes:

[0125] Step b1: After each inference, obtain the tag generated by this inference and the concatenated sequence updated by this inference for each inference request. Based on the concatenated sequence updated by this inference, update the local filter cache value stored in the local filter cache space of each inference request to obtain the updated local filter cache value in the local filter cache space of each inference request. Concatenate the updated local filter cache value to obtain the updated concatenated sequence. Use the large language model to perform the next inference based on the updated concatenated sequence and the tag generated by this inference.

[0126] In other words, before each inference operation, the following steps are performed: retrieving the local filter cache value from the local filter cache space of each inference request, concatenating the local filter cache values, and obtaining the concatenated sequence. This ensures that each retrieved local filter cache value is the one used by the currently being processed inference request, avoiding the use of the local filter cache value of an inference request that has already finished inference.

[0127] After each inference iteration, the concatenated sequence needs to be allocated. The local filter cache values ​​of each inference request in the concatenated sequence obtained from this inference are accurately distributed to the local filter cache space corresponding to each inference request, ensuring that resources can be precisely matched to meet the needs of each inference request.

[0128] The model inference method provided in this embodiment updates the concatenated sequence during each inference process. After each inference, the local filter cache value stored in the local filter cache space of each inference request is updated based on the concatenated sequence updated in this inference. The next inference is then performed based on the updated concatenated sequence obtained by concatenating the updated local filter cache value. This ensures that the local filter cache value corresponding to the inference request is accurately obtained. The next inference is then performed based on the obtained local filter cache value, thus guaranteeing the accuracy of the inference.

[0129] In some optional implementations, the large language model includes multiple transformation layers, each of which includes a local filtering attention layer, and step a1 above includes:

[0130] Step a11: After performing the first inference on any inference request using the large language model and passing through the local filtering attention layer in the first conversion layer, obtain the first local filtering cache value of the first conversion layer corresponding to the inference request from the first concatenation sequence, concatenate the first local filtering cache value of the first conversion layer and the vector sequence corresponding to the inference request to obtain the first concatenation result, and update the first local filtering cache value of the first conversion layer corresponding to the inference request in the first concatenation sequence to the vector sequence corresponding to the inference request.

[0131] The transformation layer corresponds to the aforementioned transformer block. The first local filter cache space corresponding to the inference request caches multiple first local filter cache values ​​of the transformation layer. The second local filter cache space corresponding to the inference request caches multiple second local filter cache values ​​of the transformation layer.

[0132] Step a12: Perform a convolution operation on the first concatenation result based on the first convolutional layer to obtain the first convolution result; obtain the second local filter cache value of the first transformation layer corresponding to the inference request from the second concatenation sequence; concatenate the second local filter cache value of the first transformation layer and the first convolution result to obtain the second concatenation result; perform a convolution operation on the second concatenation result to obtain the second convolution result; input the second convolution result into the self-attention layer for forward propagation calculation; update the second local filter cache value of the first transformation layer corresponding to the inference request in the second concatenation sequence to the first convolution result.

[0133] Step a13: After performing the first inference on the inference request using the large language model, and passing through the local filtering attention layer of any of the other conversion layers besides the first conversion layer, obtain the first local filtering cache value of the conversion layer corresponding to the inference request from the first concatenation sequence, concatenate the first local filtering cache value of the conversion layer with the output result of the previous conversion layer to obtain the third concatenation result, and update the first local filtering cache value of the conversion layer corresponding to the inference request in the first concatenation sequence to the output result of the previous conversion layer.

[0134] Step a14: Perform a convolution operation on the third concatenation result based on the first convolutional layer to obtain the third convolution result; obtain the second local filter cache value of the transformation layer corresponding to the inference request from the second concatenation sequence; concatenate the second local filter cache value of the transformation layer and the third convolution result to obtain the fourth concatenation result; perform a convolution operation on the fourth concatenation result to obtain the fourth convolution result; input the fourth convolution result into the self-attention layer for forward propagation calculation; update the second local filter cache value of the transformation layer corresponding to the inference request in the second concatenation sequence to the third convolution result.

[0135] The local filtering cache space includes a first local filtering cache space and a second local filtering cache space. The concatenation sequence includes a first concatenation sequence and a second concatenation sequence. The first local filtering cache value in the first local filtering cache space of each inference request is concatenated to obtain the first concatenation sequence. The second local filtering cache value in the second local filtering cache space of each inference request is concatenated to obtain the second concatenation sequence.

[0136] Understandably, the first local filter cache values, which are independent of each inference request, are concatenated to obtain a first concatenation sequence, and the second local filter cache values, which are independent of each inference request, are concatenated to obtain a second concatenation sequence, thereby forming unified lf1_caches and lf2_caches for use in the current inference process.

[0137] It should be noted that the first local filter cache value can be the lf1_cache value, and the second local filter cache value can be the lf2_cache value.

[0138] The model inference method provided in this embodiment dynamically updates the local filter cache value corresponding to each inference request in the concatenation sequence during each inference process. After each inference, the local filter cache value in the local filter cache space corresponding to each inference request is updated based on the updated concatenation sequence, ensuring that the concatenation sequence used in the next inference is concatenated with the updated local filter cache value, thereby ensuring the accuracy of model inference.

[0139] In some alternative implementations, the model inference method further includes:

[0140] Step c1: After performing a second and subsequent inference on any inference request using the large language model, and after passing through the local filtering attention layer in the first conversion layer, obtain the first local filtering cache value of the first conversion layer corresponding to the inference request from the first concatenation sequence, concatenate the first local filtering cache value of the first conversion layer and the vector corresponding to the inference request to obtain the fifth concatenation result, and update the first local filtering cache value of the first conversion layer corresponding to the inference request in the first concatenation sequence to the vector corresponding to the inference request.

[0141] Step c2: Perform a convolution operation on the fifth concatenation result based on the first convolutional layer to obtain the fifth convolution result; obtain the second local filter cache value of the first transformation layer corresponding to the inference request from the second concatenation sequence; concatenate the second local filter cache value of the first transformation layer and the fifth convolution result to obtain the sixth concatenation result; perform a convolution operation on the sixth concatenation result to obtain the sixth convolution result; input the sixth convolution result into the self-attention layer for forward propagation calculation; update the second local filter cache value of the first transformation layer corresponding to the inference request in the second concatenation sequence to the fifth convolution result.

[0142] It is understandable that the steps performed when using the large language model to perform a second inference on any inference request and subsequent inferences, after passing through the local filtering attention layer of any of the other conversion layers besides the first conversion layer, are the same as the steps performed when using the large language model to perform the first inference on the inference request, after passing through the local filtering attention layer of any of the other conversion layers besides the first conversion layer. For details, please refer to the aforementioned steps a13 and a14, which will not be repeated here.

[0143] The model inference method provided in this embodiment dynamically updates the local filter cache value corresponding to each inference request in the concatenation sequence during each inference process. After each inference, the local filter cache value in the local filter cache space corresponding to each inference request is updated based on the updated concatenation sequence, ensuring that the concatenation sequence used in the next inference is concatenated with the updated local filter cache value, thereby ensuring the accuracy of model inference.

[0144] In some alternative implementations, the model inference method further includes:

[0145] Step d1: For any inference request, when the inference request meets the preset inference termination condition, release the local filter cache space corresponding to the inference request.

[0146] Specifically, when an inference request meets the preset inference termination condition, the inference request ends, and the local filter cache space corresponding to the inference request is released, that is, the local filter cache value corresponding to the inference request is released.

[0147] Step d2 involves obtaining new inference requests, allocating local filter cache space for the new inference requests, returning to the step of obtaining the local filter cache values ​​in the local filter cache space of each inference request, concatenating the local filter cache values, and obtaining the concatenated sequence.

[0148] Understandably, when a large number of inference requests have not been processed, new inference requests are obtained from the large number of inference requests received at the same time in the order of receipt, and the inference of the new inference requests is executed until the inference of all inference requests is completed.

[0149] The model inference method provided in this embodiment releases the local filter cache space corresponding to the inference request when the inference of any inference request ends, that is, releases the local filter cache value corresponding to the inference request. If a new inference request is available at this time, the new inference request is obtained and processed simultaneously with the inference request that is currently inference. Local filter cache space is allocated for the new inference request. During the inference process of the new inference request, it will not obtain the local filter cache value that has not been cleared by other inference requests, thus ensuring the correct inference of all inference requests and the accuracy of all inference results.

[0150] In some alternative implementations, the model inference method further includes:

[0151] Step e1: During the reasoning process using the large language model, obtain the current number of inferences for each inference request.

[0152] Step e2: Classify multiple inference requests based on the current number of inferences for each inference request to distinguish between first-time inference requests and non-first-time inference requests.

[0153] Specifically, if the current inference request is the first inference request, then the inference request is determined to be the first inference request.

[0154] If the current inference attempt is not the first one, the inference request is determined to be a non-first inference request.

[0155] Step e3: Based on the large language model, respond to each initial inference request in a serial manner and respond to each non-initial inference request in a parallel manner.

[0156] It's important to note that the entire process of inputting the prompts for inference requests into the large language model for inference is clearly divided into a prefill stage and a decoding stage. The prefill stage refers to the stage when an inference request is first input into the large language model for inference. It's also worth noting that within a batch, the number of tokens corresponding to the prompts for each inference request may vary. When these tokens are input to the local filtering attention layer, due to the different input lengths, the batch concatenation method cannot be directly used for inference. Therefore, this embodiment uses a method of processing each inference request separately, i.e., a loop approach. In other words, for the first inference request, inference is performed serially in the local filtering attention layer; that is, based on the large language model, each initial inference request is responded to serially.

[0157] During this process, the lf1_cache and lf2_cache values ​​generated during the inference process of each inference request will also be used to update the lf1_cache and lf2_cache values ​​in the splicing sequence.

[0158] The decoding phase refers to the stage in which the token generated from the previous inference is input into the large language model for reasoning during the second and subsequent inference processes. Since each inference request inputs only one token into the model at this time, the tokens generated from the previous inference for all inference requests can be input into the local filtering attention layer. In other words, for non-first inference requests, inference is performed in parallel at the local filtering attention layer, i.e., based on the large language model, each non-first inference request is responded to in parallel.

[0159] Subsequently, the lf1_cache and lf2_cache values ​​generated after the local filtering attention layer are used to update the lf1_cache and lf2_cache values ​​in the spliced ​​sequence.

[0160] This design ensures the accuracy and efficiency of the model inference process, allowing data to be properly processed and optimized at different stages, thus providing strong support and guarantee for the final output. Both the complex processing in the Prefill stage and the concise operation in the Decoding stage are indispensable components of the entire model inference system.

[0161] The model reasoning method provided in this embodiment ensures the accuracy and efficiency of the model reasoning process by using different reasoning methods for reasoning requests at different reasoning stages.

[0162] In some optional implementations, step S5041 above includes:

[0163] Step f1 involves segmenting the prompts for the reasoning request to obtain multiple tokens.

[0164] In this process, the prompts for reasoning requests are divided into basic units such as words, subwords, or characters according to certain rules. These basic units are called tokens.

[0165] Step f2: For any given tag, map the tag to an integer index based on a preset vocabulary to obtain an integer index sequence.

[0166] The pre-defined vocabulary contains all the tokens and their corresponding integer indices recognized by the Source 2.0 language model. The integer index is a numerical representation that the Source 2.0 language model can understand and process.

[0167] Each tag is mapped to a unique integer index, and multiple integer indices obtained from multiple tag mappings form a sequence of integer indices.

[0168] Step f3 involves inputting the integer index sequence into the embedding layer of the large language model to obtain a vector sequence.

[0169] After obtaining the integer index sequence, the integer index sequence is input into the embedding layer of the large language model. Here, the embedding layer is the first embedding layer mentioned above, to obtain the vector sequence.

[0170] It should be noted that during the second and subsequent inference processes, based on the pre-defined vocabulary, the tags generated in the previous inference are mapped to integer indices, and these integer indices are input into the first embedding layer of the large language model to obtain vectors.

[0171] The model inference method provided in this embodiment obtains multiple tags by segmenting the prompt words, which can capture semantic information in the text in a more granular way. Based on a preset vocabulary, the tags are mapped to integer indices to obtain an integer index sequence. The integer index sequence is then input into the embedding layer of a large language model to obtain a vector sequence, which significantly improves the accuracy of the semantic representation of the prompt words.

[0172] This embodiment provides a model inference method that can be used in mobile terminals, such as servers, central processing units, etc. Figure 6 This is a flowchart of a model reasoning method according to an embodiment of the present invention, such as... Figure 6 As shown, the process includes the following steps:

[0173] Step 1: Schedule the received requests and obtain the requests that are currently ready for inference.

[0174] In this embodiment, the number of inference requests received simultaneously exceeds the maximum number of inference requests that a large language model can process simultaneously. Received requests refer to inference requests received simultaneously.

[0175] Retrieve the request for the current accurate inference from the received requests, that is, retrieve the current inference request to be processed.

[0176] Step 2: Allocate lf_cache cache space to each request, Lf1_cache(1, h, 1, 1) and Lf2_cache(1, h / 2, 1, 1).

[0177] It is understandable that allocating an lf_cache cache space for each inference request means allocating a local filter cache space for each inference request. This local filter cache space includes a first local filter cache space Lf1_cache(1, h, 1, 1) and a second local filter cache space Lf2_cache(1, h / 2, 1, 1).

[0178] Step 3: Concatenate the lf_caches of the inference request to obtain lf1_caches(b, h, 1, 1) and lf2_caches(b, h / 2, 1, 1).

[0179] The local filter cache values ​​in the local filter cache space of the inference request are concatenated to obtain the concatenated sequence.

[0180] The concatenation sequence includes a first concatenation sequence lf1_caches(b, h, 1, 1) and a second concatenation sequence lf2_caches(b, h / 2, 1, 1), where b is the number of inference requests currently being processed.

[0181] Step four: Input the request into the inference model.

[0182] This involves inputting the request into the inference model, specifically by inputting the inference request into the Source 2.0 large language model.

[0183] Step 5: Obtain the token for one inference attempt, as well as lf1_caches and lf2_caches for each request.

[0184] The large language model performs inference based on the concatenated sequence and the prompts for the inference request. It obtains the token generated for each inference attempt and the local filter cache values ​​lf1_caches and lf2_caches corresponding to each inference request.

[0185] This corresponds to updating the spliced ​​sequence in each inference process, as mentioned above.

[0186] Step 6: Update lf1_cache and lf2_cache for each input request with the returned lf1_caches and lf2_caches, and return to execute Step 1 until the inference results of the received requests are complete.

[0187] This corresponds to updating the local filter cache values ​​stored in the local filter cache space of each inference request after each inference is completed.

[0188] Understandably, if the current inference request has not changed compared to the previous inference, then step two is skipped and the steps following step two are executed directly.

[0189] If the current requests being inferred include newly added requests, then step two is modified to allocate lf_cache cache space to the newly added requests, and then the subsequent steps are executed.

[0190] If the current request to be inferred does not include the newly added request, and all the current requests to be inferred have corresponding lf_cache cache space, then skip step two and directly execute the steps after step two.

[0191] The model inference method provided in this embodiment, through its ingenious resource allocation strategy and efficient data processing mechanism, enables large language models to maintain stable performance under high concurrency pressure, respond quickly and accurately to each request, greatly improve the overall efficiency and service quality of the system, and open up a new path for the widespread application of large models in high-concurrency scenarios. It has important practical application value and broad development prospects.

[0192] This embodiment also provides a model inference apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0193] This embodiment provides a model inference device, such as Figure 7 As shown, it includes:

[0194] The receiving module 701 is used to receive multiple reasoning requests input by the user, and the reasoning requests include prompt words.

[0195] The allocation module 702 is used to allocate local filter cache space for each inference request. The local filter cache space is used to store local filter cache values ​​and is associated with the inference request.

[0196] The first acquisition module 703 is used to acquire the local filter cache value in the local filter cache space of each inference request, and concatenate the local filter cache value to obtain the concatenated sequence.

[0197] The reasoning module 704 is used to perform multiple inferences based on the prompts and concatenation sequences of each inference request using a large language model, and to obtain the reasoning results of each inference request.

[0198] Specifically, after each inference, the local filter cache value stored in the local filter cache space of each inference request is updated so that the next inference can be performed based on the updated local filter cache value.

[0199] In some alternative implementations, the inference module 704 includes:

[0200] The first reasoning unit is used to convert the prompt words of any reasoning request into a vector sequence when performing the first reasoning on any reasoning request using the large language model. The vector sequence is then input into the local filtering attention layer so that the local filtering attention layer can perform forward propagation calculation based on the vector sequence and the concatenated sequence.

[0201] The second inference unit is used to convert the tokens generated by the previous inference of any inference request into vectors when performing a second inference and subsequent inferences on any inference request using the large language model. The vectors are then input into the local filtering attention layer so that the local filtering attention layer can perform forward propagation computation based on the vectors and the updated concatenated sequence.

[0202] The first acquisition unit is used to acquire the marker generated by each inference of any inference request when the inference request meets the preset inference termination condition.

[0203] The first determining unit is used to determine the reasoning result of the reasoning request based on the tags generated in each reasoning.

[0204] In some alternative implementations, the model inference apparatus further includes:

[0205] The first update module is used to update the spliced ​​sequence during each inference process.

[0206] Inference module 704 includes:

[0207] The first update unit is used to obtain the tags generated by this inference and the concatenated sequence of this inference update for each inference request after each inference. Based on the concatenated sequence of this inference update, the local filter cache values ​​stored in the local filter cache space of each inference request are updated to obtain the updated local filter cache values ​​in the local filter cache space of each inference request. The updated local filter cache values ​​are concatenated to obtain the updated concatenated sequence. The large language model is used to perform the next inference based on the updated concatenated sequence and the tags generated by this inference.

[0208] In some alternative implementations, the first update module includes:

[0209] The second update unit is used to perform the first inference on any inference request using the large language model, and after passing through the local filtering attention layer in the first conversion layer, obtain the first local filtering cache value of the first conversion layer corresponding to the inference request from the first concatenation sequence, concatenate the first local filtering cache value of the first conversion layer and the vector sequence corresponding to the inference request to obtain the first concatenation result, and update the first local filtering cache value of the first conversion layer corresponding to the inference request in the first concatenation sequence to the vector sequence corresponding to the inference request.

[0210] The third update unit is used to perform a convolution operation on the first concatenation result based on the first convolutional layer to obtain the first convolution result, obtain the second local filter cache value of the first transformation layer corresponding to the inference request from the second concatenation sequence, concatenate the second local filter cache value of the first transformation layer and the first convolution result to obtain the second concatenation result, perform a convolution operation on the second concatenation result to obtain the second convolution result, input the second convolution result into the attention layer for forward propagation calculation, and update the second local filter cache value of the first transformation layer corresponding to the inference request in the second concatenation sequence to the first convolution result.

[0211] The fourth update unit is used to perform the first inference on the inference request using the large language model, and after passing through the local filtering attention layer of any of the other conversion layers besides the first conversion layer, obtain the first local filtering cache value of the conversion layer corresponding to the inference request from the first concatenation sequence, concatenate the first local filtering cache value of the conversion layer and the output result of the previous conversion layer to obtain the third concatenation result, and update the first local filtering cache value of the conversion layer corresponding to the inference request in the first concatenation sequence to the output result of the previous conversion layer.

[0212] The fifth update unit is used to perform a convolution operation on the third concatenation result based on the first convolutional layer to obtain the third convolution result, obtain the second local filter cache value of the transformation layer corresponding to the inference request from the second concatenation sequence, concatenate the second local filter cache value of the transformation layer and the third convolution result to obtain the fourth concatenation result, perform a convolution operation on the fourth concatenation result to obtain the fourth convolution result, input the fourth convolution result into the attention layer for forward propagation calculation, and update the second local filter cache value of the transformation layer corresponding to the inference request in the second concatenation sequence to the third convolution result.

[0213] The local filtering cache space includes a first local filtering cache space and a second local filtering cache space. The concatenation sequence includes a first concatenation sequence and a second concatenation sequence. The first local filtering cache value in the first local filtering cache space of each inference request is concatenated to obtain the first concatenation sequence. The second local filtering cache value in the second local filtering cache space of each inference request is concatenated to obtain the second concatenation sequence.

[0214] In some alternative implementations, the model inference apparatus further includes:

[0215] The second update module is used to perform a second inference and subsequent inferences on any inference request using the large language model. After passing through the local filtering attention layer in the first conversion layer, it obtains the first local filtering cache value of the first conversion layer corresponding to the inference request from the first concatenation sequence, concatenates the first local filtering cache value of the first conversion layer and the vector corresponding to the inference request to obtain the fifth concatenation result, and updates the first local filtering cache value of the first conversion layer corresponding to the inference request in the first concatenation sequence to the vector corresponding to the inference request.

[0216] The third update module is used to perform a convolution operation on the fifth concatenation result based on the first convolutional layer to obtain the fifth convolution result, obtain the second local filter cache value of the first transformation layer corresponding to the inference request from the second concatenation sequence, concatenate the second local filter cache value of the first transformation layer and the fifth convolution result to obtain the sixth concatenation result, perform a convolution operation on the sixth concatenation result to obtain the sixth convolution result, input the sixth convolution result into the attention layer for forward propagation calculation, and update the second local filter cache value of the first transformation layer corresponding to the inference request in the second concatenation sequence to the fifth convolution result.

[0217] In some alternative implementations, the model inference apparatus further includes:

[0218] The release module is used to release the local filter cache space corresponding to any inference request when the inference request meets the preset inference termination condition.

[0219] The return module is used to obtain new inference requests, allocate local filter cache space for new inference requests, and return to the steps of obtaining local filter cache values ​​in the local filter cache space of each inference request, concatenating the local filter cache values, and obtaining the concatenated sequence.

[0220] In some alternative implementations, the model inference apparatus further includes:

[0221] The second acquisition module is used to acquire the current number of inferences for each inference request during the inference process using a large language model.

[0222] The classification module is used to classify multiple inference requests based on the current number of inferences for each inference request, in order to distinguish between first-time inference requests and non-first-time inference requests.

[0223] The response module is used to respond to each initial inference request in a serial manner and to respond to each subsequent inference request in a parallel manner, based on a large language model.

[0224] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0225] In this embodiment, the model inference device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0226] This invention also provides a computer device having the above-described features. Figure 7 The model inference device shown.

[0227] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 8As shown, the computer device includes one or more processors 801, memory 802, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take the 801 processor as an example.

[0228] Processor 801 may be a central processing unit, a network processor, or a combination thereof. Processor 801 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0229] The memory 802 stores instructions executable by at least one processor 801 to cause at least one processor 801 to perform the method shown in the above embodiments.

[0230] The memory 802 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 802 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 802 may optionally include memory remotely located relative to the processor 801, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0231] The memory 802 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 802 may also include a combination of the above types of memory.

[0232] The computer device also includes a communication interface 803 for communicating with other devices or communication networks.

[0233] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0234] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0235] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A model reasoning method, characterized in that, The method includes: Receive multiple inference requests input by the user, the inference requests including prompt words; Allocate a local filter cache space for each of the inference requests, the local filter cache space being used to store local filter cache values, and the local filter cache space being associated with the inference request; Obtain the local filter cache value in the local filter cache space of each inference request, and concatenate the local filter cache values ​​to obtain a concatenation sequence; The large language model is used to perform multiple inferences based on the prompts for each inference request and the concatenated sequence to obtain the inference results for each inference request; Specifically, after each inference is completed, the local filter cache value stored in the local filter cache space of each inference request is updated so that the next inference is based on the updated local filter cache value. The method further includes: The spliced ​​sequence is updated during each inference process; The large language model includes multiple transformation layers, each of which includes a local filtering attention layer. The step of updating the concatenated sequence during each inference process includes: When performing the first inference on any inference request using a large language model, after passing through the local filtering attention layer in the first conversion layer, the first local filtering cache value of the first conversion layer corresponding to the inference request is obtained from the first concatenation sequence. The first local filtering cache value of the first conversion layer and the vector sequence corresponding to the inference request are concatenated to obtain the first concatenation result. The first local filtering cache value of the first conversion layer corresponding to the inference request in the first concatenation sequence is updated to the vector sequence corresponding to the inference request. Based on the first convolutional layer, a convolution operation is performed on the first concatenation result to obtain a first convolution result. The second local filter cache value of the first transformation layer corresponding to the inference request is obtained from the second concatenation sequence. The second local filter cache value of the first transformation layer and the first convolution result are concatenated to obtain a second concatenation result. A convolution operation is performed on the second concatenation result to obtain a second convolution result. The second convolution result is input into the attention layer for forward propagation calculation. The second local filter cache value of the first transformation layer corresponding to the inference request in the second concatenation sequence is updated to the first convolution result. When the inference request is inferred for the first time using a large language model, and after passing through the local filtering attention layer of any of the other conversion layers besides the first conversion layer, the first local filtering cache value of the conversion layer corresponding to the inference request is obtained from the first concatenation sequence. The first local filtering cache value of the conversion layer and the output result of the previous conversion layer are concatenated to obtain the third concatenation result. The first local filtering cache value of the conversion layer corresponding to the inference request in the first concatenation sequence is updated to the output result of the previous conversion layer. Based on the first convolutional layer, a convolution operation is performed on the third concatenation result to obtain a third convolution result. The second local filter cache value of the transformation layer corresponding to the inference request is obtained from the second concatenation sequence. The second local filter cache value of the transformation layer and the third convolution result are concatenated to obtain a fourth concatenation result. A convolution operation is performed on the fourth concatenation result to obtain a fourth convolution result. The fourth convolution result is input into the attention layer for forward propagation calculation. The second local filter cache value of the transformation layer corresponding to the inference request in the second concatenation sequence is updated to the third convolution result. The local filtering cache space includes a first local filtering cache space and a second local filtering cache space. The concatenation sequence includes a first concatenation sequence and a second concatenation sequence. The first local filtering cache value in the first local filtering cache space of each inference request is concatenated to obtain the first concatenation sequence. The second local filtering cache value in the second local filtering cache space of each inference request is concatenated to obtain the second concatenation sequence.

2. The method according to claim 1, characterized in that, The step of using a large language model to perform multiple inferences based on the prompts for each inference request and the concatenated sequence to obtain the inference results for each inference request includes: When performing the first reasoning on any reasoning request using a large language model, the prompt words of the reasoning request are converted into a vector sequence, and the vector sequence is input into a local filtering attention layer so that the local filtering attention layer performs forward propagation calculation based on the vector sequence and the concatenated sequence. When using a large language model to perform a second and subsequent inference for any inference request, the token generated by the previous inference of the inference request is converted into a vector, and the vector is input into a local filtering attention layer so that the local filtering attention layer performs forward propagation calculation based on the vector and the updated concatenated sequence. For any reasoning request, when the reasoning request meets the preset reasoning termination condition, obtain the tag generated for each reasoning of the reasoning request; Based on the tags generated in each inference, the inference result of the inference request is determined.

3. The method according to claim 2, characterized in that, After each inference iteration, the local filter cache value stored in the local filter cache space of each inference request is updated so that the next inference iteration is based on the updated local filter cache value. This includes: After each inference, the markers generated by this inference and the concatenated sequence of this inference update for each inference request are obtained. Based on the concatenated sequence of this inference update, the local filter cache values ​​stored in the local filter cache space of each inference request are updated to obtain the updated local filter cache values ​​in the local filter cache space of each inference request. The updated local filter cache values ​​are concatenated to obtain the updated concatenated sequence. The next inference is performed using the large language model based on the updated concatenated sequence and the markers generated by this inference.

4. The method according to claim 1, characterized in that, The method further includes: When performing a second and subsequent inference on any inference request using a large language model, after passing through the local filtering attention layer in the first conversion layer, the first local filtering cache value of the first conversion layer corresponding to the inference request is obtained from the first concatenation sequence. The first local filtering cache value of the first conversion layer and the vector corresponding to the inference request are concatenated to obtain the fifth concatenation result. The first local filtering cache value of the first conversion layer corresponding to the inference request in the first concatenation sequence is updated to the vector corresponding to the inference request. A convolution operation is performed on the fifth concatenation result based on the first convolutional layer to obtain a fifth convolution result. The second local filter cache value of the first transformation layer corresponding to the inference request is obtained from the second concatenation sequence. The second local filter cache value of the first transformation layer and the fifth convolution result are concatenated to obtain a sixth concatenation result. A convolution operation is performed on the sixth concatenation result to obtain a sixth convolution result. The sixth convolution result is input into the attention layer for forward propagation calculation. The second local filter cache value of the first transformation layer corresponding to the inference request in the second concatenation sequence is updated to the fifth convolution result.

5. The method according to claim 1, characterized in that, The method further includes: For any inference request, when the inference request meets the preset inference termination condition, the local filter cache space corresponding to the inference request is released; The steps include: obtaining a new inference request, allocating local filter cache space for the new inference request, returning to the steps of obtaining the local filter cache values ​​in the local filter cache space of each inference request, concatenating the local filter cache values, and obtaining a concatenated sequence.

6. The method according to claim 1, characterized in that, The method further includes: During the reasoning process using a large language model, the current number of inferences for each inference request is obtained; Based on the current number of inferences for each inference request, multiple inference requests are classified to distinguish between first-time inference requests and non-first-time inference requests; Based on the large language model, each initial inference request is responded to in a serial manner, and each non-initial inference request is responded to in a parallel manner.

7. A model reasoning device, characterized in that, The device includes: A receiving module is used to receive multiple reasoning requests input by the user, wherein the reasoning requests include prompt words; An allocation module is used to allocate local filter cache space for each of the inference requests. The local filter cache space is used to store local filter cache values ​​and is associated with the inference request. The first acquisition module is used to acquire the local filter cache value in the local filter cache space of each inference request, and concatenate the local filter cache value to obtain a concatenation sequence. The reasoning module is used to perform multiple inferences based on the prompts for each reasoning request and the concatenated sequence using a large language model, in order to obtain the reasoning results for each reasoning request. Specifically, after each inference is completed, the local filter cache value stored in the local filter cache space of each inference request is updated so that the next inference is based on the updated local filter cache value. The model inference device also includes: The first update module is used to update the spliced ​​sequence during each inference process; The first update module includes: The second update unit is used to perform the first inference on any inference request using the large language model, and after passing through the local filtering attention layer in the first conversion layer, obtain the first local filtering cache value of the first conversion layer corresponding to the inference request from the first concatenation sequence, concatenate the first local filtering cache value of the first conversion layer and the vector sequence corresponding to the inference request to obtain the first concatenation result, and update the first local filtering cache value of the first conversion layer corresponding to the inference request in the first concatenation sequence to the vector sequence corresponding to the inference request. The third update unit is used to perform a convolution operation on the first concatenation result based on the first convolutional layer to obtain the first convolution result, obtain the second local filter cache value of the first transformation layer corresponding to the inference request from the second concatenation sequence, concatenate the second local filter cache value of the first transformation layer and the first convolution result to obtain the second concatenation result, perform a convolution operation on the second concatenation result to obtain the second convolution result, input the second convolution result into the attention layer for forward propagation calculation, and update the second local filter cache value of the first transformation layer corresponding to the inference request in the second concatenation sequence to the first convolution result; The fourth update unit is used to perform the first inference on the inference request using the large language model, and after passing through the local filtering attention layer of any of the other conversion layers besides the first conversion layer, obtain the first local filtering cache value of the conversion layer corresponding to the inference request from the first concatenation sequence, concatenate the first local filtering cache value of the conversion layer and the output result of the previous conversion layer of the conversion layer to obtain the third concatenation result, and update the first local filtering cache value of the conversion layer corresponding to the inference request in the first concatenation sequence to the output result of the previous conversion layer; The fifth update unit is used to perform a convolution operation on the third concatenation result based on the first convolutional layer to obtain the third convolution result, obtain the second local filter cache value of the transformation layer corresponding to the inference request from the second concatenation sequence, concatenate the second local filter cache value of the transformation layer and the third convolution result to obtain the fourth concatenation result, perform a convolution operation on the fourth concatenation result to obtain the fourth convolution result, input the fourth convolution result into the attention layer for forward propagation calculation, and update the second local filter cache value of the transformation layer corresponding to the inference request in the second concatenation sequence to the third convolution result; The local filtering cache space includes a first local filtering cache space and a second local filtering cache space. The concatenation sequence includes a first concatenation sequence and a second concatenation sequence. The first local filtering cache value in the first local filtering cache space of each inference request is concatenated to obtain the first concatenation sequence. The second local filtering cache value in the second local filtering cache space of each inference request is concatenated to obtain the second concatenation sequence.

8. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the model reasoning method of any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the model reasoning method of any one of claims 1 to 6.

Images