A data processing method, device, equipment, medium and program product

Abstract

The present application discloses a data processing method, device, equipment, medium and program product in the field of computer technology. In the initialization stage of model reasoning, the present application creates a first cache area and a second cache area; if a model reasoning request is received, the context information of the preprocessing stage of the model reasoning request is stored in the first cache area; based on the context information in the first cache area, each decoding operation in the decoding stage of the model reasoning request is executed; wherein, the context information of the previous decoding operation is stored in the second cache area, and the next decoding operation of the previous decoding operation is executed based on the context information in the second cache area. This scheme is based on the principle of exchanging space for reasoning efficiency, and utilizes two cache areas to store the context information of the preprocessing stage and the decoding stage respectively, which improves the model reasoning performance and efficiency, and does not require changing the model structure, so that the model can be quickly put into use.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, equipment, medium and program product. Background Art

[0002] Currently, the model's reasoning framework only supports Key and Value caching, which requires additional local structures to store context information during model reasoning, which prolongs the model launch cycle and affects model reasoning efficiency.

[0003] Therefore, how to improve the efficiency of model reasoning is a problem that technical personnel in this field need to solve. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a data processing method, device, equipment, medium and program product to improve the efficiency of model reasoning. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a data processing method, comprising:

[0006] In the model initialization phase, a first buffer area and a second buffer area are created;

[0007] If a model inference request is received, storing the context information of the preprocessing stage of the model inference request in the first cache area;

[0008] Based on the context information in the first buffer area, perform each decoding operation in the decoding phase of the model inference request;

[0009] The context information of the previous decoding operation is stored in the second buffer area, and the next decoding operation of the previous decoding operation is performed based on the context information in the second buffer area.

[0010] Optionally, creating the first cache area and the second cache area includes:

[0011] Determine the number of model layers, batch request quantity, and unit message size;

[0012] Calculating the product of the number of model layers, the number of batch requests, and the unit message size;

[0013] The product is used as the space size of the first buffer area and the second buffer area, and the first buffer area and the second buffer area of ​​equal size are created.

[0014] Optionally, it also includes:

[0015] If there are multiple model inference requests, then calculate the starting index values ​​and the ending index values ​​of the multiple model inference requests;

[0016] The context information corresponding to the multiple model inference requests in the first cache area and the second cache area is marked using the starting index value and the ending index value.

[0017] Optionally, before calculating the start index value and the end index value of the plurality of model inference requests, the method further includes:

[0018] Create a request list;

[0019] The plurality of model inference requests are sequentially added to the request list according to the request identification information.

[0020] Optionally, it also includes:

[0021] If the preprocessing stage and the decoding stage of any model inference request are completed, the corresponding model inference request is deleted from the request list, and the corresponding context information in the first cache area and the second cache area is cleared.

[0022] Optionally, calculating the start index value and the end index value of the plurality of model inference requests includes:

[0023] If the model is run for the first time, then for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests for the multiple model inference requests.

[0024] Optionally, after calculating the start index value and the end index value of the plurality of model inference requests, the method further includes:

[0025] If other model inference requests from the same batch as the multiple model inference requests are also received, then for the other model inference requests, the starting index value is set to the ending index value of the multiple model inference requests, and the ending index value is set to the first target value; the first target value is calculated based on the ending index value of the multiple model inference requests, the total number of requests of the multiple model inference requests, and the total number of requests in a single batch.

[0026] Optionally, the calculation process of the first target value includes:

[0027] Calculate the sum of the end index values ​​of the plurality of model inference requests and the total number of requests of the plurality of model inference requests;

[0028] Using the sum, modulo twice the total number of single batch requests;

[0029] The remainder result is used as the first target value.

[0030] Optionally, calculating the start index value and the end index value of the plurality of model inference requests includes:

[0031] If the model is not running for the first time, the previous stage is the preprocessing stage, and the number of requests currently in the preprocessing stage is less than the total number of requests in a single batch, then for multiple model inference requests, the starting index value is set to the total number of requests in a single batch, and the ending index value is set to the second target value; the second target value is calculated based on the total number of requests in a single batch and the number of requests currently in the decoding stage.

[0032] Optionally, the sum of the total number of requests in the single batch and the number of requests currently in the decoding phase is used as the second target value.

[0033] Optionally, calculating the start index value and the end index value of the plurality of model inference requests includes:

[0034] If the model is not running for the first time, the previous stage is the preprocessing stage, the number of requests currently in the preprocessing stage is equal to the total number of requests in a single batch, and the number of requests currently in the decoding stage is 0, then adjust the assignment of the decoding flag so that the first cache area is used to store the context information of the decoding stage of multiple model inference requests, and the second cache area is used to store the context information of the preprocessing stage of multiple model inference requests, and the preprocessing stage is currently executed; for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests of the multiple model inference requests.

[0035] Optionally, calculating the start index value and the end index value of the plurality of model inference requests includes:

[0036] If the model is not running for the first time, the previous stage is the decoding stage, and the number of requests currently in the decoding stage is less than the number of requests in the previous decoding stage, the value of the decoding flag is adjusted so that the first cache area is used to store the context information of the decoding stage of multiple model inference requests, and the second cache area is used to store the context information of the preprocessing stage of multiple model inference requests, and the preprocessing stage is currently executed; for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests of the multiple model inference requests.

[0037] Optionally, it also includes:

[0038] Based on the context information of each decoding operation in the decoding stage of the model inference request, a corresponding model inference result is determined.

[0039] Optionally, it also includes:

[0040] The model inference result is sent to a preset management terminal to display the model inference result on the preset management terminal.

[0041] Optionally, it also includes:

[0042] A new model inference request is generated based on the model inference result, and the new model inference request is processed.

[0043] In a second aspect, the present application provides a data processing device, comprising:

[0044] A creation module, used to create a first buffer area and a second buffer area during the model initialization phase;

[0045] A first storage module, configured to store context information of a preprocessing stage of the model inference request in the first cache area if a model inference request is received;

[0046] A second storage module is used to execute each decoding operation in the decoding stage of the model inference request based on the context information in the first cache area; wherein the context information of the previous decoding operation is stored in the second cache area, and the next decoding operation of the previous decoding operation is executed based on the context information in the second cache area.

[0047] Optionally, create a module specifically for:

[0048] Determine the number of model layers, batch request quantity, and unit message size;

[0049] Calculating the product of the number of model layers, the number of batch requests, and the unit message size;

[0050] The product is used as the space size of the first buffer area and the second buffer area, and the first buffer area and the second buffer area of ​​equal size are created.

[0051] Optionally, it also includes:

[0052] A marking module is used to calculate the starting index value and the ending index value of the multiple model inference requests if there are multiple model inference requests; and use the starting index value and the ending index value to mark the context information corresponding to the multiple model inference requests in the first cache area and the second cache area respectively.

[0053] Optionally, it also includes:

[0054] The queue module is used to create a request list; and add multiple model inference requests to the request list in sequence according to the request identification information.

[0055] Optionally, the queue module is also used to:

[0056] If the preprocessing stage and the decoding stage of any model inference request are completed, the corresponding model inference request is deleted from the request list, and the corresponding context information in the first cache area and the second cache area is cleared.

[0057] Optionally, the marking module is specifically used for:

[0058] If the model is run for the first time, then for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests for the multiple model inference requests.

[0059] Optionally, the marking module is specifically used for:

[0060] If other model inference requests from the same batch as the multiple model inference requests are also received, then for the other model inference requests, the starting index value is set to the ending index value of the multiple model inference requests, and the ending index value is set to the first target value; the first target value is calculated based on the ending index value of the multiple model inference requests, the total number of requests of the multiple model inference requests, and the total number of requests in a single batch.

[0061] Optionally, the calculation process of the first target value includes:

[0062] Calculate the sum of the end index values ​​of the plurality of model inference requests and the total number of requests of the plurality of model inference requests;

[0063] Using the sum, modulo twice the total number of single batch requests;

[0064] The remainder result is used as the first target value.

[0065] Optionally, the marking module is specifically used for:

[0066] If the model is not running for the first time, the previous stage is the preprocessing stage, and the number of requests currently in the preprocessing stage is less than the total number of requests in a single batch, then for multiple model inference requests, the starting index value is set to the total number of requests in a single batch, and the ending index value is set to the second target value; the second target value is calculated based on the total number of requests in a single batch and the number of requests currently in the decoding stage.

[0067] Optionally, the sum of the total number of requests in the single batch and the number of requests currently in the decoding phase is used as the second target value.

[0068] Optionally, the marking module is specifically used for:

[0069] If the model is not running for the first time, the previous stage is the preprocessing stage, the number of requests currently in the preprocessing stage is equal to the total number of requests in a single batch, and the number of requests currently in the decoding stage is 0, then adjust the assignment of the decoding flag so that the first cache area is used to store the context information of the decoding stage of multiple model inference requests, and the second cache area is used to store the context information of the preprocessing stage of multiple model inference requests, and the preprocessing stage is currently executed; for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests of the multiple model inference requests.

[0070] Optionally, the marking module is specifically used for:

[0071] If the model is not running for the first time, the previous stage is the decoding stage, and the number of requests currently in the decoding stage is less than the number of requests in the previous decoding stage, the value of the decoding flag is adjusted so that the first cache area is used to store the context information of the decoding stage of multiple model inference requests, and the second cache area is used to store the context information of the preprocessing stage of multiple model inference requests, and the preprocessing stage is currently executed; for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests of the multiple model inference requests.

[0072] Optionally, it also includes:

[0073] The result generation module is used to determine the corresponding model reasoning result based on the context information of each decoding operation in the decoding stage of the model reasoning request.

[0074] Optionally, it also includes:

[0075] The result push module is used to send the model reasoning result to the preset management end so as to display the model reasoning result on the preset management end.

[0076] Optionally, it also includes:

[0077] The result of the loop processing is used to generate a new model inference request based on the model inference result and process the new model inference request.

[0078] In a third aspect, the present application provides an electronic device, including:

[0079] Memory for storing computer programs;

[0080] The processor is used to execute the computer program to implement the data processing method disclosed above.

[0081] Optionally, the memory includes a first cache area and a second cache area;

[0082] The first cache area is used to store context information of the preprocessing stage of the model reasoning request;

[0083] The second buffer area is used to store context information of each decoding operation in the decoding stage of the model inference request.

[0084] In a fourth aspect, the present application provides a non-volatile storage medium for storing a computer program, wherein the computer program implements the aforementioned disclosed data processing method when executed by a processor.

[0085] In a fifth aspect, the present application provides a computer program product, comprising a computer program / instruction, which implements the steps of the aforementioned disclosed data processing method when executed by a processor.

[0086] It can be seen from the above scheme that the present application provides a data processing method, including: in the model initialization stage, creating a first cache area and a second cache area; if a model inference request is received, storing the context information of the preprocessing stage of the model inference request in the first cache area; based on the context information in the first cache area, executing each decoding operation in the decoding stage of the model inference request; wherein, the context information of the previous decoding operation is stored in the second cache area, and the next decoding operation of the previous decoding operation is executed based on the context information in the second cache area.

[0087] It can be seen that the beneficial effects of the present application are: in the initialization stage of model reasoning, a first cache area and a second cache area are created; if a model reasoning request is received, the context information of the preprocessing stage of the model reasoning request is stored in the first cache area; based on the context information in the first cache area, each decoding operation in the decoding stage of the model reasoning request is executed; wherein, the context information of the previous decoding operation is stored in the second cache area, and the next decoding operation of the previous decoding operation is executed based on the context information in the second cache area. This scheme is based on the principle of exchanging space for reasoning efficiency, and uses two cache areas to store the context information of the preprocessing stage and the decoding stage respectively, which improves the model reasoning performance and efficiency, and does not require changing the model structure, so that the model can be quickly put into use.

[0088] Correspondingly, a data processing device, equipment, medium and program product provided by the present application also have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0090] Figure 1 A flow chart of a data processing method disclosed in this application;

[0091] Figure 2 A schematic diagram of cache partitioning disclosed in this application;

[0092] Figure 3 A flowchart of context information caching in a model reasoning process disclosed in this application;

[0093] Figure 4 A schematic diagram of a data processing device disclosed in this application;

[0094] Figure 5 A schematic diagram of an electronic device disclosed in this application;

[0095] Figure 6 A server structure diagram provided for this application;

[0096] Figure 7 A terminal structure diagram provided for this application. DETAILED DESCRIPTION

[0097] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other examples obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0098] At present, the model's reasoning framework only supports Key and Value caching, which requires additional local structures to store context information during model reasoning, which will extend the model launch cycle and affect the model reasoning efficiency. To this end, this application provides a data processing solution that can use two cache areas to store context information in the preprocessing stage and the decoding stage respectively based on the principle of exchanging space for reasoning efficiency, thereby improving the model reasoning performance and efficiency without changing the model structure, and enabling the model to be quickly put online for application.

[0099] See also Figure 1 As shown, the embodiment of the present application discloses a data processing method, including:

[0100] S101. In a model initialization phase, a first buffer area and a second buffer area are created.

[0101] In one embodiment, creating a first cache area and a second cache area includes: determining the number of model layers, the number of batch requests, and the unit information size; calculating the product of the number of model layers, the number of batch requests, and the unit information size; using the product as the space size of the first cache area and the second cache area, and creating a first cache area and a second cache area of ​​equal size. The number of model layers is the number of structural layers in the model, such as the number of transformer layers in the Transformer autoregressive model; the number of batch requests is: the number of requests for a single batch; and the unit information size is the number of bytes of the tensor.

[0102] In one embodiment, if there are multiple model inference requests, the starting index values ​​and ending index values ​​of the multiple model inference requests are calculated; the starting index value and the ending index value are used to mark the context information corresponding to the multiple model inference requests in the first cache area and the second cache area respectively.

[0103] In one example, before calculating the start index value and the end index value of multiple model inference requests, the method further includes: creating a request list; and sequentially adding multiple model inference requests to the request list according to the request identification information. Accordingly, if the preprocessing stage and the decoding stage of any model inference request are completed, the corresponding model inference request is deleted from the request list, and the corresponding context information in the first cache area and the second cache area is cleared.

[0104] In one example, the starting index value and the ending index value of multiple model inference requests are calculated, including: if the model is run for the first time, for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests of the multiple model inference requests.

[0105] In one example, after calculating the starting index value and the ending index value of multiple model inference requests, it also includes: if other model inference requests in the same batch as the multiple model inference requests are received, then for the other model inference requests, the starting index value is set to the ending index value of the multiple model inference requests, and the ending index value is set to the first target value; the first target value is calculated based on the ending index value of the multiple model inference requests, the total number of requests of the multiple model inference requests, and the total number of requests in a single batch. The calculation process of the first target value includes: calculating the sum of the ending index values ​​of the multiple model inference requests and the total number of requests of the multiple model inference requests; using the sum to take the modulus of twice the total number of requests in a single batch; and taking the modulus result as the first target value.

[0106] In one example, the start index value and the end index value of multiple model inference requests are calculated, including: if the model is not running for the first time, the previous stage is the preprocessing stage, and the number of requests currently in the preprocessing stage is less than the total number of requests in a single batch, then for multiple model inference requests, the start index value is set to the total number of requests in a single batch, and the end index value is set to the second target value; the second target value is calculated based on the total number of requests in a single batch and the number of requests currently in the decoding stage. Wherein, the sum of the total number of requests in a single batch and the number of requests currently in the decoding stage is taken as the second target value.

[0107] In one example, the start index value and the end index value of multiple model inference requests are calculated, including: if the model is not running for the first time, the previous stage is the preprocessing stage, the number of requests currently in the preprocessing stage is equal to the total number of requests in a single batch, and the number of requests currently in the decoding stage is 0, then adjust the assignment of the decoding flag (for example: the decoding flag is assigned to 1), so that the first cache area is used to store the context information of the decoding stage of multiple model inference requests, and the second cache area is used to store the context information of the preprocessing stage of multiple model inference requests, and the preprocessing stage is currently executed; for multiple model inference requests, the start index value is set to 0, and the end index value is set to the total number of requests of multiple model inference requests. Among them, when the decoding flag is 0, it is used to indicate that the cache area where the model is operating stores the context information of the decoding stage; when the decoding flag is 1, it indicates that the cache area where the model is operating stores the context information of the preprocessing stage. Initially, the decoding flag is assigned to 0.

[0108] In one example, the starting index value and the ending index value of multiple model inference requests are calculated, including: if the model is not running for the first time, the previous stage is the decoding stage, and the number of requests currently in the decoding stage is less than the number of requests in the previous decoding stage, then the assignment of the decoding flag bit is adjusted (for example: the decoding flag bit is assigned to 1), so that the first cache area is used to store the context information of the decoding stage of multiple model inference requests, and the second cache area is used to store the context information of the preprocessing stage of multiple model inference requests, and the preprocessing stage is currently executed; for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests of the multiple model inference requests.

[0109] S102: If a model inference request is received, the context information of the preprocessing stage of the model inference request is stored in the first cache area.

[0110] S103. Based on the context information in the first cache area, execute each decoding operation in the decoding stage of the model inference request; wherein, the context information of the previous decoding operation is stored in the second cache area, and the next decoding operation of the previous decoding operation is executed based on the context information in the second cache area.

[0111] It should be noted that the preprocessing stage of a model inference request can be considered as: the encoding stage of the first word in the sentence input to the model by the request, so the context information of the preprocessing stage of the request is: the encoding information tensor of the first word in the sentence. Correspondingly, the decoding stage of the request can be considered as: the encoding stage of the second word and other words after it in the sentence input to the model by the request, so each decoding operation in the decoding stage of the request is used to: encode the second word and other words after it, and the context information of each decoding operation is: the encoding information tensor of the second word and other words after it in the sentence; and for the entire sentence, the encoding information tensor of the next word X depends on the encoding information of at least one word before X. That is to say: when performing a decoding operation based on the context information in the first buffer or the context information in the second buffer, the information actually relied on may include the context information of the previous word or the context information of the previous words.

[0112] In this embodiment, the corresponding model reasoning result can be determined based on the context information of each decoding operation in the decoding stage of the model reasoning request. The model reasoning result can be subsequently sent to the preset management terminal to display the model reasoning result on the preset management terminal. A new model reasoning request can also be generated based on the model reasoning result, and the new model reasoning request can be processed according to the method provided in this embodiment. The model in this embodiment can be a natural language processing model of any architecture.

[0113] In this embodiment, in the initialization phase of model reasoning, a first cache and a second cache are created; if a model reasoning request is received, the context information of the preprocessing phase of the model reasoning request is stored in the first cache; based on the context information in the first cache, each decoding operation in the decoding phase of the model reasoning request is performed; wherein the context information of the previous decoding operation is stored in the second cache, and the next decoding operation of the previous decoding operation is performed based on the context information in the second cache. This solution is based on the principle of exchanging space for reasoning efficiency, and uses two caches to store the context information of the preprocessing phase and the decoding phase respectively, which improves the model reasoning performance and efficiency, and does not require changing the model structure, so that the model can be quickly put into use.

[0114] The following introduces the solution by taking the transformer architecture as an example. In the attention mechanism in the transformer architecture, query, key, and value are three core concepts, where query is used to measure the relevance to each key, key is used to build the calculation basis of attention score, and value is used to perform weighted summation based on attention score to generate the final attention output.

[0115] The query vector represents part of the query, which usually corresponds to information related to the current position (such as the current position in the decoding phase) or the entire sequence (encoding phase). The query is used to measure the relevance of information at all other positions. When the model needs to generate the output at the current moment, the query vector plays a role in finding relevant information.

[0116] The Key vector represents the "index" or "lookup table" of the information at each position in the input sequence, which can be understood as the identification feature of each position. When calculating the attention weight, the Key vector of each position is compared with the Query vector, and the correlation between the Query and each position in the sequence is evaluated by calculating the dot product or other similarity functions. The Key vector determines which input position information is most important to the current Query.

[0117] The Value vector is the carrier of the content or context information associated with each position. It contains the information that is actually to be paid attention to and combined into the output, that is, the information at each position that we think is valuable for subsequent processing. When the Key of a certain position matches the Query highly, the weight of the corresponding Value vector will increase, so this part of information will be considered more when calculating the final output.

[0118] It should be noted that the Transformer architecture model is usually trained in a right-shifted manner. Right-shift is an operation used in sequence-to-sequence models. It is used to prepare the target sequence input for the decoder during the training phase of the autoregressive model of the Transformer architecture. In the Transformer autoregressive model, the decoder generates tokens of the output sequence one at a time, using previously generated tokens as context. In order to achieve this behavior during training, the target sequence is shifted one position to the right, effectively delaying the presentation of each token in the decoder by one time step, that is, adding a mask to the input to mask the tokens after the current position. This ensures that the decoder cannot access future tokens that are not visible in the target sequence during training, thereby preserving the autoregressive properties of the model. Right shift is usually implemented by adding a special sequence start marker to the beginning of the target sequence and removing the last token, so that the target sequence is shifted one position to the right.

[0119] In the training phase, the entire sentence is input to the model, and the tags at all positions in the sentence are calculated in parallel. In the predictive reasoning phase, the entire sentence is input to the trained model, and the representation of the last position is taken as the probability distribution of the next tag. The next tag is then confirmed through the sampling strategy, and finally the tag is spliced ​​to the end of the entire sentence to prepare for the next reasoning. Since there is a mask in the training process of the Transformer architecture, during reasoning, the previously generated tags do not need to be associated with the subsequent tags, so that the previously calculated Key and Value can be cached. This avoids repeated calculations, greatly reduces the amount of parameter calculations, and improves reasoning efficiency. This strategy is called KV-Cache. This technology can improve reasoning performance by exchanging space for time without affecting any calculation accuracy.

[0120] Accordingly, this embodiment designs a cache module for storing context information and implements it in the model building phase, so that the context information can be cached without modifying the scheduling logic of the reasoning framework KV_Cache.

[0121] See also Figure 2 When the model is initialized, an additional context cache is opened in the accelerator card memory to store the context information generated by the transformer architecture. The context information is the encoding information of the words in the input sentence, which is called tensor. Furthermore, the starting index value is set to start, the ending index value is set to end, the maximum number of requests supported in the decoding stage is the hyperparameter max_num_seqs (the number of requests in a single batch) set by the program, the number of input requests in the preprocessing stage is num_prefills, the number of input requests in the decoding stage is num_decodes, the ping-pong flag value is swap_flag, swap_flag takes 0 to represent the first half of the cache (corresponding to the first cache area) in the decoding stage, swap_flag takes 1 to represent the second half of the cache (corresponding to the second cache area) in the decoding stage, and swap_flag takes 0 during initialization.

[0122] Specifically, the model reasoning process of the transformer architecture is divided into a preprocessing stage (also called a prefilling stage) and a decoding stage. Generally, there is one preprocessing stage (the first word) and multiple decoding stages to generate the reasoning content. When multiple requests occur at the same time, the preprocessing stage and the decoding stage are executed alternately in the reasoning framework. For example, for a request, the preprocessing stage is executed first, and then the decoding stage is executed; if request B is received during the decoding stage of request A, the execution of the decoding stage of request A is suspended, and then the preprocessing stage of request B is executed; the decoding and preprocessing of different requests can be carried out in parallel.

[0123] This embodiment takes into account the interaction of context information in the preprocessing stage and the decoding stage, and makes the first cache area and the second cache area continuous, that is, there is only one cache block in the physical space, and the size of the cache block is 2×max_num_seqs×single attention structure cache size (number of bytes of tensor)×number of transformer layers. The preprocessing stage and the decoding stage share the same cache block, so a unified start index value and end index value can be maintained to dynamically implement the cache mechanism, and a ping-pong mechanism is introduced to support the interaction of cache block positions in the preprocessing stage and the decoding stage. For example, scenario 1: the first decoding stage after the preprocessing stage requires the cache data of the preprocessing stage for calculation; scenario 2: the maximum number of supported requests in the decoding stage is a set hyperparameter. In actual reasoning, the number of input requests at a time is often less than the maximum number of supported requests in the decoding stage. When there is no request exit in the decoding stage, the new request generates a cache through the preprocessing stage; scenario 3: in a high concurrency scenario, all requests exit in decoding.

[0124] For scenario 1, the starting index value and the ending index value of the preprocessing stage cache are the same as those of the decoding stage cache. For scenario 2, when there is no request to exit the decoding stage and the number of requests has not reached the maximum number of requests supported by the decoding stage, the ending index value of the decoding stage cache is the starting index value of the preprocessing stage cache. For scenario 3, when all requests to exit the decoding stage are completed, the starting index value and the ending index value of the cache of the next stage, the preprocessing stage (encoding processing of the first word of the input sentence) and the decoding stage (processing of subsequent words of the input sentence), are the same. The preprocessing stage and the decoding stage do not need to exchange cache data, which can reduce the data interaction between the preprocessing stage and the decoding stage.

[0125] In scenario 1, when the request is the first request, all caches are empty, then swap_flag=0 (0 means that the cache space used for the decoding stage is the main cache), start=0, end=num_prefills. In scenario 2, according to the KV_Cache strategy, due to the large length of the request input in the preprocessing stage and the limited memory space of the AI ​​accelerator card, there are multiple preprocessing stages to generate context information and then call the decoding stage. In the second and subsequent preprocessing stages, start=pre_end, end=(pre_end+num_prefills)%(2*max_num_seqs,), pre_end is the end index value of the previous preprocessing stage, and max_num_seqs is the maximum number of requests for parallel processing. In scenario 3, the previous stage is the decoding stage, and the index value calculation method is the same as scenario 2.

[0126] Calculation of cache index in decoding phase: In scenario 1, the previous phase is the preprocessing phase, and num_prefills in the preprocessing phase is less than max_num_seqs, then start=(swap_flag×max_num_seqs), end=(swap_flag×max_num_seqs)+num_decodes. In scenario 2, the previous phase is the preprocessing phase, and num_prefills in the preprocessing phase is equal to max_num_seqs, all requests in the decoding phase before the code have been exited, swap_flag=1, swap the cache positions of the preprocessing phase and the decoding phase, and start and end calculations are the same as scenario 1. In scenario 3, the previous phase is the decoding phase, and start and end calculations are the same as scenario 1.

[0127] To ensure that the model inference obtains the correct calculation results, the preprocessing stage will clear the information of the corresponding cache location. In this embodiment, the starting index value and the ending index value are dynamically and continuously calculated. The ending index value of the preprocessing stage is modulo the maximum number of requests supported by the decoding stage, resulting in the ending index value being smaller than the starting index value. When the model is executed, the context information tensor is passed in the form of a tensor list, splitting the context information tensor into two tensors for splicing during inference.

[0128] See also Figure 3 ,A context information caching scheme in the model inference process includes:

[0129] Step 1: Initialize the transformer context information cache according to the hyperparameter max_num_seqs of the maximum number of requests supported in the decoding phase set by the inference framework. The model contains 24 transformer layers, and the hidden layer size of the transformer structure is 2048. Each transformer layer contains 2 pieces of context information, so initialize 2 cache blocks with sizes of 24*max_num_seqs*2048 and 24*max_num_seqs*2048 / 2 respectively. The cache block data type is the same as the transformer layer output data type.

[0130] Step 2: Get the input request. When the input request is the first request, build a new request ID list, calculate the starting index value start and the ending index value end of the cache block, clear the cache corresponding to start and end, and the model executes the inference process. It also updates the cache blocks corresponding to start and end accordingly, which are used to obtain the transformer context information in the next decoding stage.

[0131] Step 3: In the decoding phase, check whether cache swap occurs. Cache swap occurs in the following scenarios: 1) The current phase is the decoding phase and the number of input requests num_decodes in this decoding phase is less than the number of input requests in the previous decoding phase; 2) The previous phase is the preprocessing phase. At this time, it is necessary to combine the newly added ID list constructed in the preprocessing phase and the KV_Cache block_tables information of the previous decoding phase to determine which request has exited and perform a new request cache swap. Each time the decoding phase is executed, a tag is generated for each request, the transformer context cache information is updated, and the block_tables information in the KV_Cache is cached.

[0132] Step 4: After all request reasoning is completed, continue to wait for the next request service to arrive, or actively end the service.

[0133] This embodiment completes the entire caching process in the form of a plug-in. The designed dynamic block cache and ping-pong mechanism can effectively reduce the allocation and release of AI accelerator card video memory, reduce model latency and improve model throughput. The plug-in adds context information scheduling logic during the model building phase, avoiding modifications to the KV_Cache scheduling logic of each inference framework platform, and can support multiple inference framework platforms. Based on this logic, this embodiment can also be extended to the inference optimization of other architecture models (such as transformer-XL structure models).

[0134] A data processing device provided in an embodiment of the present application is introduced below. The data processing device described below can be referenced to other embodiments described in this document.

[0135] See also Figure 4 As shown, the embodiment of the present application discloses a data processing device, including:

[0136] A creation module 401 is used to create a first buffer area and a second buffer area during the model initialization phase;

[0137] A first storage module 402 is configured to store context information of a preprocessing phase of the model inference request in a first cache area if a model inference request is received;

[0138] The second storage module 403 is used to execute each decoding operation in the decoding stage of the model inference request based on the context information in the first cache area; wherein the context information of the previous decoding operation is stored in the second cache area, and the next decoding operation of the previous decoding operation is executed based on the context information in the second cache area.

[0139] In one embodiment, the creation module is specifically used to:

[0140] Determine the number of model layers, batch request quantity, and unit message size;

[0141] Calculate the product of the number of model layers, the number of batch requests, and the unit message size;

[0142] The product is used as the space size of the first buffer area and the second buffer area, and the first buffer area and the second buffer area of ​​equal size are created.

[0143] In one embodiment, it further includes:

[0144] A marking module is used to calculate the starting index values ​​and ending index values ​​of multiple model inference requests if there are multiple model inference requests; and use the starting index value and the ending index value to mark the context information corresponding to the multiple model inference requests in the first cache area and the second cache area respectively.

[0145] In one embodiment, it further includes:

[0146] The queue module is used to create a request list; multiple model inference requests are added to the request list in sequence according to the request identification information.

[0147] In one embodiment, the queue module is further configured to:

[0148] If the preprocessing stage and decoding stage of any model inference request are completed, the corresponding model inference request is deleted from the request list, and the corresponding context information in the first cache area and the second cache area is cleared.

[0149] In one embodiment, the marking module is specifically used for:

[0150] If the model is run for the first time, for multiple model inference requests, the start index value is set to 0, and the end index value is set to the total number of requests for multiple model inference requests.

[0151] In one embodiment, the marking module is specifically used for:

[0152] If other model inference requests from the same batch as the multiple model inference requests are also received, then for the other model inference requests, the starting index value is set to the ending index value of the multiple model inference requests, and the ending index value is set to the first target value; the first target value is calculated based on the ending index value of the multiple model inference requests, the total number of requests of the multiple model inference requests, and the total number of requests in a single batch.

[0153] In one embodiment, the calculation process of the first target value includes:

[0154] Calculate the sum of the end index values ​​of the multiple model inference requests and the total number of requests of the multiple model inference requests;

[0155] Use the sum modulo twice the total number of single batch requests;

[0156] The remainder result is used as the first target value.

[0157] In one embodiment, the marking module is specifically used for:

[0158] If the model is not running for the first time, the previous stage is the preprocessing stage, and the number of requests currently in the preprocessing stage is less than the total number of requests in a single batch, then for multiple model inference requests, the starting index value is set to the total number of requests in a single batch, and the ending index value is set to the second target value; the second target value is calculated based on the total number of requests in a single batch and the number of requests currently in the decoding stage.

[0159] In one implementation, the sum of the total number of requests in a single batch and the number of requests currently in the decoding phase is used as the second target value.

[0160] In one embodiment, the marking module is specifically used for:

[0161] If the model is not running for the first time, the previous stage is the preprocessing stage, the number of requests currently in the preprocessing stage is equal to the total number of requests in a single batch, and the number of requests currently in the decoding stage is 0, then adjust the value of the decoding flag so that the first cache area is used to store the context information of the decoding stage of multiple model inference requests, and the second cache area is used to store the context information of the preprocessing stage of multiple model inference requests, and the preprocessing stage is currently executed; for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests of the multiple model inference requests.

[0162] In one embodiment, the marking module is specifically used for:

[0163] If the model is not running for the first time, the previous stage is the decoding stage, and the number of requests currently in the decoding stage is less than the number of requests in the previous decoding stage, the value of the decoding flag is adjusted so that the first cache area is used to store the context information of the decoding stage of multiple model inference requests, and the second cache area is used to store the context information of the preprocessing stage of multiple model inference requests, and the preprocessing stage is currently executed; for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests of the multiple model inference requests.

[0164] In one embodiment, it further includes:

[0165] The result generation module is used to determine the corresponding model reasoning result based on the context information of each decoding operation in the decoding stage of the model reasoning request.

[0166] In one embodiment, it further includes:

[0167] The result push module is used to send the model inference results to the preset management terminal so as to display the model inference results on the preset management terminal.

[0168] In one embodiment, it further includes:

[0169] The results are processed in a loop to generate new model inference requests based on the model inference results and process the new model inference requests.

[0170] Among them, for more specific working processes of each module and unit in this embodiment, reference can be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0171] It can be seen that this embodiment provides a data processing device that can use two cache areas to store context information of the preprocessing stage and the decoding stage respectively based on the principle of exchanging space for reasoning efficiency, thereby improving the model reasoning performance and efficiency without changing the model structure, and enabling the model to be quickly put into use.

[0172] An electronic device provided in an embodiment of the present application is introduced below. The electronic device described below can be referenced to other embodiments described in this document.

[0173] See also Figure 5 As shown, the embodiment of the present application discloses an electronic device, including:

[0174] Memory 501, used for storing computer programs;

[0175] The processor 502 is used to execute the computer program to implement the method disclosed in any of the above embodiments.

[0176] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: in the model initialization stage, a first cache area and a second cache area are created; if a model inference request is received, the context information of the preprocessing stage of the model inference request is stored in the first cache area; based on the context information in the first cache area, each decoding operation in the decoding stage of the model inference request is executed; wherein, the context information of the previous decoding operation is stored in the second cache area, and the next decoding operation of the previous decoding operation is executed based on the context information in the second cache area.

[0177] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: determining the number of model layers, the number of batch requests and the unit information size; calculating the product of the number of model layers, the number of batch requests and the unit information size; using the product as the space size of the first cache area and the second cache area, and creating a first cache area and a second cache area of ​​equal size.

[0178] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: if there are multiple model inference requests, the starting index values ​​and the ending index values ​​of the multiple model inference requests are calculated; and the context information corresponding to the multiple model inference requests in the first cache area and the second cache area are marked using the starting index value and the ending index value.

[0179] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: creating a request list; and sequentially adding multiple model inference requests to the request list according to the request identification information.

[0180] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: if the preprocessing stage and the decoding stage of any model inference request are completed, the corresponding model inference request is deleted from the request list, and the corresponding context information in the first cache area and the second cache area is cleared.

[0181] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: if the model is run for the first time, then for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests for multiple model inference requests.

[0182] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: if other model inference requests from the same batch as the multiple model inference requests are also received, then for the other model inference requests, the starting index value is set to the ending index value of the multiple model inference requests, and the ending index value is set to the first target value; the first target value is calculated based on the ending index values ​​of the multiple model inference requests, the total number of requests of the multiple model inference requests, and the total number of requests in a single batch.

[0183] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: calculating the sum of the end index values ​​of multiple model inference requests and the total number of requests of multiple model inference requests; taking the modulus of twice the total number of single batch requests using the sum; and using the modulus result as the first target value.

[0184] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: if the model is not running for the first time, the previous stage is the preprocessing stage, and the number of requests currently in the preprocessing stage is less than the total number of requests in a single batch, then for multiple model inference requests, the starting index value is set to the total number of requests in a single batch, and the ending index value is set to the second target value; the second target value is calculated based on the total number of requests in a single batch and the number of requests currently in the decoding stage.

[0185] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: the sum of the total number of requests in a single batch and the number of requests currently in the decoding stage is used as the second target value.

[0186] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: if the model is not running for the first time, the previous stage is the preprocessing stage, the number of requests currently in the preprocessing stage is equal to the total number of requests in a single batch, and the number of requests currently in the decoding stage is 0, then adjust the assignment of the decoding flag so that the first cache area is used to store the context information of the decoding stage of multiple model inference requests, and the second cache area is used to store the context information of the preprocessing stage of multiple model inference requests, and the preprocessing stage is currently executed; for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests of the multiple model inference requests.

[0187] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: if the model is not running for the first time, the previous stage is the decoding stage, and the number of requests currently in the decoding stage is less than the number of requests in the previous decoding stage, the assignment of the decoding flag bit is adjusted so that the first cache area is used to store the context information of the decoding stage of multiple model inference requests, and the second cache area is used to store the context information of the preprocessing stage of multiple model inference requests, and the preprocessing stage is currently executed; for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests of the multiple model inference requests.

[0188] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: determining the corresponding model inference result based on the context information of each decoding operation in the decoding stage of the model inference request.

[0189] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: sending the model reasoning result to the preset management end to display the model reasoning result on the preset management end.

[0190] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: generating a new model inference request based on the model inference result, and processing the new model inference request.

[0191] Furthermore, the present application also provides an electronic device. The electronic device can be Figure 6 The server shown can also be Figure 7Terminal shown. Figure 6 and Figure 7 All of them are structural diagrams of electronic devices according to an exemplary embodiment, and the contents in the diagrams cannot be regarded as any limitation on the scope of use of the present application.

[0192] Figure 6 A schematic diagram of the structure of a server provided in an embodiment of the present application. The server may specifically include: at least one processor, at least one memory, a power supply, a communication interface, an input / output interface, and a communication bus. The memory is used to store a computer program, which is loaded and executed by the processor to implement the relevant steps in the data processing disclosed in any of the aforementioned embodiments.

[0193] In this embodiment, the power supply is used to provide working voltage for each hardware device on the server; the communication interface can create a data transmission channel between the server and external devices, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0194] In addition, the memory as a carrier for resource storage can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include operating system, computer programs and data, etc. The storage method can be temporary storage or permanent storage.

[0195] The operating system is used to manage and control the hardware devices and computer programs on the server to realize the operation and processing of the data in the memory by the processor, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to computer programs that can be used to complete the data processing method disclosed in any of the above embodiments, the computer program can also further include computer programs that can be used to complete other specific tasks. In addition to data such as application update information, the data can also include data such as application developer information.

[0196] Figure 7 A schematic diagram of the structure of a terminal provided in an embodiment of the present application, the terminal may specifically include but is not limited to a smart phone, a tablet computer, a laptop computer or a desktop computer, etc.

[0197] Generally, the terminal in this embodiment includes: a processor and a memory.

[0198] Among them, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0199] The memory may include one or more computer non-volatile storage media, which may be non-transitory. The memory may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory is at least used to store the following computer program, wherein, after the computer program is loaded and executed by the processor, it can implement the relevant steps in the data processing method performed by the terminal side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory may also include an operating system and data, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system may include Windows, Unix, Linux, etc. The data may include, but is not limited to, update information of the application.

[0200] In some embodiments, the terminal may also include a display screen, an input and output interface, a communication interface, a sensor, a power supply, and a communication bus.

[0201] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the terminal, and may include more or fewer components than those shown in the figure.

[0202] A non-volatile storage medium provided in an embodiment of the present application is introduced below. The non-volatile storage medium described below can be cross-referenced with other embodiments described herein.

[0203] A non-volatile storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the data processing method disclosed in the above-mentioned embodiment. The non-volatile storage medium is a computer-readable non-volatile storage medium, which, as a carrier for storing resources, may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system, a computer program and data, etc., and the storage method may be temporary storage or permanent storage.

[0204] A computer program product provided in an embodiment of the present application is introduced below. The computer program product described below can be cross-referenced with other embodiments described in this document.

[0205] A computer program product comprises a computer program / instruction, which implements the steps of the aforementioned disclosed data processing method when executed by a processor.

[0206] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0207] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of non-volatile storage medium known in the art.

[0208] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method, characterized in that: include: In the model initialization phase, a first buffer area and a second buffer area are created; If a model inference request is received, storing the context information of the preprocessing stage of the model inference request in the first cache area; Based on the context information in the first buffer area, perform each decoding operation in the decoding phase of the model inference request; wherein the context information of the previous decoding operation is stored in the second buffer area, and a subsequent decoding operation of the previous decoding operation is performed based on the context information in the second buffer area; If there are multiple model inference requests, then calculate the starting index values ​​and the ending index values ​​of the multiple model inference requests; Using the start index value and the end index value, marking the context information respectively corresponding to the plurality of the model inference requests in the first cache area and the second cache area; The step of calculating the starting index value and the ending index value of the plurality of model inference requests includes: If the model is not running for the first time, the previous stage is the preprocessing stage, the number of requests currently in the preprocessing stage is equal to the total number of requests in a single batch, and the number of requests currently in the decoding stage is 0, then adjust the assignment of the decoding flag so that the first cache area is used to store the context information of the decoding stage of multiple model inference requests, and the second cache area is used to store the context information of the preprocessing stage of multiple model inference requests, and the preprocessing stage is currently executed; for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests of the multiple model inference requests.

2. The method according to claim 1, characterized in that Creating a first buffer area and a second buffer area includes: Determine the number of model layers, batch request quantity, and unit message size; Calculating the product of the number of model layers, the number of batch requests, and the unit message size; The product is used as the space size of the first buffer area and the second buffer area, and the first buffer area and the second buffer area of ​​equal size are created.

3. The method according to claim 1, characterized in that Before calculating the start index value and the end index value of the plurality of model inference requests, the method further includes: Create a request list; The plurality of model inference requests are sequentially added to the request list according to the request identification information.

4. The method according to claim 3, characterized in that Also includes: If the preprocessing stage and the decoding stage of any model inference request are completed, the corresponding model inference request is deleted from the request list, and the corresponding context information in the first cache area and the second cache area is cleared.

5. The method according to claim 1, characterized in that Calculating the start index value and the end index value of the plurality of model inference requests, including: If the model is run for the first time, then for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests for the multiple model inference requests.

6. The method according to claim 5, characterized in that After calculating the start index value and the end index value of the plurality of model inference requests, the method further includes: If other model inference requests from the same batch as the multiple model inference requests are also received, then for the other model inference requests, the starting index value is set to the ending index value of the multiple model inference requests, and the ending index value is set to the first target value; the first target value is calculated based on the ending index value of the multiple model inference requests, the total number of requests of the multiple model inference requests, and the total number of requests in a single batch.

7. The method according to claim 6, characterized in that The calculation process of the first target value includes: Calculate the sum of the end index values ​​of the plurality of model inference requests and the total number of requests of the plurality of model inference requests; Using the sum, modulo twice the total number of single batch requests; The remainder result is used as the first target value.

8. The method according to claim 1, characterized in that: Calculating the start index value and the end index value of the plurality of model inference requests, including: If the model is not running for the first time, the previous stage is the preprocessing stage, and the number of requests currently in the preprocessing stage is less than the total number of requests in a single batch, then for multiple model inference requests, the starting index value is set to the total number of requests in a single batch, and the ending index value is set to the second target value; the second target value is calculated based on the total number of requests in a single batch and the number of requests currently in the decoding stage.

9. The method according to claim 8, characterized in that The sum of the total number of requests in the single batch and the number of requests currently in the decoding phase is taken as the second target value.

10. The method according to claim 1, characterized in that Calculating the start index value and the end index value of the plurality of model inference requests, including: If the model is not running for the first time, the previous stage is the decoding stage, and the number of requests currently in the decoding stage is less than the number of requests in the previous decoding stage, the value of the decoding flag is adjusted so that the first cache area is used to store the context information of the decoding stage of multiple model inference requests, and the second cache area is used to store the context information of the preprocessing stage of multiple model inference requests, and the preprocessing stage is currently executed; for multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests of the multiple model inference requests.

11. The method according to any one of claims 1 to 10, characterized in that: Also includes: Based on the context information of each decoding operation in the decoding stage of the model inference request, a corresponding model inference result is determined.

12. The method according to claim 11, characterized in that Also includes: The model inference result is sent to a preset management terminal to display the model inference result on the preset management terminal.

13. The method according to claim 12, characterized in that Also includes: A new model inference request is generated based on the model inference result, and the new model inference request is processed.

14. A data processing device, characterized in that: include: A creation module, used to create a first buffer area and a second buffer area during the model initialization phase; A first storage module, configured to store context information of a preprocessing stage of the model inference request in the first cache area if a model inference request is received; A second storage module is used to perform each decoding operation in the decoding stage of the model inference request based on the context information in the first buffer area; wherein the context information of the previous decoding operation is stored in the second buffer area, and the next decoding operation of the previous decoding operation is performed based on the context information in the second buffer area; A marking module, used for calculating the starting index value and the ending index value of the multiple model inference requests if there are multiple model inference requests; using the starting index value and the ending index value, marking the context information corresponding to the multiple model inference requests in the first cache area and the second cache area respectively; Among them, the starting index value and the ending index value of the multiple model inference requests are calculated, including: if the model is not running for the first time, the previous stage is the preprocessing stage, the number of requests currently in the preprocessing stage is equal to the total number of requests in a single batch, and the number of requests currently in the decoding stage is 0, then the assignment of the decoding flag is adjusted so that the first cache area is used to store the context information of the decoding stage of the multiple model inference requests, and the second cache area is used to store the context information of the preprocessing stage of the multiple model inference requests, and the preprocessing stage is currently executed; for the multiple model inference requests, the starting index value is set to 0, and the ending index value is set to the total number of requests of the multiple model inference requests.

15. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 13.

16. The electronic device according to claim 15, characterized in that: The memory includes a first buffer area and a second buffer area; The first cache area is used to store context information of the preprocessing stage of the model reasoning request; The second buffer area is used to store context information of each decoding operation in the decoding stage of the model inference request.

17. A non-volatile storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

18. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images