Model prediction task operation method and device, medium and product

Abstract

The invention discloses a model prediction task operation method and device, a medium and a product, and relates to the field of artificial intelligence. The method comprises the following steps: receiving a plurality of prediction requests, wherein the prediction requests comprise input contents; generating a content feature representation of the input content; and in at least one of the pre-filling stage and the decoding stage, adopting a preset constraint size to respectively execute a reply prediction process on the content feature representations corresponding to the plurality of input contents to obtain reply contents respectively corresponding to the plurality of input contents. In the embodiment of the invention, the preset constraint size is adopted to execute reply prediction represented by the content features respectively corresponding to the multiple input contents, and the constraint size is used to eliminate dynamic problems during calculation of the processor in a pre-filling stage or a decoding stage, so that the calculation efficiency of the processor is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, medium, and product for performing model prediction tasks. Background Technology

[0002] With the development of Large Language Models (LLMs), the demand for natural language question answering through LLMs is growing.

[0003] In related technologies, large language models can understand the semantics of natural language. By inputting user-input dialogue statements into a large language model, it can predict coherent and logical output text, which are the answers to the questions.

[0004] However, due to the varying lengths of different input dialogue statements and the dynamic changes in the number of input dialogue statements, computer devices incur recompilation overhead due to the aforementioned dynamic issues, reducing the prediction efficiency of large language models. Summary of the Invention

[0005] This application provides a method, apparatus, medium, and product for running a model prediction task, which can improve the prediction inference efficiency of LLM. The technical solution is as follows:

[0006] On the one hand, a method for running a model prediction task is provided, the method comprising:

[0007] Receive at least one prediction request, the prediction request including input content, the prediction request being used to request the execution of a response prediction process through a natural language model to generate response content;

[0008] Generate a content feature representation of the input content;

[0009] The response prediction process includes a pre-filling stage and a decoding stage; wherein, the pre-filling stage is used to transform the feature dimensions of the content feature representation, and the decoding stage is used to analyze the content feature representation corresponding to at least one of the input contents to obtain the response content;

[0010] Using a preset constraint size, a response prediction process is performed on the content feature representation corresponding to at least one of the input contents to obtain response contents corresponding to at least one of the input contents respectively;

[0011] Wherein, the constraint size is used to constrain the feature length of the content feature representation participating in feature transformation in the pre-filling stage, or the constraint size is used to constrain the number of at least one input content to be analyzed in batches in the decoding stage.

[0012] On the other hand, an apparatus for running a model prediction task is provided, the apparatus comprising:

[0013] A receiving module is configured to receive at least one prediction request, the prediction request including input content, the prediction request being used to request the execution of a response prediction process through a natural language model to generate response content;

[0014] The generation module is used to generate a content feature representation of the input content;

[0015] The response prediction process includes a pre-filling stage and a decoding stage; wherein, the pre-filling stage is used to transform the feature dimensions of the content feature representation, and the decoding stage is used to analyze the content feature representation corresponding to at least one of the input contents to obtain the response content;

[0016] The prediction module is used to perform a response prediction process on the content feature representation corresponding to at least one of the input contents using a preset constraint size, so as to obtain the response contents corresponding to at least one of the input contents respectively;

[0017] Wherein, the constraint size is used to constrain the feature length of the content feature representation participating in feature transformation in the pre-filling stage, or the constraint size is used to constrain the number of at least one input content to be analyzed in batches in the decoding stage.

[0018] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the running method of the model prediction task as provided in the embodiments of this application above.

[0019] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored in the storage medium, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method for running the model prediction task as provided in the embodiments of this application above.

[0020] On the other hand, a computer program product is provided that, when the computer program product is run on a computer, causes the computer to perform the running method of the model prediction task provided in the above embodiments of this application.

[0021] On the other hand, a chip is provided, the chip including at least one of programmable logic circuits and program instructions, and a computer device equipped with the chip is used to implement the running method of the model prediction task as provided in the above embodiments of this application.

[0022] The beneficial effects of the technical solutions provided in this application include at least the following:

[0023] By employing a preset constraint size in at least one of the pre-filling and decoding stages, a response prediction process for the content feature representation of the input content is performed. Specifically, in the pre-filling stage, the preset constraint size constrains the feature length of the content feature representations participating in feature transformation. This avoids the need to generate compiled data for uncompiled feature lengths during feature transformation in the pre-filling stage due to differences in feature lengths between different content feature representations. Furthermore, by constraining the feature length of the content feature representations with the constraint size, different feature lengths are constrained to approximately the same value during feature transformation in the pre-filling stage. The feature length corresponding to the bundle size allows different feature lengths to share the compiled data corresponding to the feature length after the constraint size, eliminating the compilation overhead caused by the dynamic nature of the feature length. In the decoding stage, the number of content feature representations executed in the decoding stage is constrained by a preset constraint size. Since the content feature representations executed in the decoding stage at the same time form the matrix for the decoding calculation, the shape of the matrix for the decoding calculation is constrained by the constraint size, thus avoiding dynamic changes in the matrix shape caused by the increase or decrease of content feature representations during the decoding calculation. This avoids the need for the processor to readjust computing resources for calculations with dynamically changing matrix shapes, thereby improving the processor's computing efficiency. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the inference process of an LLM provided in an exemplary embodiment of this application;

[0026] Figure 2 This is a schematic diagram illustrating the use of static batch processing to complete a task sequence, provided in an exemplary embodiment of this application.

[0027] Figure 3 This is a schematic diagram illustrating the use of a continuous batch processing to execute a sequence of tasks, provided in an exemplary embodiment of this application.

[0028] Figure 4 This is a schematic diagram of a computer system provided in one embodiment of this application;

[0029] Figure 5 This is a flowchart of a method for running a model prediction task provided in an exemplary embodiment of this application;

[0030] Figure 6 Based on Figure 5 The illustrated embodiment provides a schematic diagram of the computational flow for the pre-filling stage;

[0031] Figure 7 Based on Figure 5 The illustrated embodiment provides a schematic diagram of the computational flow for the pre-filling stage;

[0032] Figure 8 Based on Figure 5 The illustrated embodiment provides a schematic diagram of the computational flow of the decoding stage;

[0033] Figure 9 Based on Figure 5 The illustrated embodiment provides a schematic diagram of the computational flow of the decoding stage;

[0034] Figure 10 This is a schematic diagram of the operational architecture of a model prediction task provided in an exemplary embodiment of this application;

[0035] Figure 11 This is a schematic block diagram of the processor structure provided in an exemplary embodiment of this application;

[0036] Figure 12 This is a schematic diagram of matrix multiplication provided in an exemplary embodiment of this application;

[0037] Figure 13 This is a flowchart of a method for running a model prediction task provided in another exemplary embodiment of this application;

[0038] Figure 14 This is a flowchart of a method for running a model prediction task provided in another exemplary embodiment of this application;

[0039] Figure 15 This is a flowchart of a method for running a model prediction task provided in another exemplary embodiment of this application;

[0040] Figure 16 Based on Figure 15 The illustrated embodiment provides a schematic diagram of the data caching process for the natural language model computation.

[0041] Figure 17 This is a structural block diagram of a model prediction task execution apparatus provided in an exemplary embodiment of this application;

[0042] Figure 18 This is a structural block diagram of a model prediction task execution apparatus provided in another exemplary embodiment of this application;

[0043] Figure 19 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0045] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0046] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0047] It should be understood that although the terms first, second, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0048] It should be noted that this application may display a prompt interface, pop-up window, or output voice prompts before and during the collection of user-related data (e.g., first account, first text, sample text, sample sample text). These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their relevant data is being collected. This ensures that the application only begins the steps related to collecting user-related data after receiving confirmation from the user regarding the prompt interface or pop-up window; otherwise (i.e., without receiving confirmation from the user), the steps to collect user-related data end, meaning no user-related data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of relevant user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0049] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0050] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0051] A language model (LM) is a model based on machine learning or deep learning techniques. It is trained by analyzing text sequences and predicting the probability distribution of subsequent words or characters. It can grasp the grammatical structure and expression patterns of natural language and generate coherent natural language text.

[0052] Pretrained Language Models (PTMs) are models that learn in unsupervised mode on large text datasets to master contextual representations of language. PTMs can capture deep features of natural language, and by fine-tuning their parameters, performance can be significantly improved when applied to specific natural language processing tasks. Classic PTMs include Transformer-based pretrained models (Bidirectional Encoder Representations from Transformers, BERT12) and Generative Pre-Trained Transformer (GPT) models.

[0053] Large Language Models (LLMs) are language models with an extremely large number of parameters, typically ranging from billions to hundreds of billions (e.g., GPT-41, ChatGPT3, PaLM4). Training large language models requires massive datasets and computational resources. They usually possess excellent generalization capabilities and can achieve outstanding results on various natural language processing tasks, such as text classification, language translation, and question answering systems, through few-shot learning or zero-shot learning. Furthermore, large language models can generate high-quality natural language text, including but not limited to articles, dialogues, and poems. The natural language model involved in this embodiment can be at least one of the above-mentioned language models, pre-trained language models, and large language models. In some embodiments, the natural language model involved in this application embodiment can also be referred to as a generative model.

[0054] Prompts, also known as hints, are text (discrete hints) or numerical vectors (continuous hints) designed to elicit the expected output from a large language model. Prompts are text-based, typically carefully crafted by humans, and explicitly inform the large language model of the task to be performed or the type of content expected to be produced. In the application of large language models, prompts not only accelerate the model's learning of new tasks but also precisely guide the content and style of the model's output to meet specific application requirements.

[0055] During the inference and prediction process of the large language model, a sequence of prompt tokens is generated for each dialogue statement in a request. Based on this sequence of prompt tokens, the LLM generates a series of response tokens until a stop token is generated or the maximum sequence length of the response token sequence is reached. The process of generating the response token sequence is iterative. Each time a new response token is generated, the LLM performs a new calculation on the prompt token sequence and the already generated tokens to generate a new token.

[0056] Indicative Figure 1 This is a schematic diagram of the inference process of an LLM provided in an exemplary embodiment of this application, using an example where the maximum sequence length supported by the LLM model is 8 tokens. Figure 1 As shown, starting from prompt marker 110, one response marker 120 is generated for each iteration. The loop for generating response markers stops when the sequence generated by the LLM model ends at marker 130. Figure 1 In the illustrated embodiment, the example is the input sequence corresponding to a single request, that is... Figure 1 In this example, the batch size is 1.

[0057] For example, if the input sentence is “What is the capital of A city”, it takes 5 forward pass iterations to get the complete answer sequence ["a","b","c","d","e"].

[0058] Large Language Models (LLMs) typically face challenges in inference services due to high computational costs and large memory footprints. Because of the iterative generation nature of LLMs, the inference process is often memory- and I / O-intensive rather than computationally intensive. To improve inference efficiency and reduce latency, continuous batching techniques are currently being developed for LLM inference.

[0059] Static batching methods suffer from efficiency issues when handling LLM inference because the generated lengths of response tags for different prediction requests may vary, leading to insufficient utilization of Graphics Processing Unit (GPU) resources. Static batching only processes new requests after the entire sequence has been generated, which can cause significant performance degradation when sequence lengths vary considerably. Figure 2As shown, four task sequences are completed using static batching. In the first iteration, each task sequence generates a response marker 220 from cue marker 210. After n iterations, the size of each completed response sequence differs because different task sequences emit sequence end markers 230 in different iterations. Although task sequence S3 is completed after two iterations, static batching means that the GPU is not fully utilized until the last response sequence in the batch is generated, as shown below: Figure 2 The task sequence S2 in the process only generates the sequence end marker 230 after six iterations.

[0060] To address the problems of static batch processing, continuous batch processing techniques, also known as dynamic batch processing or iterative scheduling techniques, have been proposed. This technique allows new sequences to be dynamically added during batch processing, thereby improving GPU utilization and inference throughput.

[0061] like Figure 3 As shown, seven task sequences are executed using continuous batch processing. The left side shows the batch after a single iteration, and the right side shows the batch after multiple iterations. When a task sequence generates a sequence end marker, a new task sequence (e.g., task sequences S5, S6, and S7) is inserted at its position, improving GPU utilization. The GPU does not need to wait for all sequences to complete before starting to process a new sequence.

[0062] However, the inference service of large language models (LLMs) is dynamic, with inputs, outputs, and batch sizes changing constantly, which can trigger at least several hundred milliseconds of recompilation overhead, thus creating a dynamic problem.

[0063] Figure 4 A schematic diagram of a computer system provided in one embodiment of this application is shown. This computer system can implement a system architecture for running a model prediction task. The computer system may include: a terminal 100 and a server 200.

[0064] Terminal 100 can be an electronic device such as a mobile phone, tablet computer, vehicle terminal (vehicle system), wearable device, PC (Personal Computer), access control device, or unmanned vending terminal. A client application for the target application can be installed and run on terminal 100. This target application can be a game application or other applications that provide natural language processing capabilities; this application does not limit its specific form. Furthermore, this application does not limit the form of the target application, including but not limited to apps (applications), mini-programs, etc., installed on terminal 100, and can also be in web page form.

[0065] The method for running a model prediction task provided in this application embodiment can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. Figure 4 Taking the computer system shown as an example, the model prediction task can be executed by the terminal 100 (e.g., by the client of the target application installed and running in the terminal 100), or by the server 200, or by the interaction and cooperation between the terminal 100 and the server 200. This application does not limit this.

[0066] It is worth noting that the aforementioned servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0067] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Based on the cloud computing business model, cloud technology encompasses network technology, information technology, integration technology, management platform technology, and application technology. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0068] Furthermore, the aforementioned server can also be implemented as a node in a blockchain system. The technical solution of this application can be combined with blockchain technology. For example, some data involved in the natural language processing method disclosed in this application can be stored on a blockchain. Terminal 100 and server 200 can communicate via a network, such as a wired or wireless network.

[0069] The method for running the model prediction task provided in this application can be applied to at least one of the following scenarios:

[0070] First, AI dialogue scenarios.

[0071] In AI dialogue scenarios, users input dialogue content via a terminal, which is then sent to a server for response prediction. The server may receive one or more dialogue contents simultaneously, and when multiple dialogue contents are received, they are processed uniformly. The server uses a natural language model to predict responses, which includes a pre-filling stage and a decoding stage. In this embodiment, a preset constraint size is used to perform response prediction on the content feature representations corresponding to one or more dialogue contents. The constraint size includes at least one of feature constraint length and batch constraint number. The feature constraint length constrains the feature length of the content feature representations participating in feature transformation during the pre-filling stage, and the batch constraint number constrains the number of at least one dialogue contents analyzed in batches during the decoding stage. Taking the constraint size, which includes the feature constraint length and the batch constraint quantity, as an example, the length of the content feature representation of the dialogue content is padded based on the constraint size until the length of the content feature representation reaches the feature constraint length, or reaches an integer multiple of the feature constraint length. Then, a feature transformation in the pre-filling stage is performed on the padded content feature representation. For the content feature representation transformed in the pre-filling stage, if the number of content feature representations currently executing the decoding stage is less than the batch constraint quantity, then the decoding stage is performed on the transformed content feature representation. If the number of content feature representations currently executing the decoding stage reaches the batch constraint quantity, then the transformed content feature representation is made to wait until a content feature representation executing the decoding stage finishes decoding. After the number of content feature representations executing the decoding stage is less than the batch constraint quantity, the decoding stage is performed on the transformed content feature representation to obtain the response content corresponding to the dialogue content.

[0072] Second, search engine scenarios.

[0073] In a search engine scenario, a user enters search terms as input via a terminal, which is then sent to a server for response prediction. The server may receive one or more search terms simultaneously, and when multiple search terms are received, they are processed uniformly. The server uses a natural language model to predict responses to the search terms, which includes a pre-filling stage and a decoding stage. In this embodiment, the response prediction process employs preset constraint dimensions to perform response prediction on the content feature representations corresponding to one or more search terms. These constraint dimensions include at least one of feature constraint length and batch constraint quantity. Taking the constraint size, which includes the feature constraint length and the batch constraint quantity, as an example, the length of the content feature representation of the search content is padded based on the constraint size until the length of the content feature representation reaches the feature constraint length, or reaches an integer multiple of the feature constraint length. Then, a feature transformation in the pre-filling stage is performed on the padded content feature representation. For the content feature representation transformed in the pre-filling stage, if the number of content feature representations currently executing the decoding stage is less than the batch constraint quantity, then the decoding stage is performed on the transformed content feature representation. If the number of content feature representations currently executing the decoding stage reaches the batch constraint quantity, then the transformed content feature representation is made to wait until a content feature representation executing the decoding stage finishes decoding. After the number of content feature representations executing the decoding stage is less than the batch constraint quantity, the decoding stage is performed on the transformed content feature representation to obtain the response content corresponding to the search content.

[0074] It is worth noting that the above application scenarios are merely illustrative examples, and the specific scenarios for the operation of the model prediction task are not limited in the embodiments of this application.

[0075] Figure 5 This is a flowchart illustrating a method for performing a model prediction task according to an exemplary embodiment of this application. This method can be executed by a terminal, a server, or both. In this embodiment, the method is described using an application on a server as an example. Figure 5 As shown, the method includes the following steps.

[0076] Step 501: Receive at least one prediction request, which includes input content.

[0077] A prediction request is used to request that a natural language model perform a response prediction process to generate response content. That is, after receiving the input content in the prediction request, the natural language model generates and outputs the response content corresponding to that input content through prediction. The response content is used to answer the input content.

[0078] The above response prediction process includes a prefill stage and a decoding stage.

[0079] In some embodiments, when multiple prediction requests are received, these multiple prediction requests are sent by different accounts or different terminals, and are used to request that a natural language model predict the response to the input content. For example, prediction request 1 is sent by account A via a terminal, including the text content "What is the capital of country A?"; prediction request 2 is sent by account B via a terminal, including the text content "What is a neural network?".

[0080] In some embodiments, the sending of the prediction request includes at least one of the following:

[0081] 1. In an AI dialogue scenario, the user inputs and sends dialogue content as input content. This prediction request is used to request the natural language model to generate response content to the above dialogue content.

[0082] 2. In a search engine scenario, when a user enters search terms into the search engine as input, the prediction request is used to request a natural language model to generate a response that is related to the search terms.

[0083] It is worth noting that the above-described scenario for sending prediction requests is merely an illustrative example, and the embodiments of this application do not limit it.

[0084] Step 502: Generate a content feature representation of the input content.

[0085] In some embodiments, a pre-trained feature extraction network extracts content feature representations of the input content, which serve as prompts for the pre-filling and decoding stages.

[0086] In some embodiments, the training process of the feature extraction network includes: acquiring sample input content and response reference content labeled with the sample input content; extracting sample feature representations of the sample input content through the feature extraction network to be trained; generating response prediction content corresponding to the sample input content after passing through a pre-filling stage and a decoding stage; determining a loss value based on the difference between the response reference content and the response prediction content; training the feature extraction network to be trained based on the loss value; obtaining the feature extraction network after iterative training with multiple sets of sample input content and response reference content; and extracting content feature representations of the input content through the feature extraction network.

[0087] The content features extracted by the feature extraction network are represented as vectors.

[0088] In some embodiments, the process of generating content feature representations by a feature extraction network involves converting the input content into a vector form. Typically, feature extraction networks include bag-of-words networks, embedding networks, and convolutional networks. Taking an embedding network as an example, the words in the input content are mapped to a pre-trained feature space. The vector representation of each word in this feature space is then used as its corresponding feature representation. The feature representations of each word in the input content are concatenated to obtain the content feature representation corresponding to the input content. Taking a convolutional network as an example, the pre-trained convolutional network learns and generates semantic features in the input content and generates the corresponding content feature representations.

[0089] Step 503: Using a preset constraint size, perform a response prediction process on the content feature representation corresponding to at least one input content to obtain the response content corresponding to at least one input content.

[0090] First, the pre-filling stage and the decoding stage will be explained separately:

[0091] Prefill stage: This stage processes the tokens of the input content in the prediction request. The prefill stage involves inputting all the tokens of the input content at once and performing a model inference to generate the corresponding key-value (KV) values ​​of the input content, which are then stored in the cache for use in the decode stage.

[0092] The pre-filling stage includes multi-head attention (MHA) computation and feed-forward neural network (FFN) computation.

[0093] The specific calculation process of multi-head attention is as follows: The input is generated as a content feature representation of the input content. This content feature representation is obtained by encoding the input content as prompt content through an embedding network. The shape of this content feature representation is (batch_size, input_seq_len, hidden_size), where batch_size refers to the number of batches of input content processed simultaneously, input_seq_len refers to the text length of the input content, such as the number of characters in the input content, and hidden_size is a pre-set model parameter used to control the feature dimension of the content feature representation.

[0094] Using the above content feature representation and the pre-trained W Q W K W VThe weights are multiplied by three matrices, and the output is used to calculate the Attention function, yielding the Attention result. The Attention calculation formula is shown in Formula 1 below:

[0095] Formula 1:

[0096] Where Q is the content feature representation and weight W Q The result of matrix multiplication, K, is the content feature representation and W. K The result obtained by matrix multiplication, V, is the content feature representation and weight W. V The result of matrix multiplication, d k These are preset dimension parameters, and softmax is a normalized exponential function.

[0097] After the Attention result is output, the preset weights W are used. out Perform matrix multiplication and layer normalization calculations to obtain the final Attention output.

[0098] Indicative, such as Figure 6 As shown, firstly, for the content feature representation 610, respectively through W Q W K W V The weights are subjected to three parallel matrix multiplications, yielding the matrix multiplication results. These results are then transposed (620) and softmaxed (630), and finally applied using the preset weights W. out Perform matrix multiplication and layer normalization (LayerNorm) calculations to obtain the Attention output.

[0099] like Figure 7 As shown, in the pre-filling stage, after obtaining the Attention output, it still needs to be processed by the feedforward neural network, such as... Figure 7 As shown, the Attention output is first subjected to matrix multiplication with preset weights W1, and then activated by the activation function GELU710. After that, it is subjected to matrix multiplication with preset weights W2, and then the output 730 of the pre-filling stage is obtained through layer normalization calculation 720.

[0100] In the pre-filling phase, the output of the KV calculation is cached as the initial key-value pair cache data (KVCache) for the prediction request, that is, storing the aforementioned content feature representation and W. K The result K obtained by matrix multiplication, and the content feature representation and weight W VThe result V obtained from matrix multiplication is subsequently used in the Decode stage. At this point, the shapes of KCache and VCache are (batch_size, input_seq_len, num_head, head_dim), respectively. The product of num_head and head_dim is the aforementioned hidden_size, where num_head and head_dim are pre-set parameters for decomposing hidden_size.

[0101] In the pre-filling stage, the unpredictability of batch_size and input_seq_len causes changes in the shape of the input tensor of the computation graph, which can lead to dynamic problems on the processor.

[0102] The Decode phase is used to generate the response content, which is the sequence of response tokens that serves as input, based on the output of the pre-filling phase. In other words, the Decode phase produces the output tokens of the natural language model.

[0103] In some embodiments, the decoding stage is used to analyze the content feature representation of the input content and then generate the response content word by word. Specifically, the content feature representation is used as input; after generating the first token through the decoding stage's operations, the content feature representation and the key-value data of the first token are concatenated, and this concatenation is used again as input to generate the second token through the decoding stage's operations. This process continues until the decoding stage's prediction is complete, generating the complete response content or reaching a preset upper limit in the number of generated tokens.

[0104] In some embodiments, after taking the content feature representation as input, the decoding stage retrieves the character with the highest prediction probability as a token from the character library and outputs it as the first token. Then, the content feature representation and the KV data of the first token are concatenated and used as input again to retrieve the character with the highest prediction probability as a second token from the character library and output it as the second token. This process continues until the prediction of the decoding stage is completed.

[0105] Similar to the pre-filling stage described above, the decoding stage also includes Multi-Head Attention (MHA) computation and Feedforward Neural Network (FFN) computation, the difference being the shape of the input and output data. Because each analysis and prediction in the decoding stage only inputs one token for each prediction request, the shape of the input feature representation in the decoding stage is (batch_size, 1, hidden_size), which is related to W... Q W K W VAfter the weights are calculated using three matrix multiplications, they are concatenated with the K and V values ​​obtained in the current decoding stage using the KVCache cache from the pre-filling stage, and then used for Attention calculation. (Illustrative example follows.) Figure 8 As shown, in multi-head attention computation, the input feature representations 810 and W are first processed during the decoding stage. Q W K W V The weights are calculated using three parallel matrix multiplications. The KVCache from the pre-filling stage is then concatenated with the K and V values ​​obtained in the current decoding stage. After concatenation, transpose, and normalization, the weights are then applied using the preset weights W. out Matrix multiplication and layer normalization (LayerNorm) calculations are performed to obtain the Attention output.

[0106] like Figure 9 As shown, in the feedforward neural network calculation, the Attention output is subjected to matrix multiplication by preset weight W1, and after being activated by the activation function GELU910, it is subjected to matrix multiplication by preset weight W2. After layer normalization calculation 920, the output 930 of the decoding stage is obtained.

[0107] As the decoding stage involves more decoding steps, the amount of data cached by KVCache gradually increases, meaning the dimension of input_seq_len increases by one each time.

[0108] In other words, during the Decode phase, changes in batch_size and the input_seq_len dimension of KVCache cause dynamic issues in Gaudi.

[0109] Based on the above explanation, the dynamic issues of LLM mainly include two types: 1. Dynamism within a request: that is, the dynamism caused by the continuous accumulation and growth of KVCache; 2. Dynamism between requests: that is, the dynamism caused by the use of dynamic batching technology, the different number of requests participating in decoding at different times in the decoding stage, and the different batch number and dialogue statement length in the pre-filling stage.

[0110] The pre-filling stage is used to transform the feature dimensions of the content feature representation, and the decoding stage is used to analyze the content feature representations corresponding to at least one input content to obtain the response content. The feature dimension refers to the shape of the content feature representation; that is, when the content feature representation is expressed as a vector, the feature dimension includes the number of vector elements corresponding to the content feature representation. For example, when the vector is represented as a×b (a, b), it is a two-dimensional vector; when it is represented as a×b×c (a, b, c), it is a three-dimensional vector. The pre-filling stage transforms the feature dimensions of the content feature representation, meaning that the pre-filling stage transforms the vector representation dimension of the content feature representation and the expression data of each dimension in the vector. Optionally, the feature dimensions of the content feature representation are transformed using a preset transformation formula. The specific transformation process is as follows: Figure 6 and Figure 7 As shown.

[0111] Optionally, the decoding stage is used to perform response analysis on the converted content feature representation based on the pre-filling stage, to obtain the response content corresponding to the input content. In some embodiments, the pre-filling stage and the decoding stage can be executed in parallel; illustratively, the pre-filling stage for the first input content and the decoding stage for the second input content are executed in parallel. This embodiment does not limit the execution order and sequential relationship between the pre-filling stage and the decoding stage.

[0112] The constraint size is used to constrain the feature length of the content feature representations involved in feature transformation during the pre-filling stage, or the constraint size is used to constrain the number of at least one input content items analyzed in the batch during the decoding stage.

[0113] The constraint size is pre-defined data used to constrain the length of a feature, or the constraint size is pre-defined data used to constrain the number of batches processed by the processor.

[0114] In some embodiments, the constraint size is used to constrain the content feature representation to reach the feature constraint length, or the constraint size is used to constrain each content feature representation to reach an integer multiple of the feature constraint length.

[0115] In some embodiments, the constraint size is used to constrain the number of content feature representations that are executed simultaneously in the decoding phase, wherein the constraint size is used to constrain the number of content feature representations that are executed simultaneously in the decoding phase to be kept at a fixed number. Optionally, the number of content feature representations that are executed simultaneously in the decoding phase can be controlled to be kept at a fixed number by means of padding or decoding waiting.

[0116] In some embodiments, the constraint size includes at least one of feature constraint length or batch constraint number, wherein the feature constraint length is used to constrain the feature length of the content feature representations participating in feature transformation during the pre-filling stage, and the batch constraint number is used to constrain the number of content feature representations decoded simultaneously during the decoding stage.

[0117] In some embodiments, constraint sizes are used to eliminate dynamic issues during processor computation in the pre-padding or decoding phases. That is, constraint sizes are used to control the relative uniformity of the data size involved in the computation when performing computations in the pre-padding or decoding phases within the processor.

[0118] When the constraint size includes the feature constraint length, the content feature representation participating in matrix operations during the pre-filling stage is padded based on the feature constraint length until the length of the content feature representation matches the feature constraint length. Specifically, when the processor performs feature calculations during the pre-filling stage, including matrix multiplication, for each feature size multiplication operation performed by the processor, the processor stores the compiled data corresponding to that feature size. Therefore, when performing the same feature size multiplication operation again, the pre-stored compiled data is directly retrieved, avoiding recompilation of the multiplication operation for that feature size. Thus, in this embodiment, by pre-setting the feature constraint size and padding the input content feature representation to match the feature constraint length, the probability of the processor hitting a compiled feature size is increased, thereby improving the processor's computational efficiency.

[0119] Specifically, the content feature representation in the pre-filling stage is padded until the length of the content feature representation is equal to the feature constraint length; or, when the length of the content feature representation in the pre-filling stage is greater than the feature constraint length, the content feature representation in the pre-filling stage is padded until the length of the content feature representation is equal to an integer multiple of the feature constraint length.

[0120] In some embodiments, the constraint size includes multiple feature constraint lengths, and the content feature representation in the pre-filling stage is padded until the length of the content feature representation reaches a size whose length is greater than the minimum feature constraint length of the content feature representation.

[0121] In some embodiments, the feature constraint length is a preset value; or, the feature constraint length is the feature length corresponding to the historical content feature representation after extracting features from the input content in the historically received prediction requests.

[0122] The feature constraint length is greater than a length threshold, such as a length unit greater than 1. The feature constraint length is either manually specified based on processor performance, or it is the feature length obtained from historically stored compiled data based on calculations already performed by the processor. Specifically, the feature length in the historically stored compiled data refers to the feature length corresponding to the historical content feature representation obtained after extracting features from the input content of historically received prediction requests and performing matrix calculations during the pre-filling stage of this historical content feature representation.

[0123] In other words, by setting the feature constraint length, the feature representations of the content involved in matrix calculation during the pre-filling stage can be integer multiples of the feature constraint length. Thus, after the processor analyzes the feature size that is an integer multiple of the feature constraint length, it stores the compilation data corresponding to the feature size. When the same integer multiple of the feature size appears in the future, the stored compilation data is directly used for compilation, which improves compilation efficiency and avoids the dynamic problem of feature size.

[0124] When the constraint size includes the batch constraint number, the number of batches of content feature representations that are simultaneously executed in the decoding phase is limited based on the batch constraint number.

[0125] That is, when the number of decoding tasks is less than the number of batch constraints, the response prediction process in the decoding stage is performed on the transformed content feature representation corresponding to at least one input content to obtain the response content corresponding to the input content.

[0126] In other words, by setting the batch constraint number, the number of content feature representations that can be decoded simultaneously can be limited, thereby constraining the shape of the KVCache and avoiding dynamic issues caused by the continuous increase of the decoding queue in the KVCache due to dynamic batch processing technology, thus improving decoding efficiency.

[0127] In summary, the method provided in this embodiment performs a response prediction stage for the content feature representation of the input content using a preset constraint size in at least one of the pre-filling and decoding stages. Specifically, in the pre-filling stage, the preset constraint size constrains the feature length of the content feature representations participating in feature transformation of the input content. This avoids the need to generate compiled data for uncompiled feature lengths during feature transformation in the pre-filling stage due to differences in feature lengths between different content feature representations. Furthermore, by constraining the feature length of the content feature representations with the constraint size, different feature lengths are avoided during feature transformation in the pre-filling stage. The dimensions are all constrained to the feature length corresponding to the constraint size. Different feature lengths can share the compilation data corresponding to the feature length after the constraint size, eliminating the compilation overhead caused by the dynamic nature of the feature length. In the decoding stage, the number of content feature representations executed in the decoding stage is constrained by the preset constraint size. Since the content feature representations executed in the decoding stage at the same time form the matrix for executing decoding calculations, the number of content feature representations is constrained by the constraint size, which also constrains the shape of the matrix for executing decoding calculations in the decoding stage. This avoids the dynamic change of the matrix shape caused by the increase or decrease of content feature representations during the decoding calculation, and avoids the need for the processor to readjust computing resources for calculations with dynamically changing matrix shapes, thus improving the processor's computing efficiency.

[0128] The method for running the model prediction task provided in this application is explained in conjunction with the software design architecture of the LLM model. For example... Figure 10 As shown, the runtime architecture of the model prediction task includes the following parts:

[0129] Request Receiving Service 1010 (HTTP Server): This service receives prediction requests sent by users. The prediction request includes input content, which serves as a topic prompt. In other words, the input content prompts the natural language model with the topic of the dialogue, and the natural language model is requested to respond to the input content based on the topic prompt. The Request Receiving Service compiles the input content in the prediction request, used as a topic prompt, into integer identifiers (ids) through a tokenizer, and then sends these integer identifiers to the scheduler (Router).

[0130] KV Cache 1020: Used to store KV data generated during the pre-filling and decoding stages, that is, to store the content feature representation and W during the pre-filling and decoding stages. K The result K obtained by matrix multiplication, and the content feature representation and weight W VThe result V obtained by matrix multiplication. The shapes of KCache and VCache are (max_req_num, max_seq_len, num_head, head_dim), respectively. Here, max_req_num refers to the maximum batch size, i.e., the number of batch constraints mentioned above; max_seq_len refers to the maximum statement length, i.e., the length of the feature constraints mentioned above, or an integer multiple of the feature constraint length. The product of num_head and head_dim is the hidden_size mentioned above, where num_head and head_dim are pre-set parameters for decomposing hidden_size.

[0131] Scheduler 1030 (Router): Executes the prefill and decoding stages via route IDs to achieve continuous batch processing. If the Router receives a new prediction request, and there is an idle task queue in the KVCache, it allocates a row in the KVCache for that prediction request, i.e., a memory space of size (max_seq_len, num_head, headim). Then, it sends the IDs of the input content in the prediction request to the prefill compute graph executor to perform graph computation in the prefill stage. If there are multiple prediction requests simultaneously, the IDs of the input content in multiple prediction requests are sent as a batch, and then it waits for the returned output tokens. The scheduler includes at least one queue to store the token IDs corresponding to each request, i.e., the IDs of the input content corresponding to the request or the output tokens of the prefill stage corresponding to the request. If the predicted request does not require graph computation during the pre-filling phase, the Router will assemble the request token ids in the KVCache into a batch and send it to the decode compute graph executor to perform graph computation during the decoding phase.

[0132] Prefill compute graph executor 1040: Used to perform graph computations in the prefill stage on the processor.

[0133] Decode compute graph executor 1050: Used to perform graph computations during the decoding phase on the processor.

[0134] Indicative, such as Figure 11As shown, taking processor 1100 as an example, the architecture of processor 1100 is designed to accelerate deep learning training workloads and includes three main subsystems: computation, memory, and networking. Among them, the computation architecture is heterogeneous, including two components: the Matrix Multiplication Engine 1110 (MME) and several Tensor Processing Cores 1120 (TPC).

[0135] The MME1110 is responsible for performing matrix multiplication operations (such as fully connected layers, convolutions, and batch general matrix multiplication). Typically, it performs 1024×128 or 512×256 matrix multiplications per clock cycle. (Illustrative example follows.) Figure 12 As shown, matrix multiplication 1210 is a schematic diagram of matrix multiplication of 256×256; matrix multiplication 1220 is a schematic diagram of calculation between two 256×16 matrices.

[0136] The TPC1120 accelerates non-general matrix multiplication operations by executing a Single Instruction, Multiple Data (SIMD) approach simultaneously on multiple data points. SIMD is a parallel computing architecture designed to improve execution efficiency, especially when handling vector and matrix operations. In a SIMD architecture, a single control unit issues an operation instruction that simultaneously performs the same operation on different data across multiple processing units. For example, a SIMD instruction might instruct all processing units to add their respective two numbers. This ability to process multiple data points simultaneously makes SIMD ideal for applications requiring large amounts of data processing, such as image processing, digital signal processing, and scientific computing.

[0137] In some embodiments, the constraint size includes a feature constraint length. The feature constraint length is used to constrain the feature length of the content feature representation in the pre-filling stage. Figure 13 This is a flowchart illustrating a method for performing a model prediction task, provided in another exemplary embodiment of this application. This method can be executed by a terminal, by a server, or by a combination of both. In this embodiment, the method is described using an application to a server as an example. Figure 13 As shown, step 503 above can be implemented as steps 5031 to 5033 below.

[0138] Step 5031: In the pre-filling stage, the feature length of the content feature representation is padded to match the feature constraint length.

[0139] Optionally, in the pre-filling stage, the content feature representations corresponding to multiple input contents are respectively compared with the pre-trained W. Q W K W V The weights are subjected to three parallel matrix multiplication operations, and the output results are used to calculate the Attention result.

[0140] First, the feature length of the content feature representation is padded to match the feature constraint length. Then, the content feature representation that matches the feature constraint length is compared with the pre-trained W. Q W K W V The weights are used for matrix multiplication. Therefore, when the processor performs matrix multiplication, the calculation is uniformly based on the feature constraint length, avoiding the dynamic complexity problem caused by the high degree of freedom between different feature representations, which would otherwise require the processor to compile and calculate them separately.

[0141] In this context, the feature length refers to the length of the vector corresponding to the feature representation, that is, the number of elements and the length of the character elements contained in the feature representation. In this embodiment, the feature length of the content feature representation refers to the length of the character elements of the vector when the input content is converted into a vector expression.

[0142] Indicatively, the content feature representation in the expression has the shape of (batch_size, input_seq_len, hidden_size), where hidden_size is a pre-set parameter, and batch_size represents the batch size, which can be expressed by a pre-set number of characters, such as 16 characters. Therefore, the feature length in this embodiment refers to the length of the element input_seq_len, where input_seq_len represents the length of the content feature representation generated from the input content. That is, in the shape of the content feature representation in the expression, the input_seq_len element is used to characterize the content feature representation of the input content, and the feature length refers to the number of characters occupied by the input_seq_len element. In some embodiments, the content feature representation converts the input content into a one-dimensional vector representation, i.e., a string of numbers. Indicatively, when the input content contains 5 characters, the converted string contains 40 characters; when the input content contains 8 characters, the converted string contains 64 characters.

[0143] Padding refers to increasing the number of characters occupied by the content feature representation. That is, based on the number of characters already occupied by the content feature representation, the number of characters occupied is increased by adding other characters until the length of the content feature representation matches the length of the feature constraint. Padding can be done by adding a preset value, or by adding random values. This embodiment uses adding a preset value as an example.

[0144] Optionally, the feature constraint length is a pre-determined data used to constrain the length of the content feature representation. Optionally, zeros are padded starting from the first side of the content feature representation until the feature length of the content feature representation matches the feature constraint length. Here, the first side is a pre-defined data endpoint of the content feature representation. Illustratively, zeros are padded starting from the left side of the content feature representation until the feature length of the content feature representation matches the feature constraint length; or, zeros are padded starting from the right side of the content feature representation until the feature length of the content feature representation matches the feature constraint length.

[0145] In this method, zero padding is performed starting from the first side of the content feature representation to match the length of the feature constraint. This ensures that the length of the content feature representation falls within the length of the content feature representation that has already been compiled in the historical compilation process with a higher probability, avoiding the initialization calculation of the compiled data. Furthermore, the zero padding method is simple and improves the padding efficiency.

[0146] It is worth noting that the above padding method is illustrated using zero padding as an example. In some embodiments, padding can also be achieved by padding with a preset value, such as padding with the character "1" starting from the first side of the content feature representation. This embodiment does not limit the padding method.

[0147] In some embodiments, when padding the content feature representation, padding can begin from a specified position in the content feature representation, such as starting padding from the 9th bit from the left. This is illustrative; the left 8 bits of the content feature representation represent the content type, hence padding begins from the 9th bit from the left. The padding method for the content feature representation is not limited in this embodiment; however, this embodiment uses padding from the left as an example.

[0148] When padding the feature length of the content feature representation to match the feature constraint length, it includes at least one of the following methods:

[0149] 1Among the integer multiples of the feature constraint length, the smallest value that is greater than or equal to the feature length of the content feature representation is used to pad the content feature representation to the feature length corresponding to the smallest value.

[0150] For illustration, if the feature constraint length is 32 and the length of the input content feature representation is 60, then the smallest integer multiple of the feature constraint length greater than 60 is twice the feature constraint length, which is 64. Therefore, the content feature representation is padded with zeros to 64 bits. If the length of the input content feature representation is 70, then the smallest integer multiple of the feature constraint length greater than 70 is three times the feature constraint length, which is 96. Therefore, the content feature representation is padded with zeros to 96 bits.

[0151] By setting a feature constraint length and padding the content feature representation with features that are integer multiples of the constraint length, when a new content feature representation length is encountered for the first time, the computation graph corresponding to that content feature representation needs to be compiled. However, when the same content feature representation length is encountered again, recompilation is not required. Furthermore, using integer multiples reduces the randomness of the content feature representation length, improving the processor's computational efficiency during the pre-padding stage.

[0152] 2 1. Obtain multiple feature constraint lengths, determine the smallest feature constraint length among the multiple feature constraint lengths that is greater than or equal to the content feature representation as the target constraint length, and pad the content feature representation to the target constraint length.

[0153] For illustration, multiple feature constraint lengths are 16, 32, 48, and 96 respectively. The length of the input content feature representation is 40. The minimum feature constraint length greater than 40 is 48, so the content feature representation is padded with zeros to 48 bits.

[0154] In this application, the lengths of multiple feature constraints are either manually pre-set or are the lengths of the content feature representations corresponding to the input content in prediction requests received within a historical time period. This application does not limit the method for obtaining the lengths of multiple feature constraints.

[0155] By setting multiple feature constraint lengths, the content feature representation is padded to a length greater than the minimum feature constraint length of that content feature representation, thereby reducing the randomness of the content feature representation length and improving the processor's computational efficiency in the pre-padding stage. Specifically, by obtaining multiple feature constraint lengths from prediction requests received within historical time periods and from existing feature compilation data, the need to recompile the computation graph for the content feature representation corresponding to the feature constraint length is avoided, further improving the processor's computational efficiency.

[0156] In an optional embodiment, uniform feature padding can be performed on a batch of input content. The sequence feature lengths of the content feature representations corresponding to multiple input contents are obtained, and the largest first feature length among these sequence feature lengths is obtained. A target feature length that matches the feature constraint length is determined, and the content feature representations corresponding to the multiple input contents are padded to this target feature length. Schematic, the smallest integer multiple of the feature constraint length that is greater than or equal to the first feature length is determined, and the content feature representations corresponding to the multiple input contents are padded to the feature length corresponding to this smallest value.

[0157] For illustration, the sequence feature lengths of the content feature representations corresponding to multiple input contents are 18, 28, 36, and 55, respectively. Then, the first feature length, which is the largest among the multiple sequence feature lengths, is determined to be 55. The feature constraint length is 32. The integer multiple of the feature constraint length that is greater than the first feature length is twice the feature constraint length, which is 64. Therefore, the content feature representations corresponding to the multiple input contents are padded to 64 bits.

[0158] Step 5032: Convert the feature dimensions of the padded content feature representation to obtain the converted content feature representation.

[0159] After the Attention result is output, the preset weights W are used. out Perform matrix multiplication and layer normalization calculations to obtain the final Attention output.

[0160] After obtaining the Attention output, it still needs to be processed by the feedforward neural network. First, the Attention output is subjected to matrix multiplication by the preset weight W1, and then activated by the activation function. After that, it is subjected to matrix multiplication by the preset weight W2. The output of the pre-filling stage is obtained by layer normalization calculation, which is the transformed content feature representation.

[0161] Step 5033: In the decoding stage, a response prediction process is performed on the converted content feature representation to obtain response content corresponding to at least one input content.

[0162] In the decoding phase, one token is analyzed and processed at a time. Therefore, the shape of the input feature representation in the decoding phase is (batch_size, 1, hidden_size), which is related to W. Q W K W V After the weights are calculated using three matrix multiplications, they are concatenated with the K and V values ​​obtained in the current decoding stage using the KVCache from the pre-filling stage, and then Attention is calculated. Finally, the preset weights W are used... outMatrix multiplication and layer normalization (LayerNorm) calculations are performed to obtain the Attention output.

[0163] In the feedforward neural network computation of the decoding stage, the Attention output is subjected to matrix multiplication by a preset weight W1, activated by an activation function, and then subjected to matrix multiplication by a preset weight W2. After layer normalization, the output of the decoding stage is obtained.

[0164] In summary, the method provided in this embodiment, by employing a preset constraint size in at least one of the pre-filling and decoding stages, performs response prediction on the content feature representations corresponding to multiple input contents respectively. The constraint size eliminates the dynamic problems of processor calculation in the pre-filling or decoding stages, improves the processor's computational efficiency, enables the LLM model to perform accurate and efficient calculations on the processor, and improves the prediction efficiency of the LLM model.

[0165] The method provided in this embodiment, during the pre-filling stage, pads the feature length of the content feature representation to match the feature constraint length. Therefore, when the processor performs graph calculations such as matrix multiplication, the length of the content feature representation it faces is a relatively stable size that matches the feature constraint length, rather than a content feature representation with a higher degree of freedom in length determined by the input content in the prediction request. That is, because the length of the padded content feature representation matches the feature constraint length, the probability of hitting the feature length compiled within the historical time period is higher. There is no need to re-initialize the compilation process, and it can directly use the pre-stored compilation data that matches the feature length, such as using a pre-stored compilation configuration file that matches the feature length. The configuration file contains parameter settings and hyperparameter information related to feature vector calculation, avoiding the need to re-initialize the compilation process and generate the configuration file for new feature lengths, thus improving compilation efficiency and the processor's graph calculation and prediction efficiency.

[0166] In some embodiments, the constraint size includes the batch constraint number. The batch constraint number is used to constrain the number of content feature representations that are decoded simultaneously during the decoding phase. Figure 14 This is a flowchart illustrating a method for performing a model prediction task, provided in another exemplary embodiment of this application. This method can be executed by a terminal, by a server, or by a combination of both. In this embodiment, the method is described using an application to a server as an example. Figure 14 As shown, step 503 above can be implemented as steps 5041 to 5042 below.

[0167] Step 5041: In the pre-filling stage, the feature dimensions of the content feature representations corresponding to at least one input content are transformed to obtain the transformed content feature representations.

[0168] Optionally, in the pre-filling stage, the content feature representation corresponding to at least one input content is compared with the pre-trained W. Q W K W V The weights are subjected to three parallel matrix multiplication operations, and the output results are used to calculate the Attention result.

[0169] After the Attention result is output, the preset weights W are used. out Perform matrix multiplication and layer normalization calculations to obtain the final Attention output.

[0170] After obtaining the Attention output, it still needs to be processed by the feedforward neural network. First, the Attention output is subjected to matrix multiplication by the preset weight W1, and then activated by the activation function. After that, it is subjected to matrix multiplication by the preset weight W2. The output of the pre-filling stage is obtained by layer normalization calculation, which is the transformed content feature representation.

[0171] Step 5042: If the number of content feature representations decoded simultaneously is less than the number of batch constraints, perform a decoding stage on the transformed content feature representation corresponding to at least one input content to obtain the response content corresponding to the input content.

[0172] In other words, in this embodiment, to avoid dynamic issues arising from changes in the shape of the decoding computation graph caused by dynamic batch processing, a batch constraint number is pre-set to limit the number of tasks that simultaneously perform response prediction in the decoding phase. In some embodiments, when the number of tasks that need to perform the decoding phase at the same time is greater than the batch constraint number, the decoding task corresponding to the batch constraint number is selected to perform the decoding phase. When the number of tasks performing the decoding phase at the same time is less than the batch constraint number, the idle sequences in the data buffer are padded so that the sequence of tasks in the decoding phase reaches the batch constraint number.

[0173] In some embodiments, the decoding graph executor simultaneously calculates matrices composed of content feature representations from multiple decoding stages. For example, if the batch constraint is 9, the decoding graph executor simultaneously performs decoding calculations on matrices composed of content feature representations from 9 decoding stages. Optionally, the decoding matrix is ​​generated by combining the content feature representations from the 9 currently executing decoding stages with the KVCache data generated for each content feature representation in the decoding stage, and the decoding graph executor performs matrix operations on the decoding matrix. That is, the decoding stage is used to batch analyze the content feature representations corresponding to at least one input content to obtain the response content.

[0174] During the decoding phase of the transformed content feature representations corresponding to at least one input content, if the number of content feature representations being decoded simultaneously is less than the batch constraint number, idle sequences are padded to ensure that the number of content feature representations in the decoding phase reaches the batch constraint number. Specifically, since the content feature representations in the decoding phase occupy decoding sequences in the data buffer, idle sequences in the data buffer are padded if the number of simultaneously decoded content feature representations is less than the batch constraint number.

[0175] In other words, the shape of the decoding computation graph during the decoding phase is maintained unchanged through constraint waiting or padding. Constraint waiting refers to the situation where, when the number of decoding tasks reaches the batch constraint number, the tasks to be decoded are constrained to wait in the decoding phase until the number of decoding tasks is less than the batch constraint number. Padding refers to the situation where, when the number of decoding tasks is less than the batch constraint number, the idle queues are padded to maintain the total number of decoding tasks (decoding tasks + idle padding tasks) at the batch constraint number, thus keeping the total number of decoding tasks constant. Padding can be implemented in at least one of the following ways: padding with zeros, padding with preset values, padding with random values, etc. This application embodiment does not limit the location or method of the decoding phase.

[0176] Optionally, the decoded sequence cached in the data cache is obtained. The decoded sequence is the sequence data of the content feature representation stored in the data cache during the decoding process. Schematic, the decoded sequence includes the content feature representation and the KVCache data generated during the decoding process. When the number of decoding tasks corresponding to the decoded sequence is less than the number of batch constraints, the response prediction process of the decoding stage is performed on the transformed content feature representation corresponding to at least one input content to obtain the response content corresponding to the input content.

[0177] When decoding sequences are cached in the data buffer, the idle sequences in the data buffer are identified. Idle sequences refer to sequences not occupied by decoding sequences within the batch constraint. Based on the first number of idle sequences, a response prediction process is performed on the transformed content feature representations corresponding to input content that is less than or equal to the first number of idle sequences to obtain the response content corresponding to the input content. For example, if the batch constraint is 9, meaning the number of content feature representations simultaneously executing the decoding phase is less than or equal to 9, then when the number of currently executing content feature representations in the data buffer is less than 9, it indicates the existence of idle sequences. For instance, if the number of currently executing content feature representations is 7, then there are 2 idle sequences.

[0178] The number of content feature representations currently in the decoding stage is determined by obtaining the number of decoding sequences. Since the content feature representations are stored in the data buffer as decoding sequences, the number of content feature representations executing the decoding stage simultaneously can be clearly obtained through the decoding sequences, which improves the accuracy of controlling idle sequences and the decoding stage.

[0179] After obtaining the first sequence number of free sequences, the processing of the converted content feature representation is controlled according to the first sequence number to perform the decoding stage. This avoids the matrix shape of the decoding calculation being changed due to too many or too few content feature representations being performed in the decoding stage, thus keeping the shape of the decoding calculation graph unchanged and improving the processor's computational efficiency when performing graph calculations in the decoding stage.

[0180] In some embodiments, the scheduler determines whether to advance the transformed content feature representation corresponding to the input content to the decoding stage. That is, after the input content is transformed in the pre-filling stage, a scheduling request is sent to the scheduler. The scheduler determines the number of decoding sequences currently performing decoding calculations and, based on the number of decoding sequences and the number of batch constraints, determines whether to schedule the transformed content feature representation from the pre-filling stage to the decoding stage for decoding calculations.

[0181] In some embodiments, when the content feature representation after the pre-filling stage is sent to the scheduler, if the number of decoding sequences currently being executed in the decoding stage reaches the batch constraint number, the converted content feature representation is queued in the scheduler to wait for decoding; when an idle sequence appears in the data buffer, the scheduler sends the content feature representation that first enters the queue to the decoding graph executor for decoding calculation.

[0182] In summary, the method provided in this embodiment, by employing a preset constraint size in at least one of the pre-filling and decoding stages to perform response prediction on the content feature representations corresponding to multiple input contents, eliminates dynamic problems during processor computation in the pre-filling or decoding stages using the constraint size, thereby improving the processor's computational efficiency. This enables the LLM model to perform accurate and efficient computation on the processor, thus improving the prediction efficiency of the LLM model.

[0183] The method provided in this embodiment, in the decoding stage, constrains the number of tasks performing decoding calculations simultaneously by limiting the batch size, and maintains the shape of the decoding calculation graph in the decoding stage by using constraint waiting or padding methods, thereby improving the computational efficiency of the processor when performing graph calculations in the decoding stage and avoiding dynamic problems in the matrix calculation process of the decoding stage.

[0184] In some embodiments, the constraint dimensions include feature constraint length and batch constraint number. The feature constraint length is used to constrain the feature length of the content feature representations participating in feature transformation during the pre-filling stage, and the batch constraint number is used to constrain the number of content feature representations decoded simultaneously during the decoding stage. Figure 15 This is a flowchart illustrating a method for performing a model prediction task, provided in another exemplary embodiment of this application. This method can be executed by a terminal, by a server, or by a combination of both. In this embodiment, the method is described using an application to a server as an example. Figure 15 As shown, step 503 above can be implemented as steps 5051 to 5053 below.

[0185] Step 5051: In the pre-filling stage, the feature length of the content feature representation is padded to match the feature constraint length.

[0186] In some embodiments, before performing calculations in the pre-filling phase, the feature length of the content feature representation is padded to match the feature constraint length.

[0187] During the pre-filling stage, the batch size and statement length of the predicted requests sent by the HTTP Server to the Router are uncertain, leading to dynamic issues in the processor's calculations. In this embodiment, the content feature representations of different dialogue statements during the pre-filling stage are padded until the length of the content feature representation matches the feature constraint length. For example, if the feature constraint length is 32 and the length of the input content feature representation is 60, then the content feature representation needs to be padded with zeros to form a sequence of length 64 = 32 × 2.

[0188] In some embodiments, the content feature representations of input content in the same batch are padded, and the content feature representations of input content in the same batch are padded to the same length.

[0189] Prefill calculations generate temporary KCache and Vcache copies, which are then placed into the KVCache. For example... Figure 16 As shown, the KCache and Vcache for the four predicted requests are copied to the corresponding index positions in KVCache1600.

[0190] Because padding occurs during the pre-padding stage, not all input data is valid. During Attention calculation, a mask is used to mark invalid parts to ensure accuracy. Similarly, mask preparation is done on the CPU and transmitted to the processor before Decode calculation. Illustratively, when padding content feature representations, a mask is set for the padding characters. For example, if padding starts from the left of the content feature representation, the character starting to pad with zeros is first padded with the mask symbol "#", and then zeros are padded from the left of the mask symbol. It is worth noting that the above mask marking method is only an illustrative example, and this application does not limit the masking method.

[0191] Step 5052: Transform the feature dimensions of the content feature representations corresponding to at least one input content to obtain the transformed content feature representations.

[0192] The feature dimension transformation of the pre-filling stage is performed on the content feature representations corresponding to multiple input contents to obtain the transformed content feature representations. There is a one-to-one correspondence between the content feature representations corresponding to multiple input contents and the transformed content feature representations. For example, the content feature representation of the i-th input content is transformed to obtain the i-th transformed content feature representation, where i is a positive integer.

[0193] The conversion process of the pre-filling stage has been described in the above embodiments and will not be repeated here.

[0194] Step 5053: If the number of content feature representations decoded simultaneously is less than the number of batch constraints, perform the response prediction process of the decoding stage on the transformed content feature representation corresponding to at least one input content to obtain the response content corresponding to the input content.

[0195] During the decoding phase, the input shape is (batch_size, 1), where batch_size represents the number of prediction requests currently participating in the decoding phase. This batch_size is dynamic and depends on the real-time status of the scheduler's router queue. Furthermore, for the same request, the size of the effective KVCache used in the computation varies across different computation steps. Both of these factors contribute to the dynamic nature of the computation process.

[0196] To avoid the dynamism caused by changes in batch_size, the data input to the decoding computation graph was padded (Pad) to ensure it reached the system's preset batch constraint number. For example... Figure 16 As shown, the batch constraint number of the decoding computation graph in KVCache1600 is 9. If there are only 7 requests waiting for decoding at the current moment, the sequence other than the 7 requests will be padded to make the decoding task sequence in the decoding computation graph reach 9.

[0197] During the computation, an Attention Mask is used to mask invalid computations. First, the KV cache line content corresponding to the extra padding portion of the input is invalid; second, due to padding of the input data, some KVCache cache lines and data along the sequence dimension are invalid. Both of these need to be masked.

[0198] Optionally, the ever-growing KVCache during inference is a source of dynamism, so static graph computation is used. When the Router receives a request, it first checks if there are any free rows in the KVCache that can be allocated. If not, the request continues to queue. If there are free rows, the request is bound to the memory usage rights of that row.

[0199] Once a request's reasoning is complete, the corresponding row in the KVCache does not need to be cleared. We use a mask during the Prefill and Decode phases to mask invalid KVCache data, so even if these locations contain non-zero values, it won't affect subsequent request calculations.

[0200] In summary, the method provided in this embodiment improves the computational efficiency of the processor by using a preset constraint size to perform response prediction on the content feature representations corresponding to multiple input contents in the pre-filling and decoding stages. This eliminates the dynamic problems of processor computation in the pre-filling or decoding stages, enabling the LLM model to be calculated accurately and efficiently on the Gaudi processor, thus improving the prediction efficiency of the LLM model.

[0201] To construct the static graph computation task, additional computational overhead is introduced, and this paper analyzes the additional computational overhead.

[0202] First, it eliminates the dynamism within requests, consistently using the full KVCache for Attention calculation regardless of the effective sequence length, introducing unnecessary computational overhead. For example, assuming max_seq_len = 2048, the effective KVCache during decoding is 512. Then QK... T When performing matrix multiplication, the effective computation ratio is 512 / 2048; softmax(QK) T When performing matrix multiplication, the effective computation ratio is (512 / 2048). 2 Therefore, the effective computation ratio of the Attention computation part in the decoding computation graph is very low.

[0203] As an illustration, setting `max_seq_len` to 8192 and 2048 shows a small difference in performance metrics. However, under some conditions, `max_seq_len` of 8192 outperforms `max_seq_len` of 2048. This is shown in Table 1 below.

[0204] Table 1

[0205]

[0206] Second, to eliminate the dynamic nature between requests, padding is applied to the input content feature representation, introducing unnecessary computational overhead. For example... Figure 16 As shown, the number of valid requests participating in the decoding stage is only 7, but the static graph can only be calculated as 9. All operators' calculations of 2 / 9 are wasted during the decoding process.

[0207] However, this overhead is not significant. For GEMM computation in the Decode computation graph, the inputs are (m, k) × (k, n). Because seq_len = 1 in the decoding stage, the size of the m-dimensional GEMM operation is generally batch_size × 1, and the dimension of BatchGEMM is 1. If the m-dimensional dimension is less than 128, Synapse will implicitly calculate it as 128, and some computation units will be idle. Therefore, when the batch_size is small, the padding overhead is implicitly present, and explicit padding does not increase the overhead. For example, when batch_size = 7, it is actually calculated according to batch_size = 128, and padding it to 9 will not affect the GEMM computation performance.

[0208] Figure 17 This is a structural block diagram of a model prediction task execution apparatus provided in an exemplary embodiment of this application, as shown below. Figure 17 As shown, the device includes:

[0209] The receiving module 1710 is configured to receive at least one prediction request, the prediction request including input content, the prediction request being used to request the execution of a response prediction process through a natural language model to generate response content;

[0210] Generation module 1720 is used to generate a content feature representation of the input content;

[0211] The response prediction process includes a pre-filling stage and a decoding stage; wherein, the pre-filling stage is used to transform the feature dimensions of the content feature representation, and the decoding stage is used to analyze the content feature representation corresponding to at least one of the input contents to obtain the response content;

[0212] The prediction module 1730 is used to perform a response prediction process on the content feature representation corresponding to at least one of the input contents using a preset constraint size, so as to obtain the response contents corresponding to at least one of the input contents respectively.

[0213] Wherein, the constraint size is used to constrain the feature length of the content feature representation participating in feature transformation in the pre-filling stage, or the constraint size is used to constrain the number of at least one input content to be analyzed in batches in the decoding stage.

[0214] In an optional embodiment, the constraint size includes a feature constraint length, which is used to constrain the feature length of the content feature representation that participates in feature transformation during the pre-filling stage;

[0215] like Figure 18 As shown, the prediction module 1730 includes:

[0216] The padding unit 1731 is used to pad the feature length of the content feature representation to match the feature constraint length during the pre-filling stage;

[0217] The conversion unit 1732 is used to convert the feature dimension of the padded content feature representation to obtain the converted content feature representation.

[0218] The prediction unit 1733 is used to perform the response prediction process on the converted content feature representation during the decoding stage to obtain at least one response content corresponding to the input content.

[0219] In an optional embodiment, the padding unit 1731 is further configured to pad with zeros starting from the first side of the content feature representation until the feature length of the content feature representation matches the feature constraint length.

[0220] In an optional embodiment, the padding unit 1731 is further configured to determine the smallest value among the integer multiples of the feature constraint length that is greater than or equal to the feature length of the content feature representation; and pad the content feature representation to the feature length corresponding to the smallest value.

[0221] In an optional embodiment, the padding unit 1731 is further configured to obtain the sequence feature lengths of the content feature representations corresponding to the multiple input contents respectively; and to obtain the first feature length, which is the largest among the multiple sequence feature lengths.

[0222] The padding unit 1731 is further configured to determine the smallest value among the integer multiples of the feature constraint length that is greater than or equal to the first feature length; and to pad the content feature representations corresponding to the plurality of input contents to the feature length corresponding to the smallest value.

[0223] In an optional embodiment, the feature constraint length is a preset value; or,

[0224] The feature constraint length is the feature length corresponding to the historical content feature representation obtained after extracting features from the input content of the historically received prediction requests.

[0225] In an optional embodiment, the constraint size includes a batch constraint number, which is used to constrain the number of content feature representations that are decoded simultaneously during the decoding phase;

[0226] The prediction module 1730 includes:

[0227] The conversion unit 1732 is used to convert at least one feature dimension of the content feature representation corresponding to the input content in the pre-filling stage to obtain the converted content feature representation.

[0228] The prediction unit 1733 is used to perform the decoding stage on at least one converted content feature representation corresponding to the input content when the number of content feature representations decoded simultaneously is less than the number of batch constraints, so as to obtain the response content corresponding to the input content.

[0229] In an optional embodiment, the prediction module 1730 further includes:

[0230] The padding unit 1731 is used to pad the idle sequence when the decoding stage is performed on at least one of the converted content feature representations corresponding to the input content, if the number of content feature representations being decoded simultaneously is less than the batch constraint number, so that the number of content feature representations in the decoding stage reaches the batch constraint number.

[0231] In an optional embodiment, the prediction unit 1733 is further configured to obtain a decoded sequence cached in a data cache, the decoded sequence being the sequence data of the content feature representation stored in the data cache during the decoding process; when the number of decoded sequences is less than the number of batch constraints, the response prediction process of the decoding stage is performed on at least one converted content feature representation corresponding to the input content to obtain the response content corresponding to the input content.

[0232] In an optional embodiment, the prediction unit 1733 is further configured to, when the decoded sequence is cached in the data buffer, obtain a free sequence in the data buffer, wherein the free sequence refers to a sequence not occupied by the decoded sequence under the batch constraint number; and, based on a first sequence number of the free sequence, perform the decoding stage on the transformed content feature representation corresponding to the input content that is less than or equal to the first sequence number to obtain the response content corresponding to the input content.

[0233] In an optional embodiment, the constraint size includes a feature constraint length and a batch constraint number, wherein the feature constraint length is used to constrain the feature length of the content feature representations that participate in the calculation in the pre-filling stage, and the batch constraint number is used to constrain the number of content feature representations that are decoded simultaneously in the decoding stage;

[0234] The prediction module 1730 includes:

[0235] The padding unit 1731 is used to pad the feature length of the content feature representation to match the feature constraint length during the pre-filling stage;

[0236] The conversion unit 1732 is used to convert the feature dimensions of the content feature representations corresponding to the multiple input contents respectively, so as to obtain the converted content feature representations;

[0237] The prediction unit 1733 is used to perform the decoding stage on at least one converted content feature representation corresponding to the input content when the number of content feature representations decoded simultaneously is less than the number of batch constraints, so as to obtain the response content corresponding to the input content.

[0238] In summary, the apparatus provided in this embodiment performs a response prediction stage for the content feature representation of the input content using a preset constraint size in at least one of the pre-filling and decoding stages. Specifically, in the pre-filling stage, the preset constraint size constrains the feature length of the content feature representations participating in feature transformation of the input content. This avoids the need to generate compilation data for uncompiled feature lengths during feature transformation in the pre-filling stage due to differences in feature lengths between different content feature representations. Furthermore, by constraining the feature length of the content feature representations with the constraint size, different feature lengths are avoided during feature transformation in the pre-filling stage. The dimensions are all constrained to the feature length corresponding to the constraint size. Different feature lengths can share the compilation data corresponding to the feature length after the constraint size, eliminating the compilation overhead caused by the dynamic nature of the feature length. In the decoding stage, the number of content feature representations executed in the decoding stage is constrained by the preset constraint size. Since the content feature representations executed in the decoding stage at the same time form the matrix for executing decoding calculations, the number of content feature representations is constrained by the constraint size, which also constrains the shape of the matrix for executing decoding calculations in the decoding stage. This avoids the dynamic change of the matrix shape caused by the increase or decrease of content feature representations during the decoding calculation, and avoids the need for the processor to readjust computing resources for calculations with dynamically changing matrix shapes, thus improving the processor's computing efficiency.

[0239] It should be noted that the model prediction task running apparatus provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the model prediction task running apparatus provided in the above embodiments belongs to the same concept as the model prediction task running method embodiments, and its specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0240] This application also provides a computer device including a processor and a memory. The memory stores at least one instruction, which is loaded and executed by the processor to implement the model prediction task execution method provided in the above-described method embodiments. It should be noted that the computer device may be as follows: Figure 19 The computer equipment provided.

[0241] Please refer to Figure 19This illustration shows a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Specifically, the computer device 1900 includes a Central Processing Unit (CPU) 1901, a system memory 1904 including Random Access Memory (RAM) 1902 and Read-Only Memory (ROM) 1903, and a system bus 1905 connecting the system memory 1904 and the CPU 1901. The computer device 1900 also includes a basic input / output system (I / O system) 1906 that facilitates the transfer of information between various devices within the computer, and a mass storage device 1907 for storing the operating system 1913, application programs 1914, and other program modules 1915.

[0242] The basic input / output system 1906 includes a display 1908 for displaying information and an input device 1909 for user input, such as a mouse or keyboard. Both the display 1908 and the input device 1909 are connected to the central processing unit 1901 via an input / output controller 1910 connected to the system bus 1905. The basic input / output system 1906 may also include the input / output controller 1910 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1910 also provides output to a display screen, printer, or other types of output devices.

[0243] The mass storage device 1907 is connected to the central processing unit 1901 via a mass storage controller (not shown) connected to the system bus 1905. The mass storage device 1907 and its associated computer-readable media provide non-volatile storage for the computer device 1900. That is, the mass storage device 1907 may include computer-readable media (not shown) such as a hard disk or a CD-ROM drive.

[0244] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1904 and mass storage device 1907 described above can be collectively referred to as memory.

[0245] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1901. The one or more programs contain instructions for implementing the above-described model prediction task or the machine translation model-based translation method. The central processing unit 1901 executes the one or more programs to implement the model prediction task or the machine translation model-based translation method provided in the above-described method embodiments.

[0246] According to various embodiments of this application, the computer device 1900 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1900 can be connected to a network 1912 via a network interface unit 1911 connected to the system bus 1905, or the network interface unit 1911 can be used to connect to other types of networks or remote computer systems (not shown).

[0247] The memory further includes one or more programs stored in the memory, and the one or more programs include steps executed by a computer device in the running method for performing the model prediction task provided in the embodiments of this application.

[0248] This application also provides a computer device, which includes a memory and a processor. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads the at least one instruction, at least one program, code set, or instruction set to implement the above-mentioned method for running the model prediction task.

[0249] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the above-described method for running the model prediction task.

[0250] This application also provides a computer program product that, when run on a computer, causes the computer to execute the running method of the model prediction task provided in the above-described method embodiments.

[0251] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into the terminal. The computer-readable storage medium stores at least one instruction, at least one program segment, a code set, or an instruction set. The at least one instruction, the at least one program segment, the code set, or the instruction set is loaded and executed by the processor to implement the above-described model prediction task operation method or the machine translation model-based translation method. Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0252] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0253] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for running a model prediction task, characterized in that, The method includes: Receive at least one prediction request, the prediction request including input content, the prediction request being used to request the execution of a response prediction process through a natural language model to generate response content; Generate a content feature representation of the input content; The response prediction process includes a pre-filling stage and a decoding stage; wherein, the pre-filling stage is used to transform the feature dimensions of the content feature representation, and the decoding stage is used to analyze the content feature representation corresponding to at least one of the input contents to obtain the response content; Using a preset constraint size, a response prediction process is performed on the content feature representation corresponding to at least one of the input contents to obtain response contents corresponding to at least one of the input contents respectively; Wherein, the constraint size is used to constrain the feature length of the content feature representation participating in feature transformation in the pre-filling stage, or the constraint size is used to constrain the number of at least one input content to be analyzed in batches in the decoding stage.

2. The method according to claim 1, characterized in that, The constraint size includes a feature constraint length, which is used to constrain the feature length of the content feature representation that participates in feature transformation during the pre-filling stage. The process involves using a preset constraint size to perform a response prediction process on the content feature representations corresponding to at least one of the input contents, respectively, to obtain response contents corresponding to at least one of the input contents, including: In the pre-filling stage, the feature length of the content feature representation is padded to match the feature constraint length; The feature dimensions of the content feature representation after conversion and padding are used to obtain the converted content feature representation. In the decoding stage, the response prediction process is performed on the converted content feature representation to obtain at least one response content corresponding to each of the input contents.

3. The method according to claim 1 or 2, characterized in that, The step of padding the feature length of the content feature representation to match the feature constraint length includes: Zeros are padded starting from the first side of the content feature representation until the feature length of the content feature representation matches the feature constraint length.

4. The method according to any one of claims 1 to 3, characterized in that, The step of padding the feature length of the content feature representation to match the feature constraint length includes: Among the integer multiples of the feature constraint length, the smallest value that is greater than or equal to the feature length represented by the content feature is determined. The content feature representation is padded to the feature length corresponding to the minimum value.

5. The method according to any one of claims 1 to 4, characterized in that, The step of padding the feature length of the content feature representation to match the feature constraint length includes: Obtain the sequence feature lengths of the content feature representations corresponding to multiple input contents; Obtain the first feature length, which is the largest among multiple sequence feature lengths; Among the integer multiples of the feature constraint length, the smallest value is greater than or equal to the first feature length; The content feature representations corresponding to the multiple input contents are padded to the feature length corresponding to the minimum value.

6. The method according to any one of claims 1 to 5, characterized in that, The length of the feature constraint is a preset value; or, The feature constraint length is the feature length corresponding to the historical content feature representation obtained after extracting features from the input content of the historically received prediction requests.

7. The method according to any one of claims 1 to 6, characterized in that, The constraint size includes the batch constraint number, which is used to constrain the number of content feature representations that are decoded simultaneously in the decoding stage. The process involves using a preset constraint size to perform a response prediction process on the content feature representations corresponding to at least one of the input contents, respectively, to obtain response contents corresponding to at least one of the input contents, including: In the pre-filling stage, at least one feature dimension of the content feature representation corresponding to the input content is transformed to obtain the transformed content feature representation; If the number of content feature representations decoded simultaneously is less than the number of batch constraints, the decoding stage is performed on at least one of the transformed content feature representations corresponding to the input content to obtain the response content corresponding to the input content.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: When performing the decoding stage on the transformed content feature representations corresponding to at least one of the input contents, if the number of content feature representations being decoded simultaneously is less than the batch constraint number, the idle sequence is padded so that the number of content feature representations in the decoding stage reaches the batch constraint number.

9. The method according to any one of claims 1 to 8, characterized in that, When the number of content feature representations decoded simultaneously is less than the batch constraint number, the decoding stage is performed on at least one converted content feature representation corresponding to the input content to obtain the response content corresponding to the input content, including: Obtain the decoded sequence cached in the data cache area, wherein the decoded sequence is the sequence data of the content feature representation stored in the data cache area during the decoding process; When the number of decoded sequences is less than the number of batch constraints, the response prediction process of the decoding stage is performed on at least one of the converted content feature representations corresponding to the input content to obtain the response content corresponding to the input content.

10. The method according to any one of claims 1 to 9, characterized in that, When the number of decoded sequences is less than the number of batch constraints, the response prediction process in the decoding stage is performed on at least one converted content feature representation corresponding to the input content to obtain the response content corresponding to the input content, including: When the decoded sequence is cached in the data buffer, a free sequence in the data buffer is obtained. The free sequence refers to a sequence that is not occupied by the decoded sequence under the batch constraint number. Based on the first sequence number of the idle sequence, the decoding stage is performed on the converted content feature representation corresponding to the input content that is less than or equal to the first sequence number to obtain the response content corresponding to the input content.

11. The method according to any one of claims 1 to 10, characterized in that, The constraint dimensions include feature constraint length and batch constraint quantity. The feature constraint length is used to constrain the feature length of the content feature representations participating in feature transformation in the pre-filling stage, and the batch constraint quantity is used to constrain the number of content feature representations decoded simultaneously in the decoding stage. The process involves using a preset constraint size to perform a response prediction process on the content feature representations corresponding to at least one of the input contents, respectively, to obtain response contents corresponding to at least one of the input contents, including: In the pre-filling stage, the feature length of the content feature representation is padded to match the feature constraint length; Transform the feature dimensions of the content feature representations corresponding to the multiple input contents to obtain the transformed content feature representations; If the number of content feature representations decoded simultaneously is less than the number of batch constraints, the decoding stage is performed on at least one of the transformed content feature representations corresponding to the input content to obtain the response content corresponding to the input content.

12. A device for running a model prediction task, characterized in that, The device includes: A receiving module is configured to receive at least one prediction request, the prediction request including input content, the prediction request being used to request the execution of a response prediction process through a natural language model to generate response content; The generation module is used to generate a content feature representation of the input content; The response prediction process includes a pre-filling stage and a decoding stage; wherein, the pre-filling stage is used to transform the feature dimensions of the content feature representation, and the decoding stage is used to analyze the content feature representation corresponding to at least one of the input contents to obtain the response content; The prediction module is used to perform a response prediction process on the content feature representation corresponding to at least one of the input contents using a preset constraint size, so as to obtain the response contents corresponding to at least one of the input contents respectively; Wherein, the constraint size is used to constrain the feature length of the content feature representation participating in feature transformation in the pre-filling stage, or the constraint size is used to constrain the number of at least one input content to be analyzed in batches in the decoding stage.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, which is loaded and executed by the processor to implement the method of running the model prediction task as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement the method of running the model prediction task as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the method of running the model prediction task as described in any one of claims 1 to 11.

16. A chip, the chip comprising at least one of programmable logic circuitry and program instructions, wherein a computer device having the chip mounted is configured to implement a method for running a model prediction task as described in any one of claims 1 to 11.

Images