Model reasoning method and related equipment

Abstract

The invention provides a model reasoning method and related equipment, and the method comprises the steps: obtaining a reasoning request, the reasoning request is used for executing the reasoning processing of a long-sequence text, the reasoning request comprises the long-sequence text and a reasoning task, and the reasoning task comprises a plurality of reasoning sub-tasks; performing fragmentation processing on the long sequence text to obtain a plurality of fragmented text sets, each fragmented text set in the plurality of fragmented text sets being associated with a reasoning sub-task; according to the incidence relation among the multiple fragment text sets, an inference sub-task set is selected from the multiple inference sub-tasks to execute batch inference processing, and the inference sub-task set comprises two or more inference sub-tasks. According to the method, in a model reasoning scene with a long sequence and multiple reasoning tasks, the reasoning processing speed can be increased, and the model reasoning efficiency is effectively improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a model reasoning method, apparatus, chip system, computing device, computing device cluster, computer-readable storage medium, and computer program product. Background Technology

[0002] With the continuous development of artificial intelligence (AI) technology, especially machine learning (ML) technology, algorithm models have emerged. Algorithm models can include image recognition models, speech recognition models, language models, etc. Different types of algorithm models can be used to handle different tasks. For example, image recognition models can be used to handle image recognition tasks, speech recognition models can be used to handle speech recognition tasks, and language models can be used to handle natural language tasks, etc.

[0003] Specifically, language models possess natural language processing capabilities. By recognizing and analyzing input text, language models can generate output results for reasoning tasks (such as natural language processing tasks), thus completing model reasoning. When the input text is long, i.e., a long sequence of text, the process of using a language model for model reasoning can also be called long sequence reasoning.

[0004] In long-sequence reasoning, language models can have multiple reasoning tasks. For example, when the input text is an article, the reasoning tasks may include summarizing the article and translating it. For these scenarios involving long sequences and multiple reasoning tasks, the limitations of language models in terms of reasoning capabilities make it difficult to achieve fast and efficient model reasoning. Summary of the Invention

[0005] This application provides a model inference method that can improve the speed of inference processing and effectively enhance model inference efficiency in model inference scenarios involving long sequences and multiple inference tasks. This application also provides a corresponding model inference device, chip system, computing device, computing device cluster, computer-readable storage medium, and computer program product.

[0006] Firstly, this application provides a model inference method, which can be executed by a model inference device. The model inference device can be a software device deployed in a computing device cluster. The computing device cluster executes the program code of the software device, thereby executing the model inference method of this application. In some examples, the model inference device can be a hardware device; for example, the model inference device can be a computing device cluster with model inference capabilities. The aforementioned hardware device executes the model inference method of this application when it runs.

[0007] Specifically, the model inference device can first obtain an inference request, which is used to perform inference processing on a long sequence of text. The inference request includes the long sequence of text and an inference task. The inference task includes multiple inference subtasks. Then, the model inference device can perform fragmentation processing on the long sequence of text to obtain multiple fragmented text sets. Each fragmented text set is associated with an inference subtask. Then, the model inference device can select a set of inference subtasks from the multiple inference subtasks to perform batch inference processing based on the association between the multiple fragmented text sets. The set of inference subtasks includes two or more inference subtasks.

[0008] In this method, for long-sequence inference involving multiple inference subtasks, after segmenting the long-sequence text, the set of segmented texts associated with each inference subtask is determined. Then, based on the relationships between multiple sets of segmented texts, a set of inference subtasks is selected for batch inference processing. In model inference scenarios with long sequences and multiple inference tasks, on the one hand, the inference subtasks are processed using the associated sets of segmented texts, eliminating the need to process all segmented texts one by one. On the other hand, a batch and parallel inference processing approach is adopted, thus improving the speed of inference processing and effectively enhancing the model inference efficiency.

[0009] In some possible implementations, multiple inference subtasks include a first subtask and a second subtask, and multiple fragmented text sets include a first fragmented text set and a second fragmented text set. The first fragmented text set is associated with the first subtask, and the second fragmented text set is associated with the second subtask. When the fragmented texts included in the first and second fragmented text sets are different, the model inference device can select the first and second subtasks to perform batch inference processing. In this method, when the fragmented text sets associated with the two subtasks are different, computational resources are utilized efficiently to improve the parallel processing efficiency of model inference.

[0010] In some possible implementations, the model inference device can perform segmentation processing on the long sequence of text to obtain multiple segmented texts of the long sequence of text. Then, the model inference device can determine the first set of segmented texts associated with the first inference subtask from the multiple segmented texts based on the degree of matching between the multiple segmented texts and the first inference subtask, wherein the first inference subtask is one of the multiple inference subtasks.

[0011] This method selects a set of text segments that match the inference subtask from multiple text segments of a long sequence, based on the degree of matching between the inference subtask and the text segments. In subsequent inference processing, it eliminates the need to process each text segment individually. Because the text segments in the set match the inference subtask, computational resources are saved and the inference speed is improved without reducing the model's inference accuracy.

[0012] In some possible implementations, the model inference device can calculate the confidence probabilities of multiple fragmented texts for the target character unit to be generated in the first inference subtask, and determine the first fragmented text set associated with the first inference subtask based on the fragmented texts whose confidence probabilities meet the set conditions.

[0013] In this method, referring to the text length expansion method based on Naive Bayes, the fragmented text is selected as the fragmented text set associated with the inference subtask according to the confidence probability. In subsequent inference processing, the fragmented text set can be used to generate character units with higher accuracy, thereby improving the inference accuracy of the model.

[0014] In some possible implementations, the inference process includes multiple rounds of inference, each round generating a character unit. Batch inference processing is executed by a chip system deployed on a computing device. The model inference apparatus can determine the key-value information required by the inference subtask set in the current round of inference from the computing device's storage unit. Then, it transfers the key-value information to the chip system's storage unit and, based on the key-value information, executes the current round of inference to generate the character unit for the inference subtask set in the current round of inference.

[0015] This method stores reusable key-value information in the storage units of the computing device, achieving key-value cache reuse without occupying storage space in the chip system. This improves model inference speed and reduces the storage pressure on the chip system. When dealing with long sequence inference, there is no need to worry about insufficient storage space in the chip system. Applying KV Cache technology to long sequence inference improves the inference performance of the language model for long sequence inference.

[0016] Secondly, this application provides a model inference apparatus, which includes modules for performing the model inference method in the first aspect or any possible implementation of the first aspect.

[0017] Thirdly, this application provides a chip system including a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to execute the model inference method as described in the first aspect or any implementation thereof.

[0018] Fourthly, this application provides a computing device, the computing device including a chip system, the chip system including a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to execute the model inference method as described in the first aspect or any implementation thereof.

[0019] Fifthly, this application provides a computing device cluster, the computing device cluster including at least one computing device, the at least one computing device including a chip system, the chip system including a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to execute the model inference method as described in the first aspect or any implementation thereof.

[0020] In a sixth aspect, this application provides a computer-readable storage medium including computer-readable instructions for implementing the model reasoning method as described in the first aspect or any implementation thereof.

[0021] In a seventh aspect, this application provides a computer program product comprising computer-readable instructions for implementing the model reasoning method as described in the first aspect or any implementation thereof.

[0022] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0023] Figure 1 A schematic diagram illustrating the reasoning process of a language model provided in this application during a round of reasoning;

[0024] Figure 2 A schematic diagram of an encoder provided in this application;

[0025] Figure 3 A schematic diagram of the structure of a decoder provided in this application;

[0026] Figure 4 A schematic diagram of the architecture of a model inference device provided in this application;

[0027] Figure 5 A schematic diagram of the architecture of a model inference device provided in this application;

[0028] Figure 6 A flowchart illustrating a model reasoning method provided in this application;

[0029] Figure 7A flowchart illustrating a long sequence text processing method provided in this application;

[0030] Figure 8 A flowchart illustrating a text processing method provided in this application;

[0031] Figure 9 This application provides a flowchart illustrating a batch inference process.

[0032] Figure 10 A schematic diagram for recording the execution status of inference subtasks provided in this application;

[0033] Figure 11 A schematic diagram of the structure of a computing device provided in this application;

[0034] Figure 12 This application provides a schematic diagram of the structure of a computing device cluster;

[0035] Figure 13 This is a schematic diagram of another computing device cluster provided in this application. Detailed Implementation

[0036] First, let's introduce some of the technical terms used in this application.

[0037] Machine learning (ML) is a core area of ​​artificial intelligence (AI). It enables computing devices to learn from data and make intelligent decisions. Specifically, machine learning relies on algorithmic models for data processing and computation. From the perspective of model function, algorithmic models can be categorized into regression models, classification models, clustering models, reinforcement learning models, etc. From the perspective of reasoning tasks, they can be categorized into image recognition models, speech recognition models, language models, etc. Image recognition models are used to handle image recognition tasks, speech recognition models are used to handle speech recognition tasks, and language models are used to handle natural language tasks.

[0038] AI chips, also known as AI accelerators or AI computing cards, can be understood as modules used to process AI-related computations. AI chips can perform AI-related computations based on different architectures. For example, AI chips can perform AI-related computations based on general-purpose chips such as graphics processing units (GPUs), data processing units (DPUs), and neural processing units (NPUs). Alternatively, AI chips can perform AI-related computations based on application-specific integrated circuits (ASICs). Still another example is the use of field-programmable gate arrays (FPGAs).

[0039] Language models possess natural language processing (NLP) capabilities. Specifically, language models have the ability to understand and generate natural language. Applying language models to different areas can accomplish various natural language tasks. For example, language models can perform natural language tasks such as text generation, summarization, translation, sentiment analysis, and text assignment.

[0040] The following describes the specific process of inference processing in a language model. After inference processing, the language model outputs a text-formatted result. Language models are typically generative models, and the inference process for a given task can be divided into multiple rounds. Each round generates a character unit (token) in the output result, and the new token depends on the input text and the tokens generated in previous rounds. That is, in the first round of inference, the language model generates the first token based on the input text. In the Nth round of inference, the language model generates the Nth token based on the input text and the N-1 tokens generated in the previous N-1 rounds. Here, N is an integer greater than 1. After completing all rounds of inference, the tokens generated in each round constitute the output result of the language model.

[0041] Language models are typically implemented using a Transformer architecture, which consists of an encoder and a decoder. See also Figure 1The diagram illustrates the reasoning process of a language model in a single round of inference. First, the input text is tokenized to obtain multiple tokens. Next, these tokens are embedded to obtain a vector matrix. This vector matrix is ​​then input into an encoder to obtain a first encoded information matrix. The vector matrix and the first encoded information matrix are input into a decoder to obtain a second encoded information matrix. Finally, the second encoded information matrix is ​​processed to determine the tokens generated in the current round of inference. These tokens are then embedded and added to the vector matrix of the input text, updating the vector matrix for use in the next round of inference.

[0042] In practice, multiple tokens from the input text undergo embedding processing to obtain a word embedding vector and a position vector for each token. The word embedding vector can be understood as a vectorized representation of the token across multiple dimensions (e.g., part-of-speech, semantic, morphological dimensions), while the position vector is the vectorized representation of the token's location within the input text. The word embedding vector and position vector for each token are summed to obtain the vectorized representation of each token. These vectorized representations of the multiple tokens in the input text form the vector matrix obtained after embedding processing.

[0043] The encoder is implemented based on a self-attention mechanism, see [link to documentation]. Figure 2 The diagram shows the structure of an encoder, which includes a self-attention layer and a feedforward neural network layer. A vector matrix is ​​input into the self-attention layer, such that the vector matrix is ​​correlated with three weight matrices (W). Q Matrix, W K Matrix and W VThe query vector matrix (Q matrix), key vector matrix (K matrix), and value vector matrix (V matrix) are multiplied together. Each row vector in the Q matrix is ​​then multiplied by the transpose of the K matrix to obtain a self-attention score vector for each token. These self-attention score vectors represent the degree of association between different tokens, and they together form the self-attention score matrix. Next, the output is calculated using the self-attention score matrix and the V matrix. Multiplying the self-attention score matrix by the V matrix yields the self-attention vector matrix, which represents the value vector of each token under its self-attention score. This self-attention vector matrix is ​​then input into a feedforward neural network layer to obtain the first encoded information matrix. In some examples, the language model may include multiple encoders, with the output of one encoder serving as the input to the next, and the output of the last encoder becoming the first encoded information matrix.

[0044] Similar to the encoder, the decoder is also implemented based on a self-attention mechanism; see [link to documentation]. Figure 3 The diagram illustrates the structure of a decoder, which includes a first self-attention layer, a second self-attention layer, and a feedforward neural network layer. The input to the first self-attention layer is a masked vector matrix. Through processing similarly to the self-attention layers in the encoder, a first self-attention vector matrix is ​​obtained. The input to the second self-attention layer includes the first self-attention vector matrix and the first encoded information matrix output by the encoder. In the second self-attention layer, the K and V matrices are generated based on the first encoded information matrix. In some examples, the language model may include multiple decoders. For the first decoder, the Q matrix in the second self-attention layer is generated based on the vector matrix; for decoders other than the first, the Q matrix is ​​generated based on the output of the previous decoder. Then, through processing similarly to the self-attention layers in the encoder, the second self-attention vector matrix is ​​obtained, which is then processed by the feedforward neural network to obtain the second encoded information matrix.

[0045] After obtaining the second encoded information matrix, the second encoded information matrix is ​​input into the softmax layer to obtain the probability matrix of multiple tokens to be generated. Then, based on the probability of multiple tokens to be generated, the token generated in the current round of reasoning is determined, and the reasoning of the current round is completed.

[0046] The aforementioned language models based on the transformer architecture, such as bidirectional encoder representations from transformers (BERT) and generative pre-trained transformer (GPT), are pre-trained using text data and fine-tuned for specific reasoning tasks, which can effectively improve their performance in reasoning for various natural language tasks.

[0047] Long sequence reasoning refers to reasoning processing for long sequences of text. Long sequence text can be understood as text composed of a large number of words and characters. For example, when the input text is an article, the reasoning task of reading comprehension using a language model can be long sequence reasoning. Similarly, when the input text is a news article, the reasoning task of summarizing using a language model can be long sequence reasoning.

[0048] In long-sequence reasoning, language models can perform multiple reasoning tasks. In other words, users can leverage a language model to perform multiple different reasoning processes on the same long-sequence text. For example, when the input text is an article, the reasoning task could include summarizing the article and translating it. As another example, when the input text is a news article, the reasoning task could include generating a summary of the news and adjusting its tone.

[0049] In long sequence inference, considering that long text sequences may exceed the text length limit that a language model can handle, text segmentation can be used. Specifically, when the input text is a long text sequence, it is segmented into multiple text segments, and then the language model is used to process each segment one by one to obtain the inference result, thus achieving long sequence inference.

[0050] In multi-reasoning tasks, sequential reasoning can be used. Specifically, when there are multiple reasoning tasks, the reasoning processes of multiple tasks are executed sequentially according to a certain order to obtain the reasoning results of multiple tasks. For example, when there are two reasoning tasks, the reasoning process of reasoning task 1 can be executed first, according to the order of reasoning tasks indicated by the user, to obtain the reasoning result of reasoning task 1, and then the reasoning process of reasoning task 2 can be executed to obtain the reasoning result of reasoning task 2.

[0051] However, due to the considerable length of long text sequences, when using text segmentation for long sequence inference, the language model needs to process multiple text segments one by one, resulting in lengthy inference times and difficulty in quickly obtaining inference results. This is especially true in long sequence inference involving multiple inference tasks, where the language model needs to process multiple text segments one by one for each inference task, leading to low computational efficiency and hindering the achievement of fast and efficient model inference.

[0052] In view of this, this application provides a model inference method. Specifically, firstly, an inference request is obtained, wherein the inference request is used to perform inference processing on a long sequence of text. The inference request includes the long sequence of text and an inference task, wherein the inference task includes multiple inference subtasks. Next, the long sequence of text is segmented to obtain multiple sets of segmented texts, wherein each set of segmented texts is associated with an inference subtask. Based on the association between the multiple sets of segmented texts, a set of inference subtasks is selected from the multiple inference subtasks to perform batch inference processing, wherein the set of inference subtasks includes two or more inference subtasks.

[0053] In this method, for long-sequence inference involving multiple inference subtasks, after segmenting the long-sequence text, the set of segmented texts associated with each inference subtask is determined. Then, based on the relationships between multiple sets of segmented texts, a set of inference subtasks is selected for batch inference processing. In model inference scenarios with long sequences and multiple inference tasks, on the one hand, the inference subtasks are processed using the associated sets of segmented texts, eliminating the need to process all segmented texts one by one. On the other hand, a batch and parallel inference processing approach is adopted, thus improving the speed of inference processing and effectively enhancing the model inference efficiency.

[0054] To make the technical solution of this application clearer and easier to understand, the system architecture of this application will be described below with reference to the accompanying drawings.

[0055] See Figure 4The schematic diagram of the model inference device shows that the model inference device 40 processes the computation of the language model 41 during the model inference stage. The language model 41 can be a generative model. In some embodiments, the language model 41 can be deployed locally, for example, on a server in a local data center, where a local data center refers to a data center directly controlled by the user. Alternatively, the language model 41 can be deployed on a local mobile device or embedded system. In other embodiments, the language model 41 can also be deployed in the cloud, for example, on a cloud server, where the cloud server can be a central server in a central cloud computing cluster or an edge server in an edge cloud computing cluster. Furthermore, the language model 41 can be provided as a cloud service; this application does not impose any limitations on this.

[0056] The model inference device 40 includes an acquisition module 401, a segmentation module 402, and an inference module 403. Specifically, the acquisition module 401 is used to acquire an inference request, which can be understood as a model inference request when calling the language model 41 to perform natural language task processing. In this application, the inference request is used to perform inference processing on a long sequence of text. The inference request includes the long sequence of text and an inference task, and the inference task includes multiple inference subtasks. In other words, when calling the language model 41 to perform natural language task processing, the input text to the language model 41 may include a long sequence of text and descriptive text indicating multiple inference subtasks.

[0057] For example, when the input text is an article and "Help me summarize and translate this article," the long sequence of text is the article, and the description text indicating multiple inference subtasks is "Help me summarize and translate this article." In this case, the long sequence of text in the inference request is the article, and the inference tasks are summarizing the article and translating the article.

[0058] The sharding module 402 performs sharding processing on long sequence text to obtain multiple sets of sharded text. Each of these multiple sets of sharded text is associated with a reasoning subtask. In other words, after performing sharding processing on the long sequence text, the sharding module 402 determines the set of sharded text associated with each reasoning subtask, thus obtaining multiple sets of sharded text. For example, when the reasoning task includes n reasoning subtasks, the multiple sets of sharded text may include sharded text set 1 associated with reasoning subtask 1, sharded text set 2 associated with reasoning subtask 2, ..., sharded text set n associated with reasoning subtask n.

[0059] The inference module 402 is used to select a set of inference subtasks from multiple inference subtasks for batch inference processing based on the relationships between multiple sets of fragmented text. The set of inference subtasks includes two or more inference subtasks. In other words, the inference module 402 performs batch inference processing on at least two inference subtasks based on the relationships between fragmented text sets associated with different inference subtasks, thus improving the model's inference efficiency.

[0060] In some possible implementations, the inference process may include multiple rounds of inference, each round generating a character unit (token). This batch inference process can be executed by a chip system, which may consist of AI chips and be deployed on a computing device. In other words, the model inference device 40 can be deployed on the chip system, for example, on the AI ​​chip within the chip system.

[0061] As discussed earlier regarding inference processing in language models, each round of inference generates a token. The new token depends on the input text and tokens generated in previous rounds. In the first round of inference, the vector matrix of the input text needs to be compared with W... K Matrix and W V Matrix multiplication yields a key vector matrix (K matrix) and a value vector matrix (V matrix). In subsequent inference rounds, the updated vector matrix, composed of the input text and the generated tokens, needs to be multiplied by W. K Matrix and W V Matrix multiplication yields a key vector matrix (K matrix) and a value vector matrix (V matrix). It is evident that in multiple rounds of reasoning, duplicate key-value information exists in the key vector matrix (K matrix) and the value vector matrix (V matrix) and can be reused.

[0062] When performing inference processing using chip systems, the industry typically employs key-value cache (KV cache) technology to optimize inference performance. In KV cache technology, key-value information (i.e., the key vector matrix and key vector matrix generated in each round of inference) is stored in the chip system's memory. This allows the chip system to reuse the key-value information from the previous round to generate new tokens in subsequent rounds of inference, without repeatedly calculating the key-value information of the input text or the key-value information of already generated tokens. By trading space for time, KV cache technology can reduce redundant computation and improve inference speed without sacrificing computational accuracy, thereby enhancing inference performance. However, in inference processing of long text sequences, due to the large dimensionality of the key vector matrix and key vector matrix, storing the key-value information in the chip system's memory can lead to exhaustion of the chip's on-chip storage space, making it difficult to apply KV cache technology to long-sequence inference.

[0063] This application employs a key-value information migration method, which to some extent addresses the limitation of insufficient storage space in chip systems, preventing the application of KV cache technology. See also Figure 5 The diagram illustrates the architecture of a model inference device. The model inference device 40 is deployed on a chip system 42, which is deployed on a computing device 43. The chip system 42 further includes a storage unit 421, and the computing device 43 further includes a storage unit 431. Both the storage unit 421 in the chip system 42 and the storage unit 431 in the computing device 43 can be used to store data (e.g., intermediate result data generated during inference processing of long-sequence text). In some embodiments, the storage unit 421 in the chip system 42 can be on-chip memory, and the storage unit 431 in the computing device 43 can be memory. Typically, the storage space of the storage unit 421 is smaller than that of the storage unit 431.

[0064] Specifically, the model inference device 40 (e.g., inference module 402) can determine the key-value information required by the inference subtask set in the current round of inference from the storage unit 431 of the computing device 43, then transfer the key-value information to the storage unit 421 of the chip system 42, and execute the current round of inference based on the key-value information to generate the character unit of the inference subtask set in the current round of inference.

[0065] In other words, in this application, the key-value information generated during each round of inference is stored in the storage unit 431 of the computing device 43, and during the current round of inference, the key-value information is transferred from the storage unit 431 of the computing device 43 to the storage unit 421 of the chip system 42. Thus, the model inference device 40 (e.g., inference module 402) can use the key-value information in the storage unit 421 of the chip system 42 to generate the token for the current round of inference, effectively solving the problem of insufficient storage space in the storage unit 421 of the chip system 42, which makes it difficult to apply KV Cache technology to long-sequence inference.

[0066] based on Figure 4 The present application also provides a model reasoning method in addition to the model reasoning apparatus 40 shown. The model reasoning method of this application will be described below with reference to embodiments.

[0067] See Figure 6 The flowchart shown illustrates the model inference method, which can be executed by the model inference device 40. The method includes the following steps:

[0068] S601: Model inference device 40 obtains an inference request.

[0069] The inference request is used to perform inference processing on long sequences of text. In other words, in this application, the inference request can be understood as a request to invoke the language model 41 to perform long sequence inference. The language model 41 can be a machine learning model with natural language processing capabilities, capable of performing different types of natural language tasks. In some embodiments, the language model 41 can be a large language model (LLM). The LLM is based on a transformer architecture, trained on text data, and has the ability to understand and generate natural language. Typically, the language model 41 is a generative model, capable of generating character units through autoregression.

[0070] This application does not limit the method of obtaining inference requests. In some embodiments, the model inference device 40 may provide an interactive page, which may include a graphical user interface (GUI) or a command user interface (CUI). Users can interact with the language model 41 through the interactive page, and the model inference device 40 can obtain inference requests in response to text input operations triggered by the user on the interactive page. In other embodiments, the model inference device 40 may also provide a calling interface for the language model 41, such as an application programming interface (API), and the model inference device 40 can obtain inference requests in response to user or other service calls to the language model 41.

[0071] The inference request includes a long sequence of text and an inference task. The long sequence of text and the inference task can be text content representing natural language content that the language model 41 can recognize and analyze. In other words, the inference request includes two parts: one part is the text content to be processed, and the other part is the processing requirements for that text content.

[0072] In this application, long sequence text can be understood as text with a length greater than a first preset threshold, or text with a number of character units greater than a second preset threshold. The first and second preset thresholds can be determined based on the reasoning ability of the language model 41. For example, long sequence text can be an article, a report, a novel, a technical manual, etc. In some embodiments, long sequence text can be directly input in text format; in other embodiments, long sequence text can also be input in other formats (i.e., non-natural language content), such as document format, portable document format (PDF), etc., and obtained by format conversion.

[0073] Inference tasks typically refer to natural language tasks, such as text generation, text understanding, text translation, and text question answering. Understandably, language model 41 possesses natural language processing capabilities and can handle different natural language tasks. However, if the inference request only includes long sequences of text, language model 41 struggles to perform model inference that meets the user's processing needs for such long sequences. Therefore, by including an inference task in the inference request, language model 41 can perform model inference for a specific inference task, generating output text that meets the user's processing requirements. In this application, the inference task includes multiple inference sub-tasks. That is, there are multiple natural language tasks for long sequences of text.

[0074] In some embodiments, the inference task can be descriptive text indicating a natural language task. For example, the inference task could be "Help me extract the summary of this article" or "Help me translate this novel," etc., where inputting descriptive text indicates a natural language task for a long sequence of text. In other embodiments, the inference task can also be in other forms (i.e., non-natural language content). For example, the inference task can be provided in the form of a control (such as a translation control, a summary control, etc.), where triggering the control indicates a natural language task for a long sequence of text.

[0075] S602: The model inference device 40 performs segmentation processing on long sequence texts to obtain multiple sets of segmented texts.

[0076] In this application, each of the multiple fragmented text sets is associated with a reasoning subtask. The fragmented text set associated with a reasoning subtask can be understood as the fragmented text required to process the reasoning subtask; in other words, each reasoning subtask is associated with a fragmented text set, and each reasoning subtask uses the fragmented text in its associated fragmented text set for reasoning processing.

[0077] In specific implementation, the model inference device 40 can perform segmentation processing on the long sequence text to obtain multiple segmented texts. Then, based on the matching degree between the multiple segmented texts and the first inference subtask, it determines the first set of segmented texts associated with the first inference subtask from the multiple segmented texts. Here, the first inference subtask is one of the multiple inference subtasks.

[0078] Considering the considerable length of long-sequence texts, the model inference device 40 can first perform text segmentation. See also Figure 7 The diagram illustrates a flowchart of a long-sequence text processing method. The model inference device 40 can divide the long-sequence text into multiple fragments, such as fragment 1 to fragment n, where n is an integer greater than 1. Typically, the text content of the multiple fragments does not overlap, thus avoiding repeated inference processing for the same text content.

[0079] This application does not impose any restrictions on the segmentation method for long text sequences. For example, the model inference device 40 can divide a long text sequence into multiple text segments using a text segmentation algorithm. As another example, the model inference device 40 can divide a long text sequence into multiple text segments based on a fixed text length. Yet another example, the model inference device 40 can divide a long text sequence into multiple text segments based on a fixed number of paragraphs.

[0080] Furthermore, this application can perform model inference based on prompting learning technology. Specifically, prompting learning guides the language model 41 to complete specific inference tasks by designing prompts, thus eliminating the need for retraining or fine-tuning the language model 41.

[0081] In this application, the reasoning task in the reasoning request (e.g., reasoning subtask 1 to reasoning subtask m, where m is an integer greater than 1) can serve as a prompt. By combining multiple text segments with the reasoning task serving as the prompt, the model's reasoning is achieved based on the prompt's cues during subsequent reasoning processing of long text sequences. Furthermore, the prompt may also include content indicating the output format, content indicating sentiment or tone, etc., and this application does not impose any limitations on this.

[0082] After obtaining multiple text segments, the model inference device 40 can determine the set of text segments associated with each inference subtask for each inference subtask. In other words, the model inference device 40 can determine the text segments that match each inference subtask from the multiple text segments, obtain the set of text segments associated with each inference subtask, and thus obtain multiple sets of text segments.

[0083] For example, multiple text fragments include text fragment 1 to text fragment n, the reasoning task includes reasoning subtask 1 to reasoning subtask m, the set of text fragments associated with reasoning subtask 1 includes text fragment 1 and text fragment 2, the set of text fragments associated with reasoning subtask 2 includes text fragment 3 and text fragment 4, the set of text fragments associated with reasoning subtask 3 includes text fragment 2, text fragment 5 and text fragment 6, ..., the set of text fragments associated with reasoning subtask m includes text fragment 1 and text fragment n.

[0084] Thus, by assessing the matching degree between the inference subtask and the fragmented text, a set of fragmented texts matching the inference subtask is selected from multiple fragmented texts of the long sequence text for different inference subtasks. In subsequent inference processing, the model inference device 40 can utilize the set of fragmented texts associated with the inference subtask for model inference, eliminating the need to process each fragmented text individually. Because the fragmented texts in the set match the inference subtask, computational resources are saved and the model inference speed is improved without reducing the model inference accuracy.

[0085] Among some possible implementations, the Naive Bayes-based context extension (NBCE) method can be referenced to determine the set of fragmented texts associated with the inference subtask. The NBCE method, based on the Naive Bayes principle, is introduced first. It performs independent inference processing on each fragmented text, and then determines the output based on the inference results of the fragmented texts. See also... Figure 8 The diagram illustrates a text processing method where concatenated text segments 1 to n with the prompt are input into language model 41. Simultaneously, text containing only the prompt is also input into language model 41, allowing language model 41 to perform independent reasoning processing on each input. In practice... Figure 8 The content can be represented by the following formula:

[0086] log p(T|c1,c2,c3,...,c n )=(β+1)P[(log p(T|c1), log p(T|c2),..., log p(T|c n ))]-βlog p(T) (1)

[0087] In equation (1), T is the target character unit to be generated, which can be understood as the theoretical estimate generated by language model 41 based on its understanding of the reasoning task. i It is the i-th text segment, log p(T|c i ) is the confidence probability of the i-th text segment for the target character unit, that is, the probability that language model 41 can generate the target character unit using the i-th text segment. log p(T) is the confidence probability of the prompt for the target character unit, which can be used as a benchmark to reduce the illusion phenomenon of language model 41 to a certain extent. P[(log p(T|c1), log p(T|c2), ..., log p(T|c2)] n The result is the pooling of confidence probabilities for target character units across multiple text segments. β is an adjustable hyperparameter; different β values ​​can be chosen for different inference tasks. log p(T|c1, c2, c3, ..., c n ) is the overall probability of language model 41 generating the target character unit using multiple text segments.

[0088] In this application, referring to the NBCE method, the model inference device 40 can calculate the confidence probabilities of multiple fragmented texts for the target character unit to be generated in the first inference subtask. Then, the model inference device 40 can determine the first fragmented text set associated with the first inference subtask based on the fragmented texts whose confidence probabilities meet the set conditions.

[0089] In some embodiments, the text fragments whose confidence probabilities meet the set conditions can be the text fragments with the highest confidence probabilities. That is, the pooling process in equation (1) can be max pooling. In this way, by selecting the text fragments with the highest confidence probabilities as the set of text fragments associated with the inference subtask, the set of text fragments can be used to generate character units with higher accuracy in subsequent inference processing, thereby improving the accuracy of model inference.

[0090] Different inference subtasks can have different set conditions. For example, a text segment whose confidence probability meets the set conditions can also be a text segment whose confidence probability is greater than the confidence threshold. In this case, the set of text segments can be at least one text segment whose confidence probability is greater than the confidence threshold. This application does not impose any restrictions on this.

[0091] It should be noted that this application may also refer to other text processing methods related to long sequence reasoning to determine the set of fragmented text associated with the reasoning subtask, such as the product of continuous weights (PCW) method, etc., and this application does not limit it in this regard.

[0092] S603: The model inference device 40 selects a set of inference subtasks from multiple inference subtasks to perform batch inference processing based on the relationship between multiple fragmented text sets.

[0093] The inference subtask set includes two or more inference subtasks. In other words, the model inference device 40 can perform batch and parallel inference processing on at least two inference subtasks based on the association between the fragmented text sets associated with different inference subtasks.

[0094] In this application, the association between multiple fragmented text sets can be understood as the degree of overlap between them. Specifically, multiple inference subtasks may include a first subtask and a second subtask, and multiple fragmented text sets may include a first fragmented text set and a second fragmented text set. The first fragmented text set is associated with the first subtask, and the second fragmented text set is associated with the second subtask. When the fragmented texts included in the first fragmented text set and the second fragmented text set are different, the model inference device 40 can select the first subtask and the second subtask to perform batch inference processing.

[0095] For example, if the first subtask is associated with a first set of fragmented texts including fragmented text 1 and fragmented text 2, and the second subtask is associated with a second set of fragmented texts including fragmented text 3, then there are no identical fragmented texts between the first and second sets of fragmented texts. The model inference device 40 can then perform batch, parallel inference processing on both the first and second subtasks. This effectively utilizes computing resources and improves the parallel processing efficiency of model inference.

[0096] In some possible implementations, the inference process may include multiple rounds of inference, each round generating a character unit (token). A token can be understood as a basic unit of text; a token can be a letter, a word, a phrase, etc. In other words, one round of inference generates one basic unit of text, and after all rounds of inference are completed, the generated tokens constitute the output text of the model's inference.

[0097] In this application, batch inference processing can be executed by chip system 42, which can be deployed on computing device 43. That is, model inference device 40 can be deployed on chip system 42, utilizing the computing resources of chip system 42 to perform batch inference processing. Specifically, for each round of inference, model inference device 40 can determine the key-value information required by the inference subtask set in the current round of inference from storage unit 431 of computing device 43. Then, model inference device 40 can transfer the key-value information to storage unit 421 of chip system 42, and based on the key-value information, execute the current round of inference, generating character units for the inference subtask set in the current round of inference.

[0098] Here, key-value information refers to the vector matrix obtained through embedding and W in the language model 41 based on the transformer architecture. K Matrix and W V After matrix multiplication, the resulting matrix consists of a key vector matrix and a value vector matrix. The key-value information required in the current inference round can be understood as the key-value information generated in the previous inference round that can be reused in the current inference round. Typically, the key-value information required in the current inference round includes a vector matrix composed of the input text (e.g., fragmented text from a set of fragmented texts associated with the inference subtask) and the generated tokens. K Matrix and W V The key vector matrix and value vector matrix obtained after matrix multiplication.

[0099] Unlike traditional key-value cache technology, which stores reusable key-value information in the storage unit 421 of the chip system 42, this application stores the key-value information generated during each round of inference in the storage unit 431 of the computing device 43. In subsequent rounds of inference, the key-value information is migrated from the storage unit 431 of the computing device 42 to the storage unit 421 of the chip system 42. This achieves key-value cache reuse without occupying storage space in the chip system 42, improving model inference speed and reducing storage pressure on the chip system 42. When dealing with long sequence inference, there is no need to worry about insufficient storage space in the chip system 42. Applying key-value cache technology to long sequence inference improves the inference performance of the language model for long sequence inference.

[0100] In some possible implementations, inference processing can be divided into a full inference process and an incremental inference process. The full inference process can be understood as the inference process in the first round, while the incremental inference process can be understood as the inference process in subsequent rounds. In the full inference process, the model inference device 40 can generate the first token of the inference subtask based on all the fragmented text of the long sequence, improving the accuracy of the first token generation. In the incremental inference process, the model inference device 40 generates subsequent tokens for the inference subtask based on the set of fragmented text associated with the inference subtask, improving the inference efficiency for long sequence inference.

[0101] by Figure 9 The following is an example of a flowchart illustrating batch inference processing, where c ij This refers to inference processing performed on the i-th text segment and the j-th subtask. At the first moment, chip system 10 can migrate the key-value information of text segments 1 to n to the storage unit 421 of chip system 42. Model inference device 40 determines that the first text segment set associated with the first subtask includes text segment n, and uses the key-value information of text segment n to generate the token of the first subtask. At the second moment, model inference device 40 determines that the second text segment set associated with the second subtask includes text segment 3. At the third moment, model inference device 40 uses the key-value information of text segment n to continue generating the token of the first subtask, and simultaneously uses the key-value information of text segment 3 to generate the token of the second subtask, batch processing the first and second subtasks.

[0102] If the second fragmented text set associated with the second subtask also includes fragmented text n, the model inference device 40 can process the second subtask using fragmented text n after completing the entire inference process of the first subtask, i.e., obtaining the output text of the first subtask. While waiting for the inference processing of the first subtask, the model inference device 40 can continue to perform batch inference processing for other subtasks (such as the third subtask and the fourth subtask) in the manner described above. In this way, different subtasks are not processed sequentially, but are processed in batches and in parallel through the association relationship between multiple fragmented text sets, achieving out-of-order processing. On the one hand, this can improve the parallel processing efficiency of the model inference device 40, and on the other hand, it can reduce the latency caused by the switching of key-value information between the chip system 42 and the computing device 43.

[0103] In some possible implementations, the model inference device 40 can also utilize a record table to record the execution status of subtasks. For example... Figure 10 As shown, the row fields in the record table represent different subtasks, and the column fields represent different text segments. If a subtask is completed, it can be marked in the corresponding table. For example, if the set of text segments associated with the first subtask includes text segment 1, after the first subtask is completed, "1" can be entered in the first row and first column of the table to indicate that the first subtask has been executed, and the text segment for which the first subtask was executed is text segment 1. In this way, the record table clearly and intuitively records the execution status of subtasks.

[0104] As described above, this application achieves batch inference processing of inference subtasks by determining the sets of fragmented text associated with different inference subtasks and based on the relationships between these sets. This effectively improves model inference efficiency for models with long sequences and multiple inference tasks.

[0105] Based on the model reasoning method described above, this application also provides a model reasoning device 40 as described above. The model reasoning device 40 will be described below with reference to the accompanying drawings.

[0106] See Figure 4 The schematic diagram of the model inference device 40 shown above includes:

[0107] The acquisition module 401 is used to acquire inference requests. The inference requests are used to perform inference processing on long sequence texts. The inference requests include long sequence texts and inference tasks. The inference tasks include multiple inference subtasks.

[0108] The slicing module 402 is used to perform slicing processing on long sequence text to obtain multiple slicing text sets, wherein each slicing text set is associated with an inference subtask.

[0109] The inference module 403 is used to select a set of inference subtasks from multiple inference subtasks to perform batch inference processing based on the relationship between multiple fragmented text sets. The set of inference subtasks includes two or more inference subtasks.

[0110] The acquisition module 401, the sharding module 402, and the inference module 403 can all be implemented in software or in hardware. For example, the implementation of the acquisition module 401 will be described below. Similarly, the implementation of the sharding module 402 and the inference module 403 can refer to the implementation of the acquisition module 401.

[0111] As an example of a software functional unit, module 401 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, module 401 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0112] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0113] As an example of a hardware functional unit, the acquisition module 401 may include at least one computing device, such as a server. Alternatively, the acquisition module 401 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0114] The multiple computing devices included in the acquisition module 401 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition module 401 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition module 401 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0115] It should be noted that, in other embodiments, the acquisition module 401 can be used to execute any step in the model inference method, the fragmentation module 402 can be used to execute any step in the model inference method, and the steps implemented by the acquisition module 401, the fragmentation module 402, and the inference module 403 can be specified as needed. By implementing different steps in the model inference method through the acquisition module 401, the fragmentation module 402, and the inference module 403, all functions of the model inference device can be realized.

[0116] In some possible implementations, multiple inference subtasks include a first subtask and a second subtask, and multiple fragmented text sets include a first fragmented text set and a second fragmented text set. The first fragmented text set is associated with the first subtask, and the second fragmented text set is associated with the second subtask. The inference module 403 is specifically used to: when the fragmented texts included in the first fragmented text set and the second fragmented text set are different, select the first subtask and the second subtask to perform batch inference processing.

[0117] In some possible implementations, the slicing module 402 is specifically used to: perform slicing processing on the long sequence text to obtain multiple slicing texts of the long sequence text; and determine the first set of slicing texts associated with the first inference subtask from the multiple slicing texts based on the degree of matching between the multiple slicing texts and the first inference subtask, wherein the first inference subtask is one of the multiple inference subtasks.

[0118] In some possible implementations, the slicing module 402 is specifically used to: calculate the confidence probabilities of multiple sliced ​​texts for the target character unit to be generated in the first inference subtask; and determine the first sliced ​​text set associated with the first inference subtask based on the sliced ​​texts whose confidence probabilities meet the set conditions.

[0119] In some possible implementations, the inference process includes multiple rounds of inference, each round generating a character unit; the model inference device 40 is deployed on a chip system, which is deployed on a computing device; the inference module 403 is specifically used to: determine the key-value information required by the inference subtask set in the current round of inference from the storage unit of the computing device; migrate the key-value information to the storage unit of the chip system; and execute the current round of inference based on the key-value information to generate the character unit of the inference subtask set in the current round of inference.

[0120] This application also provides a chip system including a processor and a power supply circuit. The power supply circuit supplies power to the processor, which executes the operation steps corresponding to the model inference method. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices such as a DPU, NPU, XPU, SoC, offload card, or accelerator card.

[0121] This application also provides a computing device 1100. For example... Figure 11 As shown, the computing device 1100 includes a bus 1102, a processor 1104, a memory 1106, and a communication interface 1108. The processor 1104, the memory 1106, and the communication interface 1108 communicate with each other via the bus 1102. The computing device 1100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1100.

[0122] Bus 1102 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The Unified Bus is also known as the Lingqu Bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 11 The bus 1102 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1102 may include a path for transmitting information between various components of the computing device 1100 (e.g., memory 1106, processor 1104, communication interface 1108).

[0123] The processor 1104 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0124] Memory 1106 may include volatile memory, such as random access memory (RAM). Memory 1106 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 1106 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0125] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 1106, or two or more types of storage media can be configured to realize the function of memory 1106. This application does not limit this.

[0126] The memory 1106 stores executable program code, which the processor 1104 executes to implement the functions of the aforementioned acquisition module 401, slicing module 402, and inference module 403, thereby realizing the model inference method. That is, the memory 1106 stores instructions for executing the model inference method.

[0127] The communication interface 1108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1100 and other devices or communication networks.

[0128] As one possible implementation, the computing device 1100 may also include a chip system, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, and the processor executes the operation steps corresponding to the model inference method. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices or AI chips such as a DPU, NPU, XPU, SoC, offloading card, or accelerator card.

[0129] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0130] like Figure 12 As shown, the computing device cluster includes at least one computing device 1100. The memory 1106 of one or more computing devices 1100 in the computing device cluster may store the same instructions for executing model inference methods.

[0131] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing the model inference method. In other words, a combination of one or more computing devices 1100 can jointly execute the instructions for executing the model inference method.

[0132] It should be noted that the memory 1106 in different computing devices 1100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the model inference device. That is, the instructions stored in the memory 1106 of different computing devices 1100 can implement the functions of one or more modules among the acquisition module 401, the sharding module 402, and the inference module 403.

[0133] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 13 One possible implementation is shown. For example... Figure 13 As shown, the two computing devices 1100A and 1100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1106 in computing device 1100A stores instructions for executing the functions of the acquisition module 401. Meanwhile, the memory 106 in computing device 1100B stores instructions for executing the functions of the sharding module 402 and the inference module 403.

[0134] Figure 13 The connection method between the computing device clusters shown can be that, considering that the model inference method provided in this application needs to obtain inference requests and perform model inference for inference requests, the functions implemented by the sharding module 402 and the inference module 403 are to be executed by the computing device 1100B.

[0135] It should be understood that Figure 13 The functions of computing device 1100A shown can also be performed by multiple computing devices 1100. Similarly, the functions of computing device 1100B can also be performed by multiple computing devices 1100.

[0136] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute a model inference method.

[0137] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a model inference method, or instruct the computing device to perform a model inference method.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model reasoning method, characterized in that, The method includes: Obtain an inference request, which is used to perform inference processing on a long sequence of text. The inference request includes the long sequence of text and an inference task, and the inference task includes multiple inference subtasks. The long sequence of text is segmented to obtain multiple sets of segmented text, wherein each set of segmented text is associated with an inference subtask. Based on the relationships between the multiple fragmented text sets, a set of inference subtasks is selected from the multiple inference subtasks to perform batch inference processing, wherein the set of inference subtasks includes two or more inference subtasks.

2. The method according to claim 1, characterized in that, The plurality of reasoning subtasks include a first subtask and a second subtask, and the plurality of fragmented text sets include a first fragmented text set and a second fragmented text set. The first fragmented text set is associated with the first subtask, and the second fragmented text set is associated with the second subtask. The step of selecting a set of inference subtasks for batch inference processing based on the relationships between the multiple sets of fragmented text includes: When the fragmented texts included in the first fragmented text set and the second fragmented text set are different, the first subtask and the second subtask are selected for batch inference processing.

3. The method according to claim 1 or 2, characterized in that, The process of segmenting the long text sequence to obtain multiple sets of segmented text includes: The long sequence of text is segmented to obtain multiple segments of the long sequence of text; Based on the matching degree between the multiple text segments and the first inference subtask, a first set of text segments associated with the first inference subtask is determined from the multiple text segments, wherein the first inference subtask is one of the multiple inference subtasks.

4. The method according to claim 3, characterized in that, The step of determining the first set of fragmented texts associated with the first inference subtask from the plurality of fragmented texts based on the degree of matching between the plurality of fragmented texts and the first inference subtask includes: Calculate the confidence probability of each of the multiple text segments for the target character unit to be generated in the first inference subtask; Based on the text segments whose confidence probabilities meet the set conditions, determine the first set of text segments associated with the first inference subtask.

5. The method according to any one of claims 1 to 4, characterized in that, The inference process includes multiple rounds of inference, each round of inference being used to generate a character unit; the batch inference process is executed by a chip system deployed on a computing device. The step of selecting a set of inference subtasks from the plurality of inference subtasks to perform batch inference processing includes: Determine the key-value information required by the inference subtask set for the current round of inference from the storage unit of the computing device; The key value information is migrated to the storage unit of the chip system; Based on the key-value information, the current round of reasoning is executed, and character units of the reasoning subtask set in the current round of reasoning are generated.

6. A model reasoning device, characterized in that, The device includes: The acquisition module is used to acquire inference requests, which are used to perform inference processing on long sequence texts. The inference request includes long sequence texts and inference tasks, and the inference task includes multiple inference subtasks. The sharding module is used to perform sharding processing on the long sequence text to obtain multiple sharded text sets, wherein each of the multiple sharded text sets is associated with an inference subtask. The inference module is used to select a set of inference subtasks from the multiple inference subtasks to perform batch inference processing based on the correlation between the multiple sets of fragmented text. The set of inference subtasks includes two or more inference subtasks.

7. The apparatus according to claim 6, characterized in that, The plurality of inference subtasks include a first subtask and a second subtask; the plurality of fragmented text sets include a first fragmented text set and a second fragmented text set; the first fragmented text set is associated with the first subtask, and the second fragmented text set is associated with the second subtask; the inference module is specifically used for: When the fragmented texts included in the first fragmented text set and the second fragmented text set are different, the first subtask and the second subtask are selected for batch inference processing.

8. The apparatus according to claim 6 or 7, characterized in that, The sharding module is specifically used for: The long sequence of text is segmented to obtain multiple segments of the long sequence of text; Based on the matching degree between the multiple text segments and the first inference subtask, a first set of text segments associated with the first inference subtask is determined from the multiple text segments, wherein the first inference subtask is one of the multiple inference subtasks.

9. The apparatus according to claim 8, characterized in that, The sharding module is specifically used for: Calculate the confidence probability of each of the multiple text segments for the target character unit to be generated in the first inference subtask; Based on the text segments whose confidence probabilities meet the set conditions, determine the first set of text segments associated with the first inference subtask.

10. The apparatus according to any one of claims 6 to 9, characterized in that, The inference process includes multiple rounds of inference, each round generating a character unit; the device is deployed on a chip system, which is deployed on a computing device; the inference module is specifically used for: Determine the key-value information required by the inference subtask set for the current round of inference from the storage unit of the computing device; The key value information is migrated to the storage unit of the chip system; Based on the key-value information, the current round of reasoning is executed, and character units of the reasoning subtask set in the current round of reasoning are generated.

11. A chip system, characterized in that, The chip system includes a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to execute the method of any one of claims 1 to 5.

12. A computing device, characterized in that, The computing device includes a chip system, the chip system including a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to execute the method of any one of claims 1 to 5.

13. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, the at least one computing device includes a chip system, the chip system includes a processor and a power supply circuit, the power supply circuit is used to supply power to the processor, and the processor is used to execute the method of any one of claims 1 to 5.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer-readable instructions for implementing the method according to any one of claims 1 to 5.

15. A computer program product, characterized in that, The computer program product includes computer-readable instructions for implementing the method according to any one of claims 1 to 5.

Images