Data processing method and device, electronic equipment, and storage medium

Abstract

A data processing method and device, a storage medium and an electronic device, the data processing method comprising: performing inference operation on to-be-processed data by a neural network, the inference operation comprising: performing M rounds of inference on the to-be-processed data in turn by the neural network; the M rounds of inference comprising i-th round of inference and (i+1)-th round of inference executed in turn, wherein the inference process of the i-th round of inference comprises: in a first stage in the inference process of the i-th round of inference, performing inference on N2 input data elements, and in a second stage in the inference process of the i-th round of inference, performing inference on N1 input data elements; wherein the first stage is earlier than the second stage, N2 is greater than N1, N1 input data elements belonging to the same batch of input data are a subset of N2 input data elements; the inference process of the (i+1)-th round of inference comprises: in combination with the output symbols obtained by inference on N2-N1 input data elements that have not been inferred in the i-th round of inference, performing inference on the N2-N1 input data elements.

Description

Technical Field

[0001] Embodiments of this disclosure relate to a data processing method, a data processing apparatus, an electronic device, and a non-transitory computer-readable storage medium. Background Technology

[0002] Artificial neural networks (ANNs), also simply called neural networks, are mathematical models that mimic the behavioral characteristics of animal neural networks to perform distributed parallel information processing. These networks rely on the complexity of the system, adjusting the connections between a large number of internal nodes to achieve information processing. Regardless of the type, artificial neural networks share common characteristics such as massively parallel processing, distributed storage, flexible topology, high redundancy, and nonlinear operations. They possess capabilities in areas such as processing speed, associative ability, adaptability, fault tolerance, and self-organization. These characteristics and capabilities form the technological foundation for artificial neural networks to simulate intelligent activities and have found important applications in various technological fields. For example, artificial neural networks can be used in data compression, data processing, video coding, and signal processing. Summary of the Invention

[0003] At least one embodiment of this disclosure provides a data processing method, including: performing inference operations on data to be processed using a neural network, the inference operation including: using the neural network to sequentially perform M rounds of inference on the data to be processed, wherein the data to be processed includes multiple batches of input data, each batch of input data includes N1 input data elements, and in each round of inference, only one batch of input data is completely inferred, M and N1 are positive integers greater than 1; the M rounds of inference include the i-th round of inference and the (i+1)-th round of inference executed sequentially, i is a positive integer less than M, wherein the inference process of the i-th round of inference includes: a first round of inference in the inference process of the i-th round of inference. In the first stage, reasoning is performed on N2 input data elements. In the second stage of the reasoning process in the i-th round of reasoning, reasoning is performed on N1 input data elements to complete all reasoning of the N1 input data elements belonging to the same batch of input data. The first stage precedes the second stage, N2 is greater than N1, and the N1 input data elements belonging to the same batch of input data are a subset of the N2 input data elements. The reasoning process in the (i+1)-th round of reasoning includes: combining the output symbols obtained from the reasoning of the N2-N1 input data elements that were not completely reasoned in the i-th round of reasoning, and reasoning on the N2-N1 input data elements.

[0004] For example, in a data processing method provided in at least one embodiment of this disclosure, the neural network includes a first sub-network and a second sub-network. The first sub-network is configured to receive input data elements and extract semantic features corresponding to the input data elements using a self-attention mechanism. The semantic features include an initial key tensor and an initial value tensor. The second sub-network is configured to predict the output symbol sequence corresponding to the input data elements one by one based on the semantic features and using an autoregressive mechanism. In the process of predicting the output symbol sequence corresponding to the input data elements one by one using the autoregressive mechanism, the initial key tensor and the initial value tensor are continuously updated according to the predicted output symbols to obtain an updated key tensor and an updated value tensor. The length of the updated key tensor and the updated value tensor is directly proportional to the number of predicted output symbols corresponding to the input data elements.

[0005] For example, in a data processing method provided in at least one embodiment of this disclosure, N1 is determined by the number of symbols in the longest output symbol sequence supported by the neural network, the maximum length of the input data elements supported by the neural network, and the size of the storage space used to store the update key tensor and the update value tensor.

[0006] For example, in a data processing method provided in at least one embodiment of this disclosure, in the first stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N2 input data elements, including: determining X batches of input data as the N2 input data elements, wherein the X batches of input data include the first batch of input data, the first batch of input data includes the N1 input data elements that have been completely reasoned in the i-th round of reasoning, and X is a positive integer greater than 1; and reasoning is performed on the X batches of input data simultaneously in batches.

[0007] For example, in a data processing method provided in at least one embodiment of this disclosure, determining X batches of input data as the N2 input data elements includes: in response to i equal to 1, determining X batches of completely unreasoned input data as the X batches of input data; in response to i greater than 1: determining X-1 batches of input data that were not completely reasoned in the (i-1)th round of reasoning, and a batch of completely unreasoned input data, together as the X batches of input data, wherein the first batch of input data is determined from the X-1 batches of input data that were not completely reasoned in the (i-1)th round of reasoning.

[0008] For example, in a data processing method provided in at least one embodiment of this disclosure, X-1 batches of input data other than the first batch of input data have their own corresponding inference pause thresholds. Inference is performed on the X batches of input data simultaneously on a batch basis, including: inference is performed on the X batches of input data simultaneously, wherein, for any batch of input data in the X-1 batches of input data, the inference of the any batch of input data is paused when the number of output symbols obtained by the inference of the any batch of input data reaches the corresponding inference pause threshold.

[0009] For example, in a data processing method provided in at least one embodiment of this disclosure, in response to X=2, the inference pause threshold is (L-2P) / 3, where P is the maximum length of the input text corresponding to a batch of input data supported by the neural network, L is the sum of the number of characters in the longest output character sequence supported by the neural network and P, and " / " represents division.

[0010] For example, in a data processing method provided in at least one embodiment of this disclosure, when the number of output symbols obtained by reasoning from any batch of input data reaches the corresponding reasoning pause threshold, the reasoning of the any batch of input data is paused, including: pausing the reasoning of the any batch of input data; storing the output symbols obtained by reasoning from the any batch of input data; and clearing the update key tensor and update value tensor generated by the any batch of input data in the i-th round of reasoning.

[0011] For example, in a data processing method provided in at least one embodiment of this disclosure, storing the output symbols that have been inferred from any batch of input data includes: converting each output symbol in the output symbols that have been inferred from any batch of input data into a corresponding symbol index value; and storing the symbol index values ​​corresponding to the output symbols that have been inferred from any batch of input data.

[0012] For example, in a data processing method provided in at least one embodiment of this disclosure, reasoning is performed on the X batches of input data simultaneously, including: in response to i being greater than 1, for the other X-1 batches of input data other than a batch of input data that has not been reasoned at all: combining the output symbols obtained by reasoning the N2-N1 input data elements that have not been reasoned in the (i-1)th round of reasoning, reasoning is performed on the other X-1 batches of input data.

[0013] For example, in a data processing method provided in at least one embodiment of this disclosure, in the second stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N1 input data elements to complete all reasoning of the N1 input data elements belonging to the same batch of input data, including: continuing to reason on the first batch of input data until the reasoning of all input data elements in the first batch of input data is completed, wherein the first batch of input data includes the N1 input data elements that have been completely reasoned in the i-th round of reasoning.

[0014] For example, in a data processing method provided in at least one embodiment of this disclosure, reasoning is performed on the N2-N1 input data elements by combining the output symbols obtained from the reasoning of the N2-N1 input data elements that have not been reasoned in the i-th round of reasoning. This includes: obtaining the output symbols obtained from the reasoning of the N2-N1 input data elements in the i-th round of reasoning, wherein the output symbols obtained from the reasoning of the N2-N1 input data elements are stored in the storage space in the form of symbol index values; and reasoning is performed on the N2-N1 input data elements in batches, wherein for each of the N2-N1 input data elements, the obtained output symbols obtained from the reasoning of the input data element and the input data element are jointly input into the neural network for reasoning.

[0015] For example, in a data processing method provided in at least one embodiment of this disclosure, the input data element is input into the neural network for reasoning along with the output symbol obtained by reasoning from the input data element. This includes concatenating the symbol index value corresponding to the output symbol obtained by reasoning from the input data element with the symbol index value corresponding to the input data element, and inputting the concatenation result into the neural network for reasoning.

[0016] For example, in a data processing method provided in at least one embodiment of this disclosure, the neural network includes a first sub-network for performing cue reasoning and a second sub-network for performing form reasoning. While performing cue reasoning on the N2-N1 input data elements through the first sub-network, cue reasoning is also performed on a batch of completely unreasoned input data through the first sub-network. After the cue reasoning is completed on both the N2-N1 input data elements and the batch of completely unreasoned input data, form reasoning is simultaneously started on the N2-N1 input data elements and the batch of completely unreasoned input data through the second sub-network.

[0017] For example, in a data processing method provided in at least one embodiment of this disclosure, N2 is an integer multiple of N1.

[0018] At least one embodiment of this disclosure provides a data processing apparatus, including: an inference unit configured to perform inference operations on data to be processed via a neural network, wherein the inference operation includes: performing M rounds of inference on the data to be processed sequentially using the neural network, wherein the data to be processed includes multiple batches of input data, each batch of input data includes N1 input data elements, and in each round of inference, only one batch of input data is fully inferred, and M and N1 are positive integers greater than 1; the M rounds of inference include an i-th round of inference and an (i+1)-th round of inference executed sequentially, where i is a positive integer less than M, wherein the inference process of the i-th round of inference includes: in the inference of the i-th round of inference... In the first stage of the process, reasoning is performed on N2 input data elements. In the second stage of the reasoning process in the i-th round of reasoning, reasoning is performed on N1 input data elements to complete all reasoning of the N1 input data elements belonging to the same batch of input data. The first stage precedes the second stage, N2 is greater than N1, and the N1 input data elements belonging to the same batch of input data are a subset of the N2 input data elements. The reasoning process in the (i+1)-th round of reasoning includes: combining the output symbols obtained from the reasoning of the N2-N1 input data elements that were not completely reasoned in the i-th round of reasoning, and reasoning on the N2-N1 input data elements.

[0019] For example, at least one embodiment of this disclosure provides a data processing apparatus that further includes a storage unit configured to store the weight parameters of the neural network and the update key tensor and update value tensor generated during the inference operation.

[0020] At least one embodiment of this disclosure provides an electronic device, including: a memory that non-transitoryly stores computer-executable instructions; and a processor configured to execute the computer-executable instructions, wherein the computer-executable instructions are executed by the processor to implement a data processing method according to any embodiment of this disclosure.

[0021] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the data processing method according to any embodiment of this disclosure. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0023] Figure 1 A schematic structural diagram of a neural network provided for at least one embodiment of this disclosure;

[0024] Figure 2 A flowchart of a reasoning operation is provided as an embodiment of this disclosure;

[0025] Figure 3 A schematic diagram illustrating the execution process of a data processing method provided in an embodiment of this disclosure;

[0026] Figure 4 A schematic block diagram of a data processing apparatus provided in at least one embodiment of this disclosure;

[0027] Figure 5 A schematic block diagram of an electronic device provided in one embodiment of this disclosure;

[0028] Figure 6 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of this disclosure;

[0029] Figure 7 This is a schematic diagram of a hardware environment provided for at least one embodiment of the present disclosure. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0031] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of some known functions and components are omitted.

[0032] The Transformer model is a classic neural network-based text processing model that has spurred the development of important models such as BERT (Bidirectional Encoder Representation from Transformers) and GPT (Generative Pre-trained Transformer), which have greatly advanced the field of NLP (Natural Language Processing).

[0033] Self-attention is a crucial component of Transformer and its evolutionary models. It's a special structure embedded in machine learning models that automatically learns and calculates the contribution of input data to output data. By connecting information from different positions in the input sequence, self-attention incorporates the model's understanding of related words in the context of a given word. Currently, self-attention has become one of the most widely used components in deep learning, especially in natural language processing.

[0034] The self-attention mechanism primarily uses three intermediate buffer tensors: the query tensor, the key tensor, and the value tensor. The query tensor of a token (a linguistic unit in natural language processing, referring to a word or part of a word) is multiplied by the key tensors of other tokens to obtain the attention score of each token relative to the given token. The key tensor acts as a label for each token in the sequence, used to match objects when searching for related tokens. The value tensor is the actual representation of the token; after calculating the attention score, a weighted sum is performed using the value tensor to obtain a tensor representing the current positional context, which is used to predict the output.

[0035] Models like GPT also employ autoregressive mechanisms for inference and training, using deep learning to generate human-like text. After training, the model can predict the next token based on previously predicted tokens. For example, the autoregressive mechanism could include appending the current model's output token to the previously generated token sequence as new input for the next step, or directly using the current model's output token as new input to predict the next token. During inference, the model uses the autoregressive mechanism to predict tokens sequentially, outputting the predicted tokens one by one, ultimately obtaining the output token sequence corresponding to the input.

[0036] In deep learning, data can be inferred in batches to find the optimal balance between memory efficiency and memory capacity. For example, the data to be processed for inference can be divided into multiple batches, each batch size being called the batch size. Each batch of input data includes batch-size input data elements, each of which can be the smallest unit processed by the neural network; for example, an input data element could be a sentence entered by a user. The data in one batch is used to calculate a gradient descent. The loss function required for each parameter update in deep learning is not obtained from a single input data element, but rather from a weighted sum of batch-size input data elements. For example, if the batch size is 32, it means that inference can be performed on 32 input data elements simultaneously, and the network parameters can be updated once based on a weighted sum of the loss values ​​of these 32 input data elements.

[0037] During inference, the model combines the key tensors and value tensors obtained by the self-attention mechanism for computation. For example, the model may include an encoder or other sub-networks to encode the input data elements using the self-attention mechanism, thereby obtaining the semantic features corresponding to the input data elements. For example, the semantic features may include the key tensors and value tensors corresponding to the input data elements belonging to a batch of input data. The key tensors and value tensors are stored in the processor's memory space for subsequent symbol inference. At this time, the key tensors and value tensors are of the same size and are proportional to the product of the maximum length of the input text corresponding to the batch of input data and the batch size.

[0038] During the initial symbol inference, the semantic features include the original key and value tensors, which are at their smallest size throughout the entire inference process. After obtaining the first predicted output symbol, the semantic features are updated, including the updated key and value tensors, which carry information about the first predicted output symbol. This process continues. Therefore, as symbol inference progresses, the size of the key and value tensors used for inference increases progressively with the inference length. In other words, the length of the key and value tensors is directly proportional to the number of predicted output symbols, and their storage space increases accordingly.

[0039] Taking the GPT-3 model as an example, it has 175 billion parameters and an output symbol length of up to 2048, meaning the longest supported output symbol sequence can have 2048 symbols (inference length or inference steps). This model's 175 billion parameters would occupy approximately 350GB of storage space (for FP16 format parameters). For a typical single or dual-node system, assuming a total GPU storage space of 512GB, after deducting the 350GB of weight parameters that need to be stored, approximately 160GB of storage space would remain. During inference, the size of the key tensor and value tensor is proportional to the batch size * (maximum length of the input text corresponding to this batch of input data + number of predicted symbols / number of inference steps performed), thus, as inference progresses, the storage space required for the key tensor and value tensor will increase.

[0040] During inference, a larger batch size is generally desirable to more efficiently utilize the tensor computation operators provided by the processor and improve processor resource utilization. However, for a batch of input data, in order to ensure that the storage space can hold the key and value tensors at the final time step (when all input data inference is complete), the batch size of the model inference needs to match the number of inference steps at the final time step. That is, the product of the batch size and the maximum number of inference steps (the sum of the number of symbols in the longest output symbol sequence supported by the neural network and the maximum length of the input data elements supported by the neural network) must be less than an upper limit. However, in the early stages of inference, the number of inference steps executed is small, and the key and value tensors stored in the storage space have not yet fully occupied all the storage space. At this time, due to the small batch size, the system's computing power cannot be fully utilized, limiting the release of computing power in the overall inference process.

[0041] At least one embodiment of this disclosure provides a data processing method and apparatus, an electronic device, and a non-transient storage medium. The data processing method includes: performing inference operations on data to be processed using a neural network. The inference operation includes: sequentially performing M rounds of inference on the data to be processed using the neural network, wherein the data to be processed includes multiple batches of input data, each batch of input data includes N1 input data elements, and in each round of inference, only one batch of input data is fully inferred, where M and N1 are positive integers greater than 1; the M rounds of inference include the i-th round of inference and the (i+1)-th round of inference executed sequentially, where i is a positive integer less than M, wherein the inference process of the i-th round of inference includes: in the first stage of the inference process of the i-th round of inference, ... Reasoning is performed on N2 input data elements. In the second stage of the reasoning process in the i-th round of reasoning, reasoning is performed on N1 input data elements to complete the reasoning of all N1 input data elements belonging to the same batch of input data. The first stage precedes the second stage, N2 is greater than N1, and the N1 input data elements belonging to the same batch of input data are a subset of the N2 input data elements. The reasoning process in the (i+1)-th round of reasoning includes: combining the output symbols obtained from the reasoning of the N2-N1 input data elements that were not completely reasoned in the i-th round of reasoning, and reasoning on the N2-N1 input data elements.

[0042] The data processing method provided in at least one embodiment of this disclosure introduces a variable batch size inference mode. In the first stage, when the number of inference steps is small and the key tensor and value tensor occupy a small amount of storage space, a larger batch size is used to improve computational efficiency and make full use of the computing power of the operators in the processor. As inference continues (the second stage), the key tensor and value tensor gradually occupy a larger amount of storage space, and the batch size is reduced to complete the inference, thereby accelerating the overall inference process, making full use of the system computing power, and improving inference efficiency and computing power utilization.

[0043] The data processing method provided in this disclosure can be applied to the data processing apparatus provided in this disclosure, which can be configured on an electronic device. The electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a mobile phone, tablet computer, laptop computer, or other hardware device.

[0044] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, but this disclosure is not limited to these specific embodiments.

[0045] For example, at least one embodiment of this disclosure provides a data processing method that includes: performing inference operations on the data to be processed using a neural network.

[0046] For example, the data to be processed could be a dataset that includes all the data that the neural network needs to process.

[0047] Figure 1A schematic structural diagram of a neural network provided for at least one embodiment of this disclosure.

[0048] like Figure 1 As shown, the neural network 100 for performing the data processing method provided in at least one embodiment of the present disclosure includes a first subnetwork 101 and a second subnetwork 102.

[0049] For example, the first sub-network 101 is configured to receive input data elements and extract semantic features corresponding to the input data elements by combining a self-attention mechanism. The semantic features include an initial key tensor (Key stack) and an initial value tensor (Value stack).

[0050] For example, the second sub-network 102 is configured to predict the output token sequence corresponding to each input data element based on semantic features and combined with an autoregressive mechanism.

[0051] For example, the neural network is a GPT network, and accordingly, when using this neural network for reasoning, it includes prompt inference based on the first subnetwork 101 and token inference based on the second subnetwork 102.

[0052] During model inference, initial text (input data element) is provided as a prompt, and the GPT network then continues to generate relevant text based on the prompt. For example, the initial text can be text input by the user, such as questions, requests, emotional expressions, random chat, etc.

[0053] During the prompting reasoning process, the first sub-network 101 is configured to receive input data elements and extract semantic features corresponding to the input data elements using a self-attention mechanism. For example, the semantic features include initial key tensors and initial value tensors as intermediate buffers. For instance, during prompting reasoning, input data elements belonging to the same batch of input data can be input into the first sub-network 101, and after calculation, corresponding initial key tensors and initial value tensors are generated and stored in the processor's storage space for subsequent symbol reasoning. For instance, since reasoning is performed in batches, the lengths of input data elements in the same batch may vary. Before entering the neural network for processing, they can all be padded to the same length, for example, by padding the input data elements with predefined whitespace markers (such as "..."). <pad>This makes the length of the input data element reach the maximum length of the input text corresponding to the batch of input data.

[0054] For example, as mentioned above, for a batch of input data, the initial key tensor and the initial value tensor generated by the first sub-network 101 have the same length and are proportional to the product of the maximum length of the input text corresponding to the batch of input data and the batch size of the batch of input data.

[0055] Subsequently, during the symbol inference process, the second sub-network 102 is configured to predict the output symbol sequence corresponding to each input data element based on the semantic feature and an autoregressive mechanism. For example, the second sub-network 102 is configured to combine the semantic feature with the predicted output symbols to predict the output symbols one by one.

[0056] like Figure 1 As shown, the second sub-network 102 includes a first embedding layer, multiple decoding layers, and a second embedding layer.

[0057] The first embedding layer is configured to convert input symbols into corresponding token indices and encode these indices to obtain the input vector corresponding to the input symbol. For example, a pre-defined mapping table including all possible symbols is used. This table provides the mapping relationship between each user-input symbol and its corresponding token index. The mapping relationship between symbols and their corresponding token indices can be one-to-many or many-to-one, and this mapping rule is determined by the tokenizer. The token index is, for example, a floating-point number or an integer. A one-to-one correspondence exists between the token index and the encoded input vector, determined by the embedding table.

[0058] For example, during the first prediction, the input symbol is a label used to indicate the start of character text, such as " <ss>"or" <bos>Alternatively, the label can also be a label used to indicate that the text content of the previous prediction has ended, such as " <eos>Alternatively, the label may be other labels depending on the task, and this disclosure does not limit this. The goal of the label is to predict what the symbol will be at the next position.

[0059] For example, after predicting the first output symbol, the input symbol is the predicted output symbol. For instance, the first output symbol is used as the input symbol and processed by the second sub-network 102 to obtain the predicted second output symbol; then, the predicted second output symbol is used as the input symbol and processed by the second sub-network 102 to obtain the predicted third output symbol, and so on, until the end label is output, such as "". <eos>"Complete one round of symbol reasoning and obtain the output symbol sequence corresponding to the input data elements.

[0060] For example, multiple decoding layers are cascaded. The first decoding layer is configured to process the semantic features and the input symbol to obtain the processing result; the second decoding layer is configured to process the semantic features and the processing result of the first decoder to obtain the processing result; ...; the j-th decoding layer is configured to process the semantic features and the processing result output by the (j-1)-th decoding layer to obtain the processing result. Here, L is a positive integer greater than 1, and j is a positive integer greater than 1 and less than or equal to Z.

[0061] For example, such as Figure 1 As shown, during the symbol inference process, the second sub-network 102 continuously updates the semantic features based on the output symbols predicted in each inference. Thus, the semantic features used in the symbol inference process will increase accordingly as the inference length increases. The updated semantic features will carry the information of the predicted symbols.

[0062] For example, during the first prediction, the semantic features include the initial key tensor and initial value tensor from the first sub-network 101. After obtaining the output symbols of the first prediction, the semantic features are updated. The updated semantic features include the updated key tensor and updated value tensor, which carry information about the output symbols of the first prediction, and so on. Therefore, the lengths of the updated key tensor and updated value tensor are directly proportional to the number of output symbols corresponding to the predicted input data elements. For a batch of input data, the size of the updated key tensor and updated value tensor is proportional to the batch size of the input data * (maximum length of the input text corresponding to the batch of input data + number of predicted output symbols / number of inference steps performed).

[0063] Of course, it should be noted that, Figure 1 The neural network shown is an illustrative structure. The data processing method provided in this disclosure can be applied to other neural networks, such as those with intermediate buffers for data processing that grow in size as inference proceeds. This disclosure does not impose any specific limitations on these.

[0064] Figure 2 This is a flowchart of a reasoning operation provided in one embodiment of the present disclosure.

[0065] like Figure 2 As shown, the reasoning operation includes step S10, which uses a neural network to perform M rounds of reasoning on the data to be processed.

[0066] For example, the data to be processed (e.g., a dataset) includes multiple batches of input data, each batch of input data including N1 input data elements, that is, the batch size is N1, where M and N1 are positive integers greater than 1.

[0067] For example, as mentioned earlier, the data to be processed for inference can be divided into multiple batches. Each batch of input data includes N1 input data elements. Each input data element can be the smallest unit processed by the neural network. For example, an input data element can be a sentence entered by the user.

[0068] For example, in one round of inference, only one batch of input data is inferred. That is, at the last moment of one round of inference, the inference of a batch of input data, including all input data elements, is completed, resulting in the output symbol sequence corresponding to all input data elements in that batch. Therefore, in this disclosure, one round of inference refers to completing the inference of a batch of input data, and not to the inference of all data to be processed by the neural network. For example, if the data to be processed includes M batches of input data, then the inference of all M batches of input data can be completed through M rounds of inference.

[0069] For example, M rounds of reasoning include the i-th round of reasoning and the (i+1)-th round of reasoning executed sequentially, where i is a positive integer less than M. For example, the (i+1)-th round of reasoning is executed immediately after the i-th round of reasoning.

[0070] like Figure 2 As shown, step S10 may include step S20 describing the reasoning process of the i-th round of reasoning and step S30 describing the reasoning process of the (i+1)-th round of reasoning.

[0071] For example, such as Figure 2 As shown, step S20 may include steps S201 and S202.

[0072] Step S201 may include: in the first stage of the reasoning process in the i-th round of reasoning, reasoning on N2 input data elements.

[0073] Step S202 may include: in the second stage of the reasoning process in the i-th round of reasoning, reasoning is performed on N1 input data elements to complete all reasoning of N1 input data elements belonging to the same batch of input data.

[0074] For example, if the first stage is earlier than the second stage, N2 is greater than N1, and the N1 input data elements belonging to the same batch of input data are a subset of the N2 input data elements.

[0075] Step S30 may include: combining the output symbols obtained from the reasoning of the N2-N1 input data elements that were not completely reasoned in the i-th round of reasoning, and reasoning on the N2-N1 input data elements.

[0076] For example, N1 is determined by the number of symbols in the longest output symbol sequence supported by the neural network and the size of the storage space used to store the update key tensor and update value tensor. As mentioned earlier, the size of the update key tensor and update value tensor is proportional to the batch size * (maximum length of the input text corresponding to the batch of input data + number of predicted symbols / number of inference steps performed). For a batch of input data, in order to ensure that the storage space can hold the update key tensor and update value tensor at the last moment (when all the input data in the batch has been inferred) when the inference of the batch of input data is completed, the batch size of the model inference needs to match the number of inference steps at the last moment. That is, the product of N1 and the number of symbols in the longest output symbol sequence supported by the neural network step_t (for example, for GPT-3, it can be 2048) and the maximum length P of the input text corresponding to the batch of input data must be less than an upper limit, which is determined by the storage space for storing the key tensor and value tensor.

[0077] In this disclosure, in the first stage of an inference round, such as the early to mid-stage of inference, inference is performed on a larger batch size (e.g., N2) of input data, including inference on N1 input data elements belonging to the same batch of input data; in the second stage, such as the late stage of inference, the batch size is reduced to N1, that is, inference is performed on the N1 input data elements in the second stage, until all inference on the N1 input data elements is completed, thus completing one round of inference.

[0078] Therefore, in the first stage, a larger batch size can make matrix multiplication, mapping and other calculations in the neural network more efficient, thus improving computational efficiency. As a result, the computational efficiency is higher than that of inference on only N1 input data elements. The computing power of the processor operators can be fully utilized. Since these calculations account for most of the computation and weight loading in the entire inference process, increasing the computational speed of this part can accelerate the overall inference process, make full use of the system's computing power, and improve inference efficiency and computing power utilization.

[0079] It should be noted that, in this disclosure, variable batch size means that for a neural network, the number of input data elements processed simultaneously is variable, such as processing multiple batches of input data simultaneously, or processing a single batch of input data. A larger batch size refers to performing inference on more input data elements simultaneously in the early stages of inference, rather than referring to the variable batch size of each batch of input data.

[0080] The following description, in conjunction with the accompanying diagram, details the specific reasoning process for each round of reasoning.

[0081] For example, step S201 in the i-th round of inference may include: determining X batches of input data as N2 input data elements; and simultaneously performing inference on the X batches of input data in batches.

[0082] For example, the X batches of input data include the first batch of input data, Batch1, which consists of N1 input data elements that have been completely reasoned out in the i-th round of inference, where X is a positive integer greater than 1. In other words, Batch1 is the only batch of input data that needs to be completely reasoned out in the i-th round of inference; the other X-1 batches of input data will not be completely reasoned out in the i-th round of inference.

[0083] For example, each batch of input data in batch X has the same batch size, which is N1. For example, N2 is an integer multiple of N1, such as N2 = X * N1. Of course, the batch sizes of the input data in batch X can also be different, and this disclosure does not impose specific restrictions on this.

[0084] For example, determining X batches of input data as N2 input data elements can include: in response to i equal to 1, determining X batches of input data that have not been reasoned at all as X batches of input data; in response to i greater than 1: determining X-1 batches of input data that were not reasoned at all in the (i-1)th round of reasoning, and a batch of input data that have not been reasoned at all, together as X batches of input data, wherein the first batch of input data is determined from the X-1 batches of input data that were not reasoned at all in the (i-1)th round of reasoning.

[0085] For example, in the first round of inference, all batches of input data have not yet been inferred. X batches of input data are randomly selected as N2 input data elements for the first stage of inference.

[0086] For example, for M-1 rounds of reasoning other than the first round of reasoning, such as the second round of reasoning, suppose that in the first round of reasoning, input data Batch_1, input data Batch_2 to input data Batch_X (a total of X batches) were selected as N2 batches of input data for reasoning in the first round of reasoning. Among them, input data Batch_1 is the first batch of input data that was completely reasoned in the first round of reasoning. Input data Batch_2 to input data Batch_X that were not completely reasoned in the first round of reasoning, as well as a batch of input data Batch_X+1 that has not been reasoned in any way, are selected as X batches of input data as N2 input data elements used in the second round of reasoning.

[0087] For the first round of reasoning, the first batch of input data that has been completely reasoned in the first round of reasoning can be any batch of input data from the X batches of input data.

[0088] For each of the M-1 rounds of inference other than the first round, the first batch of input data needs to be determined from the X-1 batches of input data that were not fully reasoned in the (i-1)th round. For example, for the second round of inference, any batch of input data from Batch_2 to Batch_X can be selected as the first batch of input data, and it can be fully reasoned in the second round. That is to say, in the second round of inference and subsequent inference processes, the N1 input data elements that are fully reasoned in each round must have undergone at least partial reasoning in the previous round.

[0089] For example, in the X batch of input data, excluding the first batch of input data, the X-1 batch of input data has its own corresponding inference pause threshold. The inference pause threshold is used to indicate that in the i-th round of inference, if the number of output symbols obtained by any batch of input data in the X-1 batch reaches the corresponding inference pause threshold, the inference of that batch of input data is paused.

[0090] For the first batch of input data, there is no corresponding inference pause threshold. That is, in the first stage, the first batch of input data will continue to infer until the other X-1 batches of input data reach their respective inference pause thresholds and are paused, at which point the first stage ends.

[0091] For example, when the number of output symbols obtained from any batch of input data reaches the corresponding inference pause threshold, the inference of that batch of input data can be paused. This can include: pausing the inference of that batch of input data; storing the output symbols obtained from the inference of that batch of input data; and clearing the update key tensor and update value tensor generated by that batch of input data in the i-th round of inference.

[0092] For example, storing the output symbols that have been inferred from any batch of input data may include: converting each output symbol in the output symbols that have been inferred from any batch of input data into a corresponding symbol index value; and storing the symbol index values ​​corresponding to the output symbols that have been inferred from any batch of input data.

[0093] For example, taking the first round of inference as an example, the input data Batch_2 corresponds to the inference pause threshold s2, the input data Batch_3 corresponds to the inference pause threshold s3, ..., the input data Batch_X corresponds to the inference pause threshold sx.

[0094] For example, at the beginning of the first stage of the first round of inference, the input data Batch_1 to Batch_X are inferred simultaneously in batches. That is, in the first stage, the number of input data elements inferred simultaneously is X*N1. When the number of output symbols obtained from the inferred input data Batch_2 reaches s2, further inference on input data Batch_2 is paused. The output symbols obtained from the inferred input data Batch_2 are converted into symbol index values ​​(token indexes) and stored in the storage space. Simultaneously, the update key tensor and update value tensor generated during the inference of input data Batch_2 in the i-th round of inference are cleared. Since the storage space occupied by the token index is extremely small, it will not affect the subsequent inference of input data Batch_1 in the second stage.

[0095] Similarly, when the number of output symbols obtained from the input data Batch_X reaches sx, the inference of the input data Batch_X is paused, the output symbols obtained from the input data Batch_X are converted into symbol index values ​​(token index) and stored in the storage space, and the update key tensor and update value tensor generated in the inference of the input data Batch_X in the i-th round of inference are cleared.

[0096] In the first phase, the first batch of input data Batch_1 continues to infer. The first phase ends when the input data Batch_2 to Batch_X all reach their respective inference pause thresholds and are paused.

[0097] For example, the inference pause thresholds for each batch of input data can be different or the same. For example, if X = 2, the inference pause thresholds for each batch of input data can be the same. For example, if X is greater than 2, the inference pause thresholds for the X batches of input data that are inferred in the same round, excluding the first batch of input data, can be different for the other X-1 batches of input data.

[0098] For example, the inference pause threshold corresponding to each batch of input data is related to the number of symbols in the longest output symbol sequence supported by the neural network, the maximum length of the input text corresponding to a batch of input data supported by the neural network, and the number of input data elements currently being inferred.

[0099] For example, in response to X=2, the inference pause thresholds corresponding to each batch of input data are the same and are all (L-2P) / 3, where P is the maximum length of the input text corresponding to a batch of input data supported by the neural network, L is the sum of the number of characters in the longest output character sequence supported by the neural network and P, and " / " represents division.

[0100] For example, taking the inference pause threshold s2 corresponding to input data Batch_2 as an example, assuming that s2 is the minimum value among all inference pause thresholds, when the number of output symbols obtained by input data Batch_2 reaches s2, the number of output symbols obtained by other X-1 batches of input data such as input data Batch_1 and input data Batch_X also reaches s2. At this time, the update key tensor and update value tensor generated by each batch of input data during the inference process have filled the entire storage space. Therefore, the inference of input data Batch_2 can be paused, the update key tensor and update value tensor generated by input data Batch_2 can be cleared, and the output symbols obtained by input data Batch_2 can be stored in the storage space in the form of symbol index values.

[0101] Afterwards, inference continues on the other X-1 batches of input data. The available storage space created by the update key tensors and update value tensors generated by input data Batch_2 is cleared. This space can then be used to store the update key tensors and update value tensors generated by other X-1 batches of input data during the continued inference process, until the storage space is full again. For example, if the number of output symbols obtained by the other X-1 batches of input data has reached the inference pause threshold s3 corresponding to input data Batch_3 (s3 is greater than s2 but less than other inference pause thresholds), inference on input data Batch_3 is paused. The update key tensors and update value tensors generated by input data Batch_3 during the inference process are cleared. Inference continues on the other X-2 batches of input data (input data other than input data Batch_2 and input data Batch_3). The available storage space created by the update key tensors and update value tensors generated by input data Batch_3 is cleared. This space can then be used to store the update key tensors and update value tensors generated by other X-2 batches of input data during the continued inference process.

[0102] The above process continues until all X-1 batches of input data, except for the first batch, are paused, at which point the first phase ends.

[0103] For example, performing inference on X batches of input data simultaneously may include: in response to i being greater than 1, for the other X-1 batches of input data (excluding one batch of input data that has not been inferred at all) in the X batches of input data: combining the output symbols obtained from the inferred N2-N1 input data elements that were not inferred in the (i-1)th round of inference, performing inference on the other X-1 batches of input data.

[0104] For example, in the first round of inference, the X batch of input data can be inferred according to the conventional neural network inference process, such as referring to... Figure 1 The relevant descriptions are used for prompting reasoning and symbolic reasoning.

[0105] For example, in inference rounds other than the first round, the conventional neural network inference process is also used for a batch of input data that has not been inferred at all in that round, which will not be elaborated here. For the other X-1 batches of input data in the X batches, excluding the batch of input data that has not been inferred at all, these input data have already undergone partial inference in the previous round or several previous rounds, and the inferred output symbols are stored. Therefore, the inference of these X-1 batches of input data can be performed by combining the previously inferred output symbols, thus accelerating the overall inference process.

[0106] For example, in this round of inference, the inferred output symbols and the original input data can be used together as new inputs and fed into the neural network for processing. For instance, after the first sub-network, the resulting semantic features can contain information about the inferred output symbols. When using the second sub-network for symbol inference, we can directly proceed to the last symbol in the inferred output symbols to begin inference. At this point, the lengths of the initial key tensor and initial value tensor obtained through the first sub-network are directly proportional to the sum of the maximum length of the input text corresponding to this batch of input data and the number of inferred output symbols.

[0107] Therefore, in each round of inference, multiple batches of input data are processed, and partial inference is also performed on the other X-1 batches of input data besides the first batch. This results in additional partial inference of the other X-1 batches of input data being executed in each round of inference, thus accelerating the overall inference process. Furthermore, the weight loading time is the same, and since N2 input data elements are inferred simultaneously in the first stage, it has virtually no impact on the inference speed, while increasing the computational efficiency and utilization of the processing core.

[0108] For details on the specific process of reasoning the X-1 batch of input data by combining the previously inferred output symbols, please refer to the relevant description of step S30 in the following text, which will not be repeated here.

[0109] For example, step S202 may include: in the second stage of the reasoning process in the i-th round of reasoning, continuing to reason on the first batch of input data until the reasoning of all input data elements in the first batch of input data is completed. For example, the first batch of input data includes N1 input data elements that have been completely reasoned in the i-th round of reasoning.

[0110] In the first stage, all X-1 batches of input data other than the first batch of input data reach their respective inference pause thresholds and are paused. In the second stage, inference continues on the first batch of input data until all N1 input data elements included in the first batch of input data are inferred, thus completing the i-th round of inference.

[0111] For the (i+1)th round of reasoning, it can also include a first stage and a second stage.

[0112] For example, in the first stage, X batches of input data are first determined as N2 input data elements to be reasoned in the (i+1)th round of inference. Similar to the i-th round of inference, the N2 input data elements can include the N2-N1 input data elements that were not completely reasoned in the i-th round of inference, as well as a batch of input data (N1 input data elements) that have not been reasoned at all. For example, in the (i+1)-th round of inference, N1 input data elements from the N2-N1 input data elements that were not completely reasoned in the i-th round of inference are determined as the first batch of input data that needs to be reasoned in the (i+1)-th round of inference.

[0113] Then, in the first stage, reasoning is performed on N2 input data elements in batches.

[0114] Similar to the i-th round of inference, for a batch of input data that has not been inferred at all among the N2 input data elements, inference can be performed according to the conventional neural network inference process. For the N2-N1 input data elements that have been partially inferred in the i-th round of inference, inference is performed on the N2-N1 input data elements by combining the output symbols obtained from the inference of the N2-N1 input data elements in the i-th round of inference.

[0115] For example, step S30 may include: obtaining the output symbols that have been inferred for the N2-N1 input data elements in the i-th round of inference, wherein the output symbols that have been inferred for the N2-N1 input data elements are stored in the storage space in the form of symbol index values; and performing inference on the N2-N1 input data elements in batches, wherein for each of the N2-N1 input data elements, the obtained output symbols that have been inferred for the input data elements and the input data elements are jointly input into the neural network for inference.

[0116] For example, inputting the obtained input data elements and the inferred output symbol together into the neural network for inference can include: concatenating the symbol index value corresponding to the inferred output symbol of the input data elements and the symbol index value corresponding to the input data elements, and inputting the concatenation result into the neural network for inference.

[0117] For example, taking the second round of inference as an example, the N2-N1 input data elements include input data Batch_2 to Batch_X, which have already undergone partial inference in the first round of inference. For example, for input data Batch_2, s2 steps of inference have been performed in the first round of inference, that is, each input data element in input data Batch_2 has output s2 output symbols, which are stored in the storage space in the form of symbol index values.

[0118] Extract the index values ​​of the s2 output symbols corresponding to each input data element from the storage space. Concatenate the s2 output symbol index values ​​with the index values ​​of the input data elements. Use the concatenated result as a new input data element. Input each new input data element into the neural network for inference in batches.

[0119] For example, after the first sub-network, the semantic features obtained can contain information from the s2-step reasoning that has been performed. When using the second sub-network for symbol reasoning, we can directly proceed to the last symbol in the output symbols that have been reasoned and start reasoning.

[0120] For example, the neural network includes a first sub-network for cue-based reasoning and a second sub-network for form-based reasoning. Cue-based reasoning involves extracting semantic features corresponding to input data elements using the first sub-network, while form-based reasoning involves predicting the output form sequence corresponding to each input data element based on the extracted semantic features and an autoregressive mechanism. The specific processes of cue-based reasoning and form-based reasoning can be found in the aforementioned description of neural networks, and will not be repeated here.

[0121] For example, while providing hints and reasoning for N2-N1 input data elements through the first sub-network, the system also provides hints and reasoning for a batch of completely unreasoned input data through the first sub-network. After providing hints and reasoning for both N2-N1 input data elements and the batch of completely unreasoned input data, the system simultaneously begins to provide form and symbol reasoning for both N2-N1 input data elements and the batch of completely unreasoned input data through the second sub-network.

[0122] For example, taking the second round of reasoning as an example, the new input data elements obtained after the aforementioned process are input into the first sub-network in batches for prompting reasoning, thereby obtaining the semantic features corresponding to each new input data element. At the same time, the first sub-network is used to perform prompting reasoning on a batch of input data that has not been reasoned about at all (e.g., input data Batch_X+1), thereby obtaining the semantic features of this batch of input data.

[0123] After the prompting reasoning for both the N2-N1 input data elements and the input data Batch_X+1 has been completed, the form reasoning for both begins simultaneously. For example, for the N2-N1 input data elements, semantic features containing the reasoned output form information can be used to directly advance the reasoning to the last form in the reasoned output form, thus accelerating the reasoning process.

[0124] For example, for the N2 input data elements processed in the (i+1)th round of inference, the other X-1 batches of input data besides the first batch of input data are paused when the number of output symbols obtained in the (i+1)th round of inference reaches the corresponding inference pause threshold. The specific process is as described in the relevant description of step S201, and will not be repeated here.

[0125] Then, in the second stage, reasoning continues on the first batch of input data in the (i+1)th round of reasoning until all reasoning on the first batch of input data is completed, thus completing the (i+1)th round of reasoning.

[0126] Repeat the above process until all reasoning of the data to be processed is completed.

[0127] For example, if i+1 equals M, in some embodiments, for the Mth round of reasoning, there may be no more completely unreasoned input data to reason. In this case, the first stage may not be included for the Mth round of reasoning. Only step S30 is executed to complete all reasoning of the N2-N1 input data elements that were not reasoned in the ith round of reasoning.

[0128] In the data processing method provided in at least one embodiment of this disclosure, in the early stage of an inference round, such as the first stage, a larger batch size is used, for example, inference is performed on multiple batches of input data simultaneously. For systems with redundant computing power, this can improve the utilization rate of computing power in this part of the inference, thereby improving the overall inference efficiency. In the later stage of an inference round, such as the second stage, the batch size is reduced to N1 to complete the entire inference of a batch of input data. Thus, the system computing power is fully utilized in each inference round, and partial inference of N2-N1 more input data elements is performed, thereby accelerating the overall inference process and improving the system's computational efficiency and computing power utilization efficiency.

[0129] Figure 3 This is a schematic diagram illustrating the execution process of a data processing method provided in an embodiment of the present disclosure.

[0130] For example, in Figure 3 In the example shown, assuming that the batch size * (step_t + P) for each batch of input data must satisfy the upper limit due to storage space, the maximum possible batch size is N1. step_t represents the number of characters in the longest output character sequence supported by the neural network, and P represents the maximum length of the input text corresponding to a batch of input data supported by the neural network. The batch size for each batch of input data is N1, meaning that each round of inference completes the inference of all N1 input data elements.

[0131] For example, in Figure 3 The example shows that all inference for input data Batch_1, Batch_2, Batch_3, ..., Batch_M was completed through M rounds of inference. The batch size of input data Batch_1 to Batch_M is N1, N2 is twice N1, and X = 2. That is, in the first stage, inference is performed on two batches of input data at the same time, and in the second stage, inference is performed on one batch of input data.

[0132] For example, the neural network used in executing this data processing method can be referred to the aforementioned. Figure 1 The structure and reasoning process of neural networks will not be elaborated here.

[0133] exist Figure 3 In the diagram, boxes in the same row represent the reasoning process of a batch (a batch of input data). For example, in the first round of reasoning, the boxes in the top row represent the reasoning process of input data Batch_2, and the boxes in the bottom row represent the reasoning process of input data Batch_1.

[0134] For example, in a batch, the slashed box portion represents the cue reasoning performed by the first subnetwork, the remaining portion represents the symbol reasoning performed by the second subnetwork, and the gray boxes in the remaining portion represent the reasoning that has been performed, while the white boxes represent the portion that has not yet been reasoned.

[0135] For example, for a batch, p represents the maximum length of the input text corresponding to a batch of input data supported by the neural network. For a batch of input data, if the lengths of the various input data elements are different, they are all padded to the maximum length p, as described above. L represents the longest inference step to complete all inference for a batch of input data. This longest inference step is proportional to the length of the updated key tensor and updated value tensor at the last moment of a round of inference. The longest inference step L = p + step_t, where step_t represents the number of symbols in the longest output symbol sequence supported by the neural network (i.e., the longest inference step for symbol inference). s represents the inference pause threshold. Figure 3 In the example, the inference pause thresholds for input data Batch_2 to Batch_M are all s, which means that when the number of output symbols obtained in this round of inference reaches s, the inference of this batch of input data is paused.

[0136] The following is combined Figure 3 This document will specifically describe the execution process of the data processing method provided in at least one embodiment of the present disclosure.

[0137] In the first round of reasoning, such as Figure 3 As shown, in the first stage, two batches of input data, each with a batch size of N1, are selected from the data to be processed and inference begins simultaneously. For example, input data Batch_1 and input data Batch_2 are selected as the N2 input data elements to be processed in the first stage. Among them, input data Batch_1 is selected as the first batch of input data in the first round of inference, that is, in the first round of inference, all inference of input data Batch_1 needs to be completed.

[0138] For example, in the first stage, input data Batch_1 and input data Batch_2 each perform inference. The inference process includes first performing p steps of prompting inference, and then performing form-sign inference based on the semantic features generated by the prompting inference. For details, please refer to the relevant description of the aforementioned neural network.

[0139] During symbol inference, the update key tensor and update value tensor continuously grow, occupying more and more storage space. For example, when the input data Batch_1 and Batch_2 have reached the s-th step (outputting s symbols), the inference pause threshold corresponding to input data Batch_2 is reached. At this point, the update key tensor and update value tensor of input data Batch_1 and Batch_2 have filled the storage space, so the inference of input data Batch_2 is paused. The output symbols that have been inferred from input data Batch_2 are stored, for example, in the form of symbol index values. The update key tensor and update value tensor generated by input data Batch_2 in the first round of inference are cleared, and the first stage ends.

[0140] The space occupied by the index value of the output symbol corresponding to the input data Batch_2 that has been inferred is extremely small and can be ignored. It does not affect the storage of the update key tensor and update value tensor generated by the input data Batch_1 in the second stage of inference.

[0141] In the second stage, such as Figure 3 As shown, inference for input data Batch_2 is paused, while inference for input data Batch_1 continues until all input data elements in Batch_1 have been inferred, thus completing the first round of inference. For example, after completing the first round of inference, the storage space occupied by the update key tensor and update value tensor of input data Batch_1 can be released to perform the second round of inference.

[0142] In the first round of inference, all input data Batch_1 was inferred, and input data Batch_2 was inferred for s steps.

[0143] It should be noted that the first stage and the second stage are two descriptive stages. In reality, the reasoning process does not need to stop in the first stage and then restart in the second stage. For example, for input data Batch_1, its symbol reasoning process is continuous. The relevant operations to pause the reasoning of input data Batch_2 can be completed before the (s+1)th step of input data Batch_1 is executed.

[0144] In the second round of reasoning, such as Figure 3 As shown, in the first stage, a batch of input data Batch_3 with a batch size of N1 that has not been inferred and the input data Batch_2 that was not inferred in the first round of inference are selected from the data to be processed to start inference. For example, input data Batch_2 and input data Batch_3 are selected as the N2 input data elements processed in the first stage, and input data Batch_2 is selected as the first batch of input data in the second round of inference. That is, in the second round of inference, all inference of input data Batch_2 needs to be completed.

[0145] For example, in the first stage, input data Batch_2 and input data Batch_3 each perform inference.

[0146] like Figure 3 As shown, while prompting and reasoning are performed on input data Batch_2 through the first sub-network, prompting and reasoning are also performed on input data Batch_3 through the first sub-network. After prompting and reasoning are completed on both input data Batch_2 and input data Batch_3, form and symbol reasoning is then performed on both input data Batch_2 and input data Batch_3 through the second sub-network simultaneously.

[0147] For the input data Batch_3, the prompting and reasoning process is the same as the normal process, and will not be repeated here.

[0148] For input data Batch_2, it is necessary to combine the reasoning results of the s-step reasoning already performed in the first round of reasoning (the output symbols already obtained) to reason about input data Batch_2. Specifically, as described in step S30, for each input data element in input data Batch_2, the symbol index value corresponding to the output symbol already obtained and the symbol index value corresponding to the input data element are concatenated, and the concatenation result is used as a new input data element. Reasoning is then performed on these new input data elements through the neural network. The hint reasoning process for the new input data elements is the same as the regular hint reasoning process, but the length is (p+s). Thus, the initial key tensor and initial value tensor obtained through the new input data elements contain relevant information about the reasoning results of the s-step reasoning already performed in the first round of reasoning. When performing symbol reasoning on input data Batch_2, it can directly advance to the s-th step of reasoning without repeating the reasoning of the previous s steps performed in the first round of reasoning.

[0149] During the symbol inference process, the update key tensor and update value tensor continuously grow, occupying more and more storage space. For example, when the input data Batch_3 is inferred to the s-th step (outputting s output symbols), the inference pause threshold corresponding to the input data Batch_3 is reached. At this time, the update key tensor and update value tensor of the input data Batch_2 and input data Batch_3 have filled the storage space, so the inference of the input data Batch_3 is paused, and the output symbols that have been inferred by the input data Batch_3 are stored, for example, in the form of symbol index values. The update key tensor and update value tensor generated by the input data Batch_3 in the second round of inference are cleared, and the first stage ends.

[0150] In the second stage, such as Figure 3 As shown, the inference of input data Batch_3 is paused, while the inference of input data Batch_2 continues until the inference of all input data elements in input data Batch_2 is completed, thus completing the second round of inference.

[0151] In the second round of reasoning, all input data Batch_2 was reasoned out, and input data Batch_3 was reasoned out for s steps.

[0152] The reasoning process from the third to the (M-1)th rounds is similar to that of the second round. In each round, a batch of input data that has already completed s-step symbol reasoning in the previous round and a batch of completely new, unreasoned input data are reasoned simultaneously. The batch of input data that has already completed s-step symbol reasoning completes all reasoning in this round, while the other batch of input data that has not been reasoned completes s-step symbol reasoning in this round. The update key tensor and update value tensor of the other batch of input data are released, and the reasoning result of the s-step symbol reasoning performed on the other batch of input data in this round is stored in the form of symbol index value. The specific process will not be described in detail.

[0153] For the Mth round of reasoning, since there is no more input data that has not been reasoned at all, it is only necessary to reason the input data Batch_M that has completed the s-step symbol reasoning in the (M-1)th round of reasoning according to the relevant process of Batch_2 in the second round of reasoning.

[0154] In the above embodiments, after deducting the space occupied by the weight parameters of the neural network itself, the remaining part of the storage space is basically used to place the update key tensor and update value tensor generated by inference. Its volume is proportional to the completed inference portion represented by the shaded or gray box in the figure. Figure 3 As can be seen from this, the completed reasoning portion is the largest in all rounds of reasoning except for the first round and the Mth round.

[0155] For example, taking the second round of reasoning as an example, for a batch of input data with a total length of L that needs to be output, when reasoning to the Sth step of the hint reasoning of input data Batch_3, all the storage space is used up, which is the same as the maximum storage space occupied when a batch of input data is normally reasoned to the Lth step. Therefore, we have 3S+2P=L, that is, the size of S is determined by the formula S=(L-2P) / 3.

[0156] It should be noted that in the above embodiments, the specific execution process of the data processing method provided by at least one embodiment of this disclosure is described using X=2 as an example. However, the execution logic is similar for cases where X takes a larger value, and will not be repeated here.

[0157] In this embodiment, the batch size in the early stage of inference is increased by 2 times, that is, inference is performed on 2*N1 input data elements at the same time. For systems with redundant computing power, this part of the inference can improve the utilization of computing power and the utilization of tensor kernel. Each round of inference executes s more inference steps, thereby improving the overall inference efficiency. In the later stage of inference, the batch size is reduced to N1 so that all inference of a batch of input data can be completed, thereby accelerating the inference.

[0158] At least one embodiment of this disclosure also provides a data processing apparatus. Figure 4 This is a schematic block diagram of a data processing apparatus provided for at least one embodiment of the present disclosure.

[0159] like Figure 4 As shown, the data processing device 200 may include an inference unit 201 and a storage unit 202. It should be noted that... Figure 4 The components and structure of the data processing apparatus 200 shown are merely exemplary and not limiting. The data processing apparatus 200 may also have other components and structures as needed.

[0160] For example, these modules can be implemented through hardware (e.g., circuit) modules, software modules, or any combination of both. The following embodiments are the same and will not be described again. For example, these units can be implemented through a central processing unit (CPU), a data processor (GPU), a tensor processor (TPU), a field-programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities, as well as corresponding computer instructions.

[0161] For example, the data processing device can be a processor that needs to perform inference operations. This could be a general-purpose processor such as a central processing unit (CPU), multi-core processor, graphics processing unit (GPU), general-purpose graphics processing unit (GPGPU), or digital signal processor (DSP), or a dedicated processor such as an AI processor, tensor processor (TPU), or field-programmable gate array (FPGA). Alternatively, the data processing device could be a component specifically designed for optimized computation within GPUs or AI accelerators, such as a deep learning accelerator. Of course, this disclosure is not limited to these examples; the data processing device can be any device that needs to execute data processing methods to perform corresponding inference operations.

[0162] For example, the inference unit 201 is configured to perform inference operations on the data to be processed through a neural network.

[0163] For example, the inference operation includes: using a neural network to perform M rounds of inference on the data to be processed. Here, the data to be processed includes multiple batches of input data, each batch of input data includes N1 input data elements, and in each round of inference, only one batch of input data is inferred completely, where M and N1 are positive integers greater than 1.

[0164] For example, M rounds of reasoning include the i-th round of reasoning and the (i+1)-th round of reasoning, which are executed sequentially, where i is a positive integer less than M.

[0165] For example, the reasoning process of the i-th round of reasoning includes: in the first stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N2 input data elements; in the second stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N1 input data elements to complete all reasoning of N1 input data elements belonging to the same batch of input data; wherein, the first stage is earlier than the second stage, N2 is greater than N1, and the N1 input data elements belonging to the same batch of input data are a subset of the N2 input data elements;

[0166] For example, the reasoning process of the (i+1)th round of reasoning includes: combining the output symbols obtained from the reasoning of the N2-N1 input data elements that were not completely reasoned in the i-th round of reasoning, and reasoning about the N2-N1 input data elements.

[0167] For example, inference unit 201 includes neural network 203, which includes a first subnetwork and a second subnetwork.

[0168] The first sub-network is configured to receive input data elements and extract semantic features corresponding to the input data elements using a self-attention mechanism. The semantic features include an initial key tensor and an initial value tensor.

[0169] The second sub-network is configured to predict the output symbol sequence corresponding to each input data element based on semantic features and combined with an autoregressive mechanism.

[0170] For example, in the process of predicting the output symbol sequence corresponding to each input data element by combining the autoregressive mechanism, the initial key tensor and the initial value tensor are continuously updated according to the predicted output symbols to obtain the updated key tensor and the updated value tensor. The length of the updated key tensor and the updated value tensor are directly proportional to the number of output symbols corresponding to the predicted input data elements.

[0171] It should be noted that the neural network 203 in the inference unit 201 has the same structure and function as the neural network 100 in the above-described data processing method embodiment, and will not be described again here.

[0172] For example, storage unit 202 is configured to store the weight parameters of the neural network and the update key tensor and update value tensor generated during the inference operation. For example, storage unit 202 can be implemented in the form of memory, such as HBM (High Bandwidth Memory), etc. This disclosure does not impose specific limitations on the specific form of the storage unit.

[0173] For example, the inference unit 201 may include code and programs stored in memory; the processor may execute the code and programs to implement some or all of the functions of the inference unit 201 as described above. For example, the inference unit 201 may be a dedicated hardware device used to implement some or all of the functions of the inference unit 201 as described above. For example, the inference unit 201 may be a circuit board or a combination of multiple circuit boards used to implement the functions described above. In the embodiments of this application, the circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processor; and (3) processor-executable firmware stored in memory.

[0174] It should be noted that the inference unit 201 can be used to implement Figure 1 Steps S10-S30 are shown. For a detailed description of the functions that the inference unit 201 can perform, please refer to the relevant descriptions of steps S10 to S30 in the embodiments of the above data processing method; repeated details will not be repeated.

[0175] At least one embodiment of the data processing apparatus provided in this disclosure employs a larger batch size (simultaneously reasoning on more input data elements) to improve computational efficiency during the first stage (when the number of inference steps is small and the key tensors and value tensors occupy less storage space), fully utilizing the computing power of the operators in the processor. As inference continues (i.e., the second stage), the key tensors and value tensors gradually occupy more storage space, reducing the batch size to complete the inference, thereby accelerating the overall inference process, fully utilizing the system's computing power, and improving inference efficiency and computing power utilization. The data processing apparatus 200 can achieve similar technical effects to the aforementioned data processing method, and will not be described in detail here.

[0176] It should be noted that, in the embodiments of this disclosure, the data processing device 200 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited; it can be constructed from analog devices, digital chips, or other suitable methods according to circuit principles.

[0177] Figure 5 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. For example... Figure 5 As shown, the electronic device 300 is, for example, suitable for implementing the data processing method provided in the embodiments of this disclosure. It should be noted that... Figure 5 The components of the electronic device 300 shown are merely exemplary and not limiting. The electronic device 300 may have other components as needed for the actual application.

[0178] like Figure 5 As shown, electronic device 300 may include processing device (e.g., central processing unit, graphics processor, etc.) 301, which can perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in memory to achieve various functions.

[0179] For example, when the computer-readable instructions are executed by the processing device 301, one or more steps of the data processing method according to any of the above embodiments can be performed. It should be noted that a detailed description of the processing procedure of the data processing method can be found in the relevant descriptions in the embodiments of the above data processing method, and repeated descriptions will not be repeated here.

[0180] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory, etc., whereby computer-readable instructions can be loaded from storage device 308 into RAM 303 to execute. Non-volatile memory may include, for example, read-only memory (ROM) 302, hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by the applications, may also be stored in the computer-readable storage medium.

[0181] For example, the processing device 301, ROM 302, and RAM 303 are interconnected via bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0182] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, flash memory, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 5 An electronic device 300 with various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown, and the electronic device 300 may alternatively implement or possess more or fewer devices. For example, a processor 301 can control other components in the electronic device 300 to perform desired functions. The processor 301 may be a device with data processing capabilities and / or program execution capabilities, such as a central processing unit (CPU), a tensor processor (TPU), or a graphics processing unit (GPU). The CPU may be an x86 or ARM architecture, etc. The GPU may be integrated directly onto the motherboard or built into the northbridge chip of the motherboard. The GPU may also be built into the CPU.

[0183] Figure 6 This is a schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. For example, such as Figure 6 As shown, storage medium 400 can be a non-transitory computer-readable storage medium on which one or more computer-readable instructions 401 can be stored non-transitory. For example, when the computer-readable instructions 401 are executed by a processor, the data processing method described above can be performed.

[0184] For example, the storage medium 400 can be used in the aforementioned electronic device, such as the storage medium 400 may include the memory in the electronic device.

[0185] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet computer, a hard disk for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0186] For example, the description of storage medium 400 can be found in the description of memory in the embodiments of the electronic device, and the repeated parts will not be repeated.

[0187] Figure 7 This is a schematic diagram of a hardware environment provided for at least one embodiment of the present disclosure. The electronic device provided in this disclosure can be applied to an Internet system.

[0188] use Figure 7 The computer system provided herein can implement the functions of the data processing apparatus and / or electronic device involved in this disclosure. Such computer systems may include personal computers, laptops, tablets, mobile phones, personal digital assistants, smart glasses, smartwatches, smart rings, smart helmets, and any smart portable or wearable device. A specific system in this embodiment uses a functional block diagram to explain a hardware platform including a user interface. Such a computer device can be a general-purpose computer device or a purpose-specific computer device. Both types of computer devices can be used to implement the data processing apparatus and / or electronic device of this embodiment. The computer system may include any components necessary to implement the information required for the data processing described herein. For example, a computer system can be implemented by a computer device through its hardware, software programs, firmware, and combinations thereof. For convenience, Figure 7 Although only one computer device is shown in the figure, the computer functions related to the information required for data processing described in this embodiment can be implemented in a distributed manner by a group of similar platforms, thus distributing the processing load of the computer system.

[0189] like Figure 7 As shown, the computer system may include a communication port 250 connected to a network for data communication. For example, the computer system can send and receive information and data through the communication port 250, enabling the computer system to exchange data via wireless or wired communication with other electronic devices. The computer system may also include a processor group 220 (i.e., the processor described above) for executing program instructions. The processor group 220 may consist of at least one processor (e.g., a CPU). The computer system may include an internal communication bus 210. The computer system may include different forms of program storage units and data storage units (i.e., the memory or storage media described above), such as a hard disk 270, read-only memory (ROM) 230, and random access memory (RAM) 240, capable of storing various data files used for computer processing and / or communication, as well as possible program instructions executed by the processor group 220. The computer system may also include an input / output component 260 for implementing input / output data flow between the computer system and other components (e.g., user interface 280, etc.).

[0190] Typically, the following devices can be connected to the input / output component 260: input devices such as touch screens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices such as displays (e.g., LCD, OLED displays, etc.), speakers, vibrators, etc.; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication interfaces.

[0191] Although Figure 7 A computer system with various devices is shown, but it should be understood that the computer system is not required to have all the devices shown, and alternatively, the computer system may have more or fewer devices.

[0192] Those skilled in the art will understand that the contents disclosed herein can be varied and modified in many ways. For example, the various devices or components described above can be implemented in hardware, or in software, firmware, or a combination of some or all of the three.

[0193] Furthermore, while this disclosure makes various references to certain elements of systems according to embodiments of this disclosure, any number of different elements may be used and operated on clients and / or servers. Elements are merely illustrative, and different aspects of the system and method may use different elements.

[0194] Flowcharts are used in this disclosure to illustrate the steps of a method according to embodiments of this disclosure. It should be understood that the preceding or following steps are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes.

[0195] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This disclosure is not limited to any particular combination of hardware and software.

[0196] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It should also be understood that terms such as those defined in a common dictionary should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or highly formalized meaning, unless expressly defined herein.

[0197] The foregoing description is intended to illustrate the present disclosure and should not be construed as limiting it. While several exemplary embodiments of the present disclosure have been described, those skilled in the art will readily understand that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure as defined by the claims. It should be understood that the foregoing description is intended to illustrate the present disclosure and should not be construed as limiting it to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present disclosure is defined by the claims and their equivalents.< / eos> < / eos> < / bos> < / ss> < / pad>

Claims

1. A data processing method, comprising: Inference operations are performed on the data to be processed using a neural network, wherein the data to be processed includes data for text processing, the neural network includes a network model for text processing with a self-attention mechanism, and the inference operations include: The neural network is used to perform M rounds of reasoning on the data to be processed. The data to be processed includes multiple batches of input data, each batch of input data includes N1 input data elements. In each round of reasoning, only one batch of input data is completely reasoned. M and N1 are positive integers greater than 1. The M rounds of reasoning include the i-th round of reasoning and the (i+1)-th round of reasoning, executed sequentially, where i is a positive integer less than M. The reasoning process of the i-th round of reasoning includes: In the first stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N2 input data elements. In the second stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N1 input data elements to complete all reasoning of the N1 input data elements belonging to the same batch of input data; Wherein, the first stage is earlier than the second stage, N2 is greater than N1, and the N1 input data elements belonging to the same batch of input data are a subset of the N2 input data elements; The reasoning process of the (i+1)th round of reasoning includes: Based on the output symbols obtained from the inference of the N2-N1 input data elements that were not fully inferred in the i-th round of inference, inference is performed on the N2-N1 input data elements. Wherein, N1 is determined by the number of symbols in the longest output symbol sequence supported by the neural network and the size of the storage space used to store the key tensors and value tensors generated by the inference operation, and N2 is an integer multiple of N1 and is related to the size of the storage space.

2. The data processing method according to claim 1, wherein, The neural network includes a first sub-network and a second sub-network. The first sub-network is configured to receive input data elements and extract semantic features corresponding to the input data elements using a self-attention mechanism. These semantic features include an initial key tensor and an initial value tensor. The second sub-network is configured to predict the output symbol sequence corresponding to each input data element based on the semantic features and using an autoregressive mechanism. In the process of predicting the output symbol sequence corresponding to each input data element by combining the autoregressive mechanism, the initial key tensor and the initial value tensor are continuously updated according to the predicted output symbols to obtain the updated key tensor and the updated value tensor. The length of the updated key tensor and the updated value tensor is directly proportional to the number of output symbols corresponding to the predicted input data elements.

3. The data processing method according to claim 2, wherein, N1 is determined by the number of symbols in the longest output symbol sequence supported by the neural network, the maximum length of the input data elements supported by the neural network, and the size of the storage space used to store the update key tensor and the update value tensor.

4. The data processing method according to claim 1, wherein, In the first stage of the reasoning process of the i-th round of reasoning, reasoning is performed on the N2 input data elements, including: X batches of input data are determined as the N2 input data elements, wherein the X batches of input data include the first batch of input data, and the first batch of input data includes the N1 input data elements that have been completely reasoned in the i-th round of reasoning, and X is a positive integer greater than 1; Inference is performed simultaneously on the X batches of input data, in batches as a whole.

5. The data processing method according to claim 4, wherein, Determining X batches of input data as the N2 input data elements includes: In response to i equaling 1, X batches of completely unreasoned input data are determined as the X batches of input data; In response to i being greater than 1: The X-1 batch of input data that was not fully reasoned in the (i-1)th round of reasoning, and a batch of input data that was not reasoned at all, are determined together as the X batch of input data, wherein the first batch of input data is determined from the X-1 batch of input data that was not fully reasoned in the (i-1)th round of reasoning.

6. The data processing method according to claim 4, wherein, The X-1 batches of input data, excluding the first batch, each have their own corresponding inference pause threshold. Simultaneously performing inference on the X batches of input data on a batch-by-batch basis includes: Inference is performed simultaneously on the X batches of input data. For any batch of input data in the X-1 batches of input data, the inference of the input data is paused when the number of output symbols obtained by the inference of the input data in the given batch reaches the corresponding inference pause threshold.

7. The data processing method according to claim 6, wherein, In response to X=2, the inference pause threshold is (L-2P) / 3, where P is the maximum length of the input text corresponding to a batch of input data supported by the neural network, L is the sum of the number of characters in the longest output character sequence supported by the neural network and P, and " / " represents division.

8. The data processing method according to claim 6, wherein, When the number of output symbols obtained from any batch of input data reaches the corresponding inference pause threshold, the inference of any batch of input data is paused, including: Pause the inference of any batch of input data; Store the output symbols that have been inferred from any batch of input data; Clear the update key tensor and update value tensor generated by any batch of input data during the i-th round of inference.

9. The data processing method according to claim 8, wherein, The storage of output symbols derived from the reasoning of any batch of input data includes: Convert each output symbol in the output symbols obtained by reasoning from any batch of input data into its corresponding symbol index value; Store the symbol index values ​​corresponding to the output symbols obtained by reasoning from any batch of input data.

10. The data processing method according to claim 6, wherein, Simultaneous inference on the X batches of input data includes: In response to i being greater than 1, for the other X-1 batches of input data in the X batches of input data, excluding one batch of input data that has not been reasoned at all: Combining the output symbols obtained from the N2-N1 input data elements that were not completely reasoned in the (i-1)th round of reasoning, reasoning is performed on the other X-1 batches of input data.

11. The data processing method according to any one of claims 1-10, wherein, In the second stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N1 input data elements to complete the reasoning of all N1 input data elements belonging to the same batch of input data, including: The reasoning continues on the first batch of input data until the reasoning of all input data elements in the first batch of input data is completed, wherein the first batch of input data includes the N1 input data elements that have been completely reasoned in the i-th round of reasoning.

12. The data processing method according to any one of claims 1-10, wherein, Combining the output symbols obtained from the inferred N2-N1 input data elements that were not fully inferred in the i-th round of inference, inference is performed on the N2-N1 input data elements, including: Obtain the output symbols that have been reasoned for the N2-N1 input data elements in the i-th round of reasoning, wherein the output symbols that have been reasoned for the N2-N1 input data elements are stored in the storage space in the form of symbol index values; Reasoning is performed on the N2-N1 input data elements in batches. For each of the N2-N1 input data elements, the output symbol obtained from the reasoning of the input data element and the input data element are input together into the neural network for reasoning.

13. The data processing method according to claim 12, wherein, The acquired input data elements, the output symbol obtained through reasoning, and the input data elements are jointly input into the neural network for reasoning, including: The symbol index value corresponding to the output symbol obtained by reasoning of the input data element is concatenated with the symbol index value corresponding to the input data element, and the concatenation result is input into the neural network for reasoning.

14. The data processing method according to claim 12, wherein, The neural network includes a first sub-network for prompting reasoning and a second sub-network for form-based reasoning. While performing suggestive reasoning on the N2-N1 input data elements through the first sub-network, suggestive reasoning is also performed on a batch of completely unreasoned input data through the first sub-network. After the prompting reasoning is completed for the N2-N1 input data elements and the batch of completely unreasoned input data, the form and symbol reasoning of the N2-N1 input data elements and the batch of completely unreasoned input data begins simultaneously through the second sub-network.

15. A data processing apparatus, comprising: The inference unit is configured to perform inference operations on data to be processed via a neural network, wherein the data to be processed includes data for text processing, and the neural network includes a network model for text processing with a self-attention mechanism. The inference operation includes: The neural network is used to perform M rounds of reasoning on the data to be processed. The data to be processed includes multiple batches of input data, each batch of input data includes N1 input data elements. In each round of reasoning, only one batch of input data is completely reasoned. M and N1 are positive integers greater than 1. The M rounds of reasoning include the i-th round of reasoning and the (i+1)-th round of reasoning, executed sequentially, where i is a positive integer less than M. The reasoning process of the i-th round of reasoning includes: In the first stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N2 input data elements. In the second stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N1 input data elements to complete all reasoning of the N1 input data elements belonging to the same batch of input data; Wherein, the first stage is earlier than the second stage, N2 is greater than N1, and the N1 input data elements belonging to the same batch of input data are a subset of the N2 input data elements; The reasoning process of the (i+1)th round of reasoning includes: Based on the output symbols obtained from the inferred N2-N1 input data elements that were not fully reasoned in the i-th round of reasoning, reasoning is performed on the N2-N1 input data elements. Wherein, N1 is determined by the number of symbols in the longest output symbol sequence supported by the neural network and the size of the storage space used to store the key tensors and value tensors generated by the inference operation, and N2 is an integer multiple of N1 and is related to the size of the storage space.

16. The data processing apparatus according to claim 15, further comprising a storage unit, The storage unit is configured to store the weight parameters of the neural network and the update key tensor and update value tensor generated during the inference operation.

17. An electronic device comprising: Memory stores computer-executable instructions non-transiently; The processor is configured to run computer-executable instructions. The computer-executable instructions are executed by the processor to implement the data processing method according to any one of claims 1-14.

18. A non-transitory computer-readable storage medium, wherein, The non-transitory computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the data processing method according to any one of claims 1-14.

Images