Data processing method and apparatus, electronic device and storage medium

By adopting a batch size variable inference mode in neural networks, the problems of small inference steps and low computing power utilization and large storage space occupancy in the prior art are solved, and a more efficient data processing process is achieved.

WO2024230342A1PCT designated stage expired Publication Date: 2024-11-14SHANGHAI BIREN TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/083793
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-10
Filing Date
2024-03-26
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

During the data processing process, existing neural networks cannot fully utilize the processor's computing power due to small inference steps. As the number of inference steps increases, the storage space is too large, resulting in low computing efficiency and computing power utilization.

Method used

The inference model with variable batch size is adopted, and larger batch sizes are used when the inference steps are small to improve the calculation efficiency. As the inference steps increase, the batch size is gradually reduced to accelerate the overall inference process and make full use of the system's computing power.

Benefits of technology

The computing efficiency and computing power utilization rate of data processing are improved. By fully utilizing the processor computing power in the early stage of inference, the storage space is effectively managed in the later stage, and the overall inference efficiency is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024083793_14112024_PF_FP_ABST
    Figure CN2024083793_14112024_PF_FP_ABST
Patent Text Reader

Abstract

A data processing method and apparatus, a storage medium, and an electronic device. The data processing method comprises: by means of a neural network, performing an inference operation on data to be processed, the inference operation comprising using the neural network to successively carry out M rounds of inference on said data, and the M rounds of inference comprising an i-th round of inference and an (i+1)-th round of inference which are successively executed. The process of the i-th round of inference comprises: at a first stage in the process of the i-th round of inference, performing inference on N2 input data elements, and at a second stage in the process of the i-th round of inference, performing inference on N1 input data elements, the first stage being earlier than the second stage, N2 being greater than N1, and the N1 input data elements, which belong to the same batch of input data, being a subset of the N2 input data elements. The process of the (i+1)-th round of inference comprises: for N2-N1 input data elements for which the inference is incomplete in the i-th round of inference but for which the the output tokens have been acquired, performing inference on the N2-N1 input data elements while taking into account said output tokens.
Need to check novelty before this filing date? Find Prior Art

Description

Data processing method and device, electronic device, and storage medium

[0001] This application claims priority to Chinese patent application No. 202310524156.9 filed on May 10, 2023, the entire text of which is incorporated herein by reference as a part of this application. Technical Field

[0002] Embodiments of the present disclosure relate to a data processing method, a data processing apparatus, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0003] Artificial neural networks (ANNs), also known as neural networks, are mathematical models that mimic the behavioral characteristics of animal neural networks and perform distributed, parallel information processing. These networks rely on the complexity of the system to adjust the interconnections between its numerous nodes to achieve information processing. Regardless of the type of ANN, they all share common characteristics such as massively parallel processing, distributed storage, flexible topology, high redundancy, and nonlinear computation. These characteristics and capabilities are reflected in their computational speed, associative ability, adaptability, fault tolerance, and self-organization. These characteristics and capabilities form the technical foundation for ANNs to simulate intelligent activities and have found important applications in various technological fields. For example, ANNs can be used for data compression, data processing, video coding, and signal processing.

[0004] Summary of the Invention

[0005] At least one embodiment of the present disclosure provides a data processing method, comprising: performing an inference operation on data to be processed by a neural network, the inference operation comprising: performing M rounds of inference on the data to be processed in sequence using the neural network, wherein the data to be processed comprises multiple batches of input data, each batch of input data comprises N1 input data elements, and in each round of inference, only all inferences of one batch of input data are completed, M and N1 are positive integers greater than 1; the M rounds of inference comprise an i-th round of inference and an i+1-th round of inference, i being a positive integer less than M, wherein the inference process of the i-th round of inference comprises: in the first round of inference process of the i-th round of inference, In the first stage, N2 input data elements are inferred, and in the second stage of the reasoning process of the i-th round of reasoning, N1 input data elements are inferred to complete all reasoning of the N1 input data elements belonging to the same batch of input data; wherein, the first stage is earlier than the second stage, N2 is greater than N1, and the N1 input data elements belonging to the same batch of input data are a subset of the N2 input data elements; the reasoning process of the i+1-th round of reasoning includes: combining the inferred output symbols of the N2-N1 input data elements that have not been reasoned in the i-th round of reasoning, and reasoning on the N2-N1 input data elements.

[0006] For example, in a data processing method provided by at least one embodiment of the present disclosure, the neural network includes a first subnetwork and a second subnetwork, the first subnetwork is configured to receive input data elements, and extract semantic features corresponding to the input data elements in combination with a self-attention mechanism, wherein the semantic features include an initial key tensor and an initial value tensor, and the second subnetwork is configured to predict the output shape symbol sequence corresponding to the input data elements one by one in combination with an autoregressive mechanism based on the semantic features, wherein in the process of predicting the output shape symbol sequence corresponding to the input data elements one by one in combination with the autoregressive mechanism, the initial key tensor and the initial value tensor are continuously updated according to the predicted output shapes to obtain an updated key tensor and an updated value tensor, and the lengths of the updated key tensor and the updated value tensor are directly proportional to the number of output shapes corresponding to the predicted input data elements.

[0007] For example, in a data processing method provided by at least one embodiment of the present disclosure, N1 is determined by the number of tokens in the longest output token sequence supported by the neural network, the maximum length of the input data elements supported by the neural network, and the size of the storage space used to store the update key tensor and the update value tensor.

[0008] For example, in a data processing method provided by at least one embodiment of the present disclosure, in the first stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N2 input data elements, including: determining X batches of input data as the N2 input data elements, wherein the X batches of input data include a first batch of input data, and the first batch of input data includes the N1 input data elements that have been completely reasoned in the i-th round of reasoning, and X is a positive integer greater than 1; and reasoning is performed on the X batches of input data simultaneously in batches.

[0009] For example, in a data processing method provided by at least one embodiment of the present disclosure, determining X batches of input data as the N2 input data elements includes: in response to i being equal to 1, determining X batches of input data that have not been completely reasoned as the X batches of input data; in response to i being greater than 1: determining X-1 batches of input data that have not been completely reasoned in the i-1th round of reasoning, and a batch of input data that has not been completely reasoned, together as the X batches of input data, wherein the first batch of input data is determined from the X-1 batches of input data that have not been completely reasoned in the i-1th round of reasoning.

[0010] For example, in a data processing method provided by at least one embodiment of the present disclosure, X-1 batches of input data other than the first batch of input data among the X batches of input data have respective corresponding inference suspension thresholds, and inference is performed simultaneously on the X batches of input data in batches, including: simultaneously inference is performed on the X batches of input data, wherein, for any batch of input data among the X-1 batches of input data, when the number of output tokens inferred from the any batch of input data reaches the corresponding inference suspension threshold, the inference of the any batch of input data is suspended.

[0011] For example, in a data processing method provided by at least one embodiment of the present disclosure, in response to X=2, the inference suspension threshold is (L-2P) / 3, where P is the maximum length of the input text corresponding to a batch of input data supported by the neural network, L is the sum of the number of glyphs in the longest output glyph sequence supported by the neural network and P, and " / " represents division.

[0012] For example, in a data processing method provided by at least one embodiment of the present disclosure, when the number of output tokens inferred from any batch of input data reaches a corresponding inference suspension threshold, the inference of any batch of input data is suspended, including: suspending the inference of any batch of input data; storing the output tokens inferred from any batch of input data; and clearing the update key tensor and update value tensor generated by any batch of input data during the i-th round of inference.

[0013] For example, in a data processing method provided by at least one embodiment of the present disclosure, the output shape symbols inferred from any batch of input data are stored, including: converting each output shape symbol in the output shape symbols inferred from any batch of input data into a corresponding shape symbol index value; and storing the shape symbol index values ​​corresponding to the output shape symbols inferred from any batch of input data.

[0014] For example, in a data processing method provided in at least one embodiment of the present disclosure, reasoning is performed simultaneously on the X batches of input data, including: in response to i being greater than 1, for the other X-1 batches of input data in the X batches of input data except for a batch of input data that has not been completely reasoned, combining the inferred output shapes of the N2-N1 input data elements that have not been completely reasoned in the i-1th round of reasoning, and reasoning on the other X-1 batches of input data.

[0015] For example, in a data processing method provided by at least one embodiment of the present disclosure, in the second stage of the reasoning process of the i-th round of reasoning, N1 input data elements are reasoned on to complete all reasoning of the N1 input data elements belonging to the same batch of input data, including: continuing to reason on the first batch of input data until the reasoning of all input data elements in the first batch of input data is completed, wherein the first batch of input data includes the N1 input data elements that have all been reasoned in the i-th round of reasoning.

[0016] For example, in a data processing method provided by at least one embodiment of the present disclosure, the N2-N1 input data elements are inferred in combination with the inferred output shapes of the N2-N1 input data elements that have not been inferred in the i-th round of reasoning, including: obtaining the inferred output shapes of the N2-N1 input data elements in the i-th round of reasoning, wherein the inferred output shapes of the N2-N1 input data elements are stored in a storage space in the form of shape index values; and reasoning is performed on the N2-N1 input data elements in batches, wherein, for each input data element in the N2-N1 input data elements, the obtained inferred output shape of the input data element and the input data element are input into the neural network for reasoning.

[0017] For example, in a data processing method provided by at least one embodiment of the present disclosure, the output shape symbol obtained by inference of the input data element and the input data element are input into the neural network for inference, including: splicing the shape symbol index value corresponding to the output shape symbol obtained by inference of the input data element and the shape symbol index value corresponding to the input data element, and inputting the splicing result into the neural network for inference.

[0018] For example, in a data processing method provided by at least one embodiment of the present disclosure, the neural network includes a first subnetwork for performing prompt reasoning and a second subnetwork for performing token reasoning. While the first subnetwork is performing prompt reasoning on the N2-N1 input data elements, the first subnetwork is also performing prompt reasoning on a batch of input data that has not been reasoned at all. After the prompt reasoning on the N2-N1 input data elements and the batch of input data that has not been reasoned at all is completed, the second subnetwork is simultaneously started to perform token reasoning on the N2-N1 input data elements and the batch of input data that has not been reasoned at all.

[0019] For example, in a data processing method provided in at least one embodiment of the present disclosure, N2 is an integer multiple of N1.

[0020] At least one embodiment of the present disclosure provides a data processing device, comprising: an inference unit, configured to perform an inference operation on data to be processed through a neural network, wherein the inference operation comprises: sequentially performing M rounds of inference on the data to be processed using the neural network, wherein the data to be processed comprises multiple batches of input data, each batch of input data comprises N1 input data elements, and only all inferences for one batch of input data are completed in each round of inference, where M and N1 are positive integers greater than 1; the M rounds of inference comprise an i-th round of inference and an i+1-th round of inference, which are sequentially executed, where i is a positive integer less than M, and wherein the inference process of the i-th round of inference comprises: in a first stage of the inference process of the i-th round of inference, Reasoning is performed on N2 input data elements, and in the second stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N1 input data elements to complete all reasoning of the N1 input data elements belonging to the same batch of input data; wherein, the first stage is earlier than the second stage, N2 is greater than N1, and the N1 input data elements belonging to the same batch of input data are a subset of the N2 input data elements; the reasoning process of the i+1-th round of reasoning includes: combining the inferred output symbols of the N2-N1 input data elements that have not been reasoned in the i-th round of reasoning, and reasoning on the N2-N1 input data elements.

[0021] For example, a data processing device provided by at least one embodiment of the present disclosure also includes a storage unit, which is configured to store weight parameters of the neural network and update key tensors and update value tensors generated during the inference operation.

[0022] At least one embodiment of the present disclosure provides an electronic device, comprising: a memory, which non-transiently stores computer-executable instructions; and a processor, configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the data processing method according to any embodiment of the present disclosure.

[0023] At least one embodiment of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the data processing method according to any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure.

[0025] FIG1 is a schematic structural diagram of a neural network provided by at least one embodiment of the present disclosure;

[0026] FIG2 is a flowchart of an inference operation provided by an embodiment of the present disclosure;

[0027] FIG3 is a schematic diagram of the execution process of the data processing method provided by an embodiment of the present disclosure;

[0028] FIG4 is a schematic block diagram of a data processing device provided by at least one embodiment of the present disclosure;

[0029] FIG5 is a schematic block diagram of an electronic device provided in accordance with an embodiment of the present disclosure;

[0030] FIG6 is a schematic diagram of a non-transitory computer-readable storage medium provided by at least one embodiment of the present disclosure;

[0031] FIG7 is a schematic diagram of a hardware environment provided by at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0033] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of some known functions and known components.

[0034] The Transformer model is a classic neural network-based text processing model. It has given rise to important models such as BERT (Bidirectional Encoder Representation from Transformers) and GPT (Generative Pre-trained Transformer), which have significantly advanced the development of the NLP (Natural Language Processing) field.

[0035] The self-attention mechanism is a crucial component of the Transformer and models based on it. It's a specialized structure embedded in machine learning models that automatically learns and calculates the contribution of input data to output data. It connects information at different positions in the input sequence, incorporating the model's understanding of related words in the context of a particular word. Currently, the self-attention mechanism has become one of the most widely used components in deep learning, particularly in natural language processing.

[0036] The self-attention mechanism primarily uses three intermediate cache tensors: the query tensor (Query), the key tensor (Key), and the value tensor (Value). The query tensor of a token (a linguistic unit in natural language processing, which can refer to a word or part of a word) is multiplied with the key tensors of other tokens to obtain the attention scores of other tokens relative to the token itself. The key tensor acts like a label for each token in a sequence, used to match objects when searching for related tokens. The value tensor is the true representation of the token. After calculating the attention score, the value tensor is weighted and summed to obtain a tensor that represents the context of the current position, which is used to predict the output.

[0037] Models like GPT also use autoregressive mechanisms for inference or training, using deep learning to generate text that resembles human speech. After training, the model can predict the next token based on previously predicted tokens. For example, the autoregressive mechanism can include adding the token output by the current model to the end of the previously generated token sequence as the new input for the next step of the model, or directly using the token output by the current model as the new input for the next step of the model to predict the next token based on the new input. During inference, the model uses the autoregressive mechanism to make backward predictions one by one, outputting the predicted tokens one by one, and ultimately obtaining a sequence of output tokens corresponding to the input.

[0038] In deep learning, data can be inferred in batches to find the best balance between memory efficiency and memory capacity. For example, the data to be processed for inference can be divided into multiple batches. The size of each batch is called the batch size. Each batch of input data includes batch size input data elements. Each input data element can be the smallest unit processed by the neural network. For example, an input data element can be in the form of a sentence input by the user. The data in a batch is used to calculate a gradient descent. The loss function required for each parameter update in deep learning is not obtained from a single input data element, but is obtained by weighting the batch size input data elements. For example, if the batch size is 32, it means that 32 input data elements can be inferred at the same time, and the network parameters are updated based on the weighted loss values ​​of the 32 input data elements.

[0039] During the reasoning process, the model will perform calculations based on the key tensor and value tensor obtained by the self-attention mechanism. For example, a sub-network with an encoder or other structure is provided in the model to encode the input data elements using the self-attention mechanism to obtain the semantic features corresponding to the input data elements. For example, the semantic features include the key tensor and value tensor corresponding to the input data elements belonging to a batch of input data. The key tensor and value tensor are stored in the storage space of the processor for subsequent shape reasoning. At this time, the key tensor and value tensor are the same size and are proportional to the product of the maximum length of the input text corresponding to the batch of input data and the batch size.

[0040] During the first token inference, the semantic features include the original key and value tensors. At this point, the size of the key and value tensors is minimal throughout the token inference process. After obtaining the first predicted output token, the semantic features are updated. The updated semantic features include the updated key and value tensors, which carry the information of the first predicted output token, and so on. Therefore, as token inference continues, the size of the key and value tensors used for token inference increases with the length of the inference. In other words, the length of the key and value tensors is directly proportional to the number of predicted output tokens, and the storage space they occupy increases.

[0041] Taking the GPT-3 model as an example, it has 175 billion parameters and an output token length of up to 2048, which means that the number of tokens in the longest output token sequence it supports (inference length or number of inference steps) can reach 2048. The model's 175 billion parameters will occupy about 350GB of storage space (for parameters in FP16 format). For a typical single- or dual-node system, assuming that the total storage space of its graphics card is 512GB, then after deducting the 350GB of weight parameters that need to be fixedly saved, there is approximately 160GB of storage space left. During the inference process, the size of the key tensor and the value tensor is proportional to the batch size * (the maximum length of the input text corresponding to the batch input data + the number of predicted tokens / the number of inference steps executed), so as the inference proceeds, the storage space required for the key tensor and the value tensor will become larger and larger.

[0042] During the inference process, a larger batch size is usually desired to more efficiently use the tensor calculation operators provided by the processor and improve the processor's resource utilization. However, for a batch of input data, in order to ensure that the storage space can store the key tensors and value tensors of the next final moment (when all the input data inference is completed) when the inference of the batch of input data is completed, the batch size of the model inference needs to match the number of inference steps at the final moment, that is, the product of the batch size and the maximum number of inference steps (the sum of the number of tokens in the longest output token sequence supported by the neural network and the maximum length of the input data elements supported by the neural network) must be less than an upper limit. However, in the early stage of inference, the number of executed inference steps is small, and the key tensors and value tensors stored in the storage space have not yet completely occupied all the storage space. At this time, since the batch size is also small, the system computing power cannot be fully utilized, which limits the release of computing power in the overall inference process.

[0043] At least one embodiment of the present disclosure provides a data processing method and device, an electronic device, and a non-transient storage medium. The data processing method includes: performing reasoning operations on the data to be processed through a neural network, the reasoning operations including: using the neural network to perform M rounds of reasoning on the data to be processed, wherein the data to be processed includes multiple batches of input data, each batch of input data includes N1 input data elements, and only all reasoning of one batch of input data is completed in each round of reasoning, M and N1 are positive integers greater than 1; the M rounds of reasoning include the i-th round of reasoning and the i+1-th round of reasoning performed sequentially, i is a positive integer and less than M, wherein the reasoning process of the i-th round of reasoning includes: in the first stage of the reasoning process of the i-th round of reasoning, Inference is performed on N2 input data elements. In the second stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N1 input data elements to complete the reasoning of all N1 input data elements belonging to the same batch of input data; wherein, the first stage is earlier than the second stage, N2 is greater than N1, and the N1 input data elements belonging to the same batch of input data are a subset of the N2 input data elements; the reasoning process of the i+1-th round of reasoning includes: combining the output shapes obtained by reasoning of the N2-N1 input data elements that have not been completely reasoned in the i-th round of reasoning, and reasoning on the N2-N1 input data elements.

[0044] The data processing method provided by at least one embodiment of the present disclosure introduces an inference mode with a variable batch size. In the stage where the number of inference steps is small and the key tensor and value tensor occupy a small storage space (i.e., the first stage), a larger batch size is adopted to improve computing efficiency and fully utilize the computing power of the operators in the processor; as the inference continues (i.e., the second stage), the key tensor and value tensor gradually occupy a larger storage space, and the batch size is reduced to complete the inference, thereby accelerating the overall inference process, fully utilizing the system computing power, and improving the inference efficiency and computing power utilization.

[0045] The data processing method provided in the embodiments of the present disclosure can be applied to the data processing device provided in the embodiments of the present disclosure, and the data processing device can be configured on an electronic device. The electronic device can be a personal computer, a mobile terminal, etc. The mobile terminal can be a hardware device such as a mobile phone, a tablet computer, or a laptop computer.

[0046] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

[0047] For example, a data processing method provided by at least one embodiment of the present disclosure includes: performing reasoning operations on the data to be processed through a neural network.

[0048] For example, the data to be processed can be a dataset that includes all the data that the neural network needs to process.

[0049] FIG1 is a schematic structural diagram of a neural network provided by at least one embodiment of the present disclosure.

[0050] As shown in FIG1 , a neural network 100 for executing a data processing method provided by at least one embodiment of the present disclosure includes a first sub-network 101 and a second sub-network 102 .

[0051] For example, the first sub-network 101 is configured to receive input data elements and extract semantic features corresponding to the input data elements in combination with the self-attention mechanism, wherein the semantic features include an initial key tensor (Key stack) and an initial value tensor (Value stack).

[0052] For example, the second sub-network 102 is configured to predict the output token sequence corresponding to the input data elements one by one based on the semantic features in combination with an autoregressive mechanism.

[0053] For example, the neural network is a GPT network. Accordingly, when the neural network is used for reasoning, prompt inference is performed based on the first sub-network 101 and token inference is performed based on the second sub-network 102.

[0054] During model inference, the GPT network is given initial text (input data elements) as a prompt, and then generates relevant text based on the prompt. For example, the initial text can be user-entered text, such as questions, requests, emotional expressions, random chats, and so on.

[0055] During the prompt reasoning process, the first sub-network 101 is configured to receive input data elements, and extract semantic features corresponding to the input data elements in combination with the self-attention mechanism. For example, the semantic features include an initial key tensor and an initial value tensor as an intermediate cache. For example, during the prompt reasoning process, all input data elements belonging to the same batch of input data can be input into the first sub-network 101, and the corresponding initial key tensor and initial value tensor generated after calculation are stored in the storage space of the processor for subsequent shape reasoning. For example, since reasoning is performed in batches, the lengths of the same batch of input data elements may be different. Before entering the neural network for processing, they can all be padded to the same length, for example, a predefined blank marker (such as " <pad>”) so that the length of the input data element reaches the maximum length of the input text corresponding to the batch of input data.

[0056] For example, as mentioned above, for a batch of input data, the length of the initial key tensor and the initial value tensor generated by the first sub-network 101 are the same and are proportional to the product of the maximum length of the input text corresponding to the batch of input data and the batch size of the batch of input data.

[0057] Then, during the token inference process, the second sub-network 102 is configured to predict the output token sequence corresponding to the input data elements one by one based on the semantic features and in combination with an autoregressive mechanism. For example, the second sub-network 102 is configured to combine the semantic features with the predicted output tokens to predict the output tokens one by one.

[0058] As shown in FIG1 , the second sub-network 102 includes a first embedding layer, a plurality of decoding layers, and a second embedding layer.

[0059] The first embedding layer is configured to convert the input token into a corresponding token index value (token index), and encode the token index value to obtain an input vector corresponding to the input token. For example, a mapping table including all possible tokens is pre-set, and the mapping table provides a mapping relationship between tokens input by each user and their corresponding token index values. There is a one-to-many or many-to-one mapping relationship between tokens and their corresponding token index values, and the corresponding rule is determined by a tokenizer. The token index value is, for example, a floating point number or an integer. The token index value has a one-to-one correspondence with the encoded input vector, and the correspondence is determined by a word embedding table.

[0060] For example, when making a prediction for the first time, the input shape is a label indicating the beginning of a character text, such as " <ss>"or" <bos>Alternatively, the tag can be a tag indicating that the text content of the previous round of prediction has ended, such as " <eos>Alternatively, the label may be other labels depending on the task, etc., and the present disclosure does not limit this. The goal of the label is to predict what the shape symbol of the next position is.

[0061] For example, after the first output token is predicted, the input token is the output token of the predicted output. For example, the first output token is input as the input token to the second sub-network 102 for processing to obtain the predicted second output token; then, the predicted second output token is input as the input token to the second sub-network 102 for recognition to obtain the predicted third output token, and so on, until the end label is output, for example, output " <eos>", complete a round of shape symbol reasoning and obtain the output shape symbol sequence corresponding to the input data elements.

[0062] For example, multiple decoding layers are cascaded, and the first decoding layer is configured to process the semantic features and the input shape to obtain a processing result; the second decoding layer is configured to process the semantic features and the processing result of the first decoder to obtain a processing result; ...; the j-th decoding layer is configured to process the semantic features and the processing result output by the j-1-th decoding layer to obtain a processing result, where L is a positive integer greater than 1, and j is a positive integer greater than 1 and less than or equal to Z.

[0063] For example, as shown in Figure 1, during the shape-to-symbol reasoning process, the second sub-network 102 will continuously update the semantic features based on the output shape predicted by each reasoning, so that the semantic features used in the shape-to-symbol reasoning process will increase accordingly as the reasoning length increases one by one, and the updated semantic features will carry the information of the predicted shape.

[0064] For example, when making a prediction for the first time, the semantic features include the initial key tensor and initial value tensor from the first sub-network 101; after obtaining the output glyphs of the first prediction output, the semantic features are updated, and the updated semantic features include the updated key tensor and the updated value tensor, which carry information about the output glyphs of the first prediction output, and so on. Therefore, the lengths of the updated key tensor and the updated value tensor are directly proportional to the number of output glyphs corresponding to the predicted input data elements. For a batch of input data, the size of the updated key tensor and the updated value tensor is proportional to the batch size of the batch of input data * (the maximum length of the input text corresponding to the batch of input data + the number of predicted output glyphs / the number of inference steps executed).

[0065] Of course, it should be noted that the neural network shown in Figure 1 is a schematic structure, and the data processing method provided by the present disclosure can be applied to neural networks of other structures, such as other neural networks in which there are intermediate cache variables for data processing during the reasoning process, and the size of the variables will increase as the reasoning continues. The present disclosure does not impose specific restrictions on this.

[0066] FIG2 is a flowchart of an inference operation provided by an embodiment of the present disclosure.

[0067] As shown in FIG2 , the reasoning operation includes step S10 , in which a neural network is used to perform M rounds of reasoning on the data to be processed.

[0068] For example, the data to be processed (eg, a data set) includes multiple batches of input data, each batch of input data includes N1 input data elements, that is, the batch size is N1, where M and N1 are positive integers greater than 1.

[0069] For example, as mentioned above, the data to be processed for inference can be divided into multiple batches, each batch of input data includes N1 input data elements, and each input data element can be the smallest unit processed by the neural network. For example, an input data element can be in the form of a sentence input by the user.

[0070] For example, in a single round of reasoning, only the entire inference of a batch of input data is completed. That is, at the end of a round of reasoning, the inference of the batch of input data, including all input data elements, is completed, and the output token sequence corresponding to all input data elements in the batch of input data is obtained. Therefore, in this disclosure, a single round of reasoning refers to the completion of the entire inference of a batch of input data, rather than the completion of all inference on the data to be processed by the neural network. For example, if the data to be processed includes M batches of input data, then the entire inference of the M batches of input data can be completed through M rounds of reasoning.

[0071] For example, M rounds of reasoning include the i-th round of reasoning and the i+1-th round of reasoning that are executed sequentially, where i is a positive integer and is less than M. For example, the i-th round of reasoning is immediately followed by the i+1-th round of reasoning.

[0072] As shown in FIG. 2 , step S10 may include step S20 describing the reasoning process of the i-th round of reasoning and step S30 describing the reasoning process of the (i+1)-th round of reasoning.

[0073] For example, as shown in FIG2 , step S20 may include step S201 and step S202 .

[0074] Step S201 may include: in the first stage of the reasoning process of the i-th round of reasoning, reasoning on N2 input data elements.

[0075] Step S202 may include: in the second stage of the reasoning process of the i-th round of reasoning, reasoning on N1 input data elements to complete all reasoning of N1 input data elements belonging to the same batch of input data.

[0076] For example, the first stage is earlier than the second stage, N2 is greater than N1, and N1 input data elements belonging to the same batch of input data are a subset of N2 input data elements.

[0077] Step S30 may include: combining the inferred output shapes of the N2-N1 input data elements that have not been inferred in the i-th round of inference to infer the N2-N1 input data elements.

[0078] For example, N1 is determined by the number of tokens in the longest output token sequence supported by the neural network and the size of the storage space used to store the update key tensor and the update value tensor. As mentioned above, the size of the update key tensor and the update value tensor is proportional to the batch size * (the maximum length of the input text corresponding to the batch of input data + the number of predicted tokens / the number of inference steps performed). For a batch of input data, in order to ensure that the storage space can store the update key tensor and update value tensor at the next last moment (when all the input data inference is completed) when the inference of the batch of input data is completed, the batch size of the model inference needs to match the number of inference steps at the last moment. That is, the product of N1, the number of tokens in the longest output token sequence supported by the neural network step_t (for example, for GPT-3, it can be 2048) and the maximum length P of the input text corresponding to the batch of input data must be less than an upper limit, which is determined by the storage space for storing the key tensor and the value tensor.

[0079] In the present disclosure, in the first stage of a round of reasoning, such as the early to mid-stage of reasoning, reasoning is performed on input data of a larger batch size (such as N2), including reasoning on N1 input data elements belonging to the same batch of input data; in the second stage, such as the late stage of reasoning, the batch size is reduced to N1, that is, in the second stage, reasoning is performed on the N1 input data elements until all reasoning on the N1 input data elements is completed, completing one round of reasoning process.

[0080] Therefore, in the first stage, a larger batch size can make matrix multiplication, mapping and other calculations in the neural network more efficient, thereby improving computing efficiency. At this time, the computing efficiency is higher than that of only reasoning on N1 input data elements, and the computing power of the operators in the processor can be fully utilized. Since these calculations occupy most of the operations and weight loading in the entire reasoning, increasing the computing rate of this part can accelerate the overall reasoning process, make full use of the system computing power, and improve reasoning efficiency and computing power utilization.

[0081] It should be noted that in the present disclosure, variable batch size means that for a neural network, the number of input data elements processed simultaneously is variable, for example, multiple batches of input data are processed simultaneously, or a batch of input data is processed. A larger batch size means that more input data elements are inferred simultaneously in the early stage of inference, rather than that the batch size of each batch of input data is variable.

[0082] The specific reasoning process of each round of reasoning is described in detail below with reference to the accompanying drawings.

[0083] For example, step S201 in the i-th round of reasoning may include: determining X batches of input data as N2 input data elements; and performing reasoning on the X batches of input data simultaneously in batches.

[0084] For example, X batches of input data include the first batch of input data, Batch 1, which includes N1 input data elements that are fully inferred in the i-th round of inference, where X is a positive integer greater than 1. In other words, Batch 1 is the only batch of input data that needs to be fully inferred in the i-th round of inference; the other X-1 batches of input data will not be fully inferred in the i-th round of inference.

[0085] For example, the batch size of each of the X batches of input data is the same, and is N1. For example, N2 is an integer multiple of N1, such as N2 = X * N1. Of course, the batch sizes of the X batches of input data can also be different, and this disclosure does not impose any specific limitations on this.

[0086] For example, determining X batches of input data as N2 input data elements may include: in response to i being equal to 1, determining X batches of input data that have not been completely reasoned as the X batches of input data; in response to i being greater than 1: determining X-1 batches of input data that have not been completely reasoned in the i-1th round of reasoning, and a batch of input data that has not been completely reasoned, together as the X batches of input data, wherein the first batch of input data is determined from the X-1 batches of input data that have not been completely reasoned in the i-1th round of reasoning.

[0087] For example, for the first round of reasoning, all batches of input data have not yet been reasoned, and X batches of input data are arbitrarily selected as N2 input data elements for the first stage of reasoning.

[0088] For example, for the other M-1 rounds of reasoning other than the first round of reasoning, for example, for the second round of reasoning, it is assumed that in the first round of reasoning, input data Batch_1, input data Batch_2 to input data Batch_X, a total of X batches of input data, are selected as the N2 input data for reasoning in the first round of reasoning, where input data Batch_1 is the first batch of input data that has been completely reasoned in the first round of reasoning, and input data Batch_2 to input data Batch_X that have not been reasoned in the first round of reasoning, and a batch of input data Batch_X+1 that has not been subjected to any reasoning are selected as the X batches of input data as N2 input data elements used in the second round of reasoning.

[0089] For the first round of reasoning, the first batch of input data that is completely reasoned in the first round of reasoning can select any batch of input data from the X batches of input data.

[0090] For the M-1 rounds of reasoning other than the first, the first batch of input data must be determined from the X-1 batches of input data that were not fully reasoned in the i-1 round. For example, for the second round of reasoning, any batch of input data from Batch_2 to Batch_X can be selected as the first batch of input data and fully reasoned in the second round. In other words, in the second and subsequent rounds of reasoning, the N1 input data elements that are fully reasoned in each round must have been at least partially reasoned in the previous round.

[0091] For example, X-1 batches of input data other than the first batch of input data in the X batches of input data have their own corresponding inference suspension thresholds. The inference suspension thresholds are used to indicate that in the i-th round of inference, if the number of output tokens inferred from any batch of input data in the X-1 batches of input data reaches the corresponding inference suspension threshold, the inference of the batch of input data is suspended.

[0092] For the first batch of input data, there is no corresponding inference suspension threshold. That is, in the first stage, the first batch of input data will continue to be inferred until the other X-1 batches of input data reach their respective inference suspension thresholds and are suspended. At this time, the first stage ends.

[0093] For example, when the number of output shapes inferred from any batch of input data reaches a corresponding inference suspension threshold, pausing the inference of any batch of input data may include: pausing the inference of any batch of input data; storing the output shapes inferred from any batch of input data; and clearing the update key tensor and update value tensor generated by any batch of input data during the i-th round of inference.

[0094] For example, storing the output shape symbols inferred from any batch of input data may include: converting each output shape symbol in the output shape symbols inferred from any batch of input data into a corresponding shape symbol index value; and storing the shape symbol index values ​​corresponding to the output shape symbols inferred from any batch of input data.

[0095] For example, taking the first round of reasoning as an example, the input data Batch_2 corresponds to the reasoning suspension threshold s2, the input data Batch_3 corresponds to the reasoning suspension threshold s3, ..., and the input data Batch_X corresponds to the reasoning suspension threshold sx.

[0096] For example, at the beginning of the first stage of the first round of reasoning, the input data Batch_1 to the input data Batch_X are reasoned simultaneously in batches. That is, in the first stage, the number of input data elements being reasoned simultaneously is X*N1. When the number of output tokens inferred from the input data Batch_2 reaches s2, the continued reasoning of the input data Batch_2 is suspended, and the output tokens inferred from the input data Batch_2 are converted into token index values ​​(token index) and stored in the storage space. At the same time, the update key tensor and update value tensor generated by reasoning the input data Batch_2 in the i-th round of reasoning are cleared. Since the storage space occupied by the token index is extremely small, it will not affect the continued reasoning of the input data Batch_1 in the subsequent second stage.

[0097] Similarly, when the number of output tokens inferred from the input data Batch_X reaches sx, the reasoning of the input data Batch_X is suspended, and the output tokens inferred from the input data Batch_X are converted into token index values ​​(token index) and stored in the storage space. At the same time, the update key tensor and update value tensor generated by inferring the input data Batch_X in the i-th round of reasoning are cleared.

[0098] In the first stage, the first batch of input data Batch_1 is continuously inferred. When the input data Batch_2 to Batch_X all reach their corresponding inference pause thresholds and are paused, the first stage ends.

[0099] For example, the inference pause thresholds for each batch of input data can be different or the same. For example, if X = 2, the inference pause thresholds for each batch of input data can be the same. For example, if X is greater than 2, the inference pause thresholds for the X batches of input data other than the first batch of input data in the same round can be different.

[0100] For example, the inference pause threshold corresponding to each batch of input data is related to the number of tokens in the longest output token sequence supported by the neural network, the maximum length of the input text corresponding to a batch of input data supported by the neural network, the number of input data elements currently being inferred, etc.

[0101] For example, in response to X=2, the inference suspension threshold corresponding to each batch of input data is the same and is (L-2P) / 3, where P is the maximum length of the input text corresponding to a batch of input data supported by the neural network, L is the sum of the number of glyphs in the longest output glyph sequence supported by the neural network and P, and " / " represents division.

[0102] For example, taking the inference suspension threshold s2 corresponding to the input data Batch_2 as an example, assuming that s2 is the minimum value of all inference suspension thresholds, when the number of output shapes inferred from the input data Batch_2 reaches s2, the number of output shapes inferred from other X-1 batches of input data such as input data Batch_1 and input data Batch_X also reaches s2. At this time, the update key tensors and update value tensors generated by each batch of input data during the inference process have filled all the storage space. Therefore, the inference of the input data Batch_2 can be suspended, the update key tensors and update value tensors generated by the input data Batch_2 can be cleared, and the output shapes inferred from the input data Batch_2 can be stored in the storage space in the form of shape index values.

[0103] Afterwards, reasoning continues for the remaining X-1 batches of input data, clearing the available storage space created by the update key tensors and update value tensors generated by input data Batch_2. This storage space can then be used to store the update key tensors and update value tensors generated by the remaining X-1 batches of input data during the continued reasoning process, until the storage space is full again. For example, if the number of output tokens inferred from the remaining X-1 batches of input data reaches the inference pause threshold s3 corresponding to input data Batch_3 (s3 is greater than s2 but less than the other inference pause thresholds), reasoning for input data Batch_3 is paused, the update key tensors and update value tensors generated by input data Batch_3 during the inference process are cleared, and reasoning continues for the remaining X-2 batches of input data (other batches of input data excluding input data Batch_2 and Batch_3). The available storage space created by the update key tensors and update value tensors generated by input data Batch_3 during the continued reasoning process can then be used to store the update key tensors and update value tensors generated by the remaining X-2 batches of input data during the continued reasoning process.

[0104] The above process is performed until X-1 batches of input data except the first batch of input data are paused, and the first stage ends.

[0105] For example, simultaneously reasoning on X batches of input data may include: in response to i being greater than 1, for X-1 batches of input data other than a batch of input data that has not been completely reasoned about, combining the inferred output shapes of the N2-N1 input data elements that have not been completely reasoned about in the i-1th round of reasoning, and reasoning on the other X-1 batches of input data.

[0106] For example, in the first round of reasoning, X batches of input data can be reasoned according to the conventional neural network reasoning process, such as performing prompt reasoning and shape reasoning with reference to the relevant description described in FIG1 .

[0107] For example, in rounds of reasoning other than the first, the conventional neural network inference process is also used for inference on a batch of completely unreasoned input data processed in that round, which will not be described in detail here. For the X-1 batches of input data other than the batch of completely unreasoned input data, these input data have already been partially inferred in the previous round or several previous rounds of inference, and the inferred output tokens have been stored. Therefore, inference can be performed on these X-1 batches of input data in combination with the previously inferred output tokens, accelerating the overall inference process.

[0108] For example, in this round of reasoning, the inferred output tokens and the original input data can be combined as new inputs and fed into the neural network for processing. For example, after passing through the first sub-network, the resulting semantic features can include information about the inferred output tokens. When performing token reasoning using the second sub-network, reasoning can proceed directly to the last token in the inferred output tokens. At this point, the lengths of the initial key tensor and initial value tensor obtained by the first sub-network are directly proportional to the sum of the maximum length of the input text corresponding to the batch of input data and the number of inferred output tokens.

[0109] Therefore, in each round of inference, multiple batches of input data are processed, and partial inference is also performed on X-1 batches of input data in addition to the first batch. This results in an additional portion of inference being performed on the other X-1 batches in each round of inference, speeding up inference overall. Furthermore, weight loading time remains constant. Since inference is performed on N2 input data elements simultaneously in the first phase, inference speed is largely unaffected, and the computational efficiency and utilization of the processing cores are increased.

[0110] Regarding the specific process of reasoning the X-1 batches of input data in combination with the previously inferred output shape, reference may be made to the relevant description of step S30 in the following text, which will not be repeated here.

[0111] For example, step S202 may include: in the second stage of the reasoning process of the i-th round of reasoning, continuing to reason on the first batch of input data until reasoning on all input data elements in the first batch of input data is completed. For example, the first batch of input data includes N1 input data elements that have all been reasoned in the i-th round of reasoning.

[0112] In the first stage, all X-1 batches of input data except the first batch of input data reach their respective inference suspension thresholds and are suspended. In the second stage, the inference is continued on the first batch of input data until all N1 input data elements included in the first batch of input data are inferred, thus completing the i-th round of inference.

[0113] For the (i+1)th round of reasoning, it may also include the first stage and the second stage.

[0114] For example, in the first stage, X batches of input data are first determined as the N2 input data elements to be inferred in the i+1 round of reasoning. Similar to the i-th round of reasoning, the N2 input data elements can include the N2-N1 input data elements that were not fully inferred in the i-th round of reasoning, as well as a batch of input data that has not been reasoned about at all (N1 input data elements). For example, in the i+1 round of reasoning, the N1 input data elements of the N2-N1 input data elements that were not fully inferred in the i-th round of reasoning are determined as the first batch of input data that needs to be reasoned about in the i+1 round of reasoning.

[0115] Afterwards, in the first stage, inference is performed on N2 input data elements in batches.

[0116] Similar to the i-th round of reasoning, for a batch of input data that has not been reasoned about in the N2 input data elements, reasoning can be performed according to the conventional neural network reasoning process. For the N2-N1 input data elements that have been partially reasoned about in the i-th round of reasoning, reasoning is performed on these N2-N1 input data elements in combination with the output shapes obtained from the reasoning of these N2-N1 input data elements in the i-th round of reasoning.

[0117] For example, step S30 may include: obtaining the inferred output shapes of N2-N1 input data elements in the i-th round of reasoning, wherein the inferred output shapes of N2-N1 input data elements are stored in the storage space in the form of shape index values; reasoning on the N2-N1 input data elements in batches, wherein for each input data element in the N2-N1 input data elements, the obtained inferred output shape of the input data element and the input data element are input into the neural network for reasoning.

[0118] For example, inputting the output shape symbol obtained by inferring the input data element and the input data element into the neural network for reasoning can include: splicing the shape symbol index value corresponding to the output shape symbol obtained by inferring the input data element and the shape symbol index value corresponding to the input data element, and inputting the splicing result into the neural network for reasoning.

[0119] For example, in the second round of reasoning, the N2-N1 input data elements include input data Batch_2 to input data Batch_X that have been partially reasoned in the first round of reasoning. For example, for input data Batch_2, s2 steps of reasoning have been performed in the first round of reasoning. That is, each input data element in input data Batch_2 has output s2 output tokens, which are stored in the storage space as token index values.

[0120] The shape symbol index values ​​corresponding to the s2 output shape symbols corresponding to each input data element are extracted from the storage space, the s2 output shape symbol index values ​​and the shape symbol index values ​​corresponding to the input data elements are spliced, and the splicing result is used as the new input data element. Each new input data element is input into the neural network for inference in batches.

[0121] For example, after passing through the first sub-network, the obtained semantic features can contain information about the executed s2-step reasoning. When using the second sub-network for token reasoning, it is possible to directly advance to the last token in the inferred output token to start reasoning.

[0122] For example, the neural network includes a first subnetwork for performing hint reasoning and a second subnetwork for performing shape-token reasoning. For example, hint reasoning includes using the first subnetwork to extract semantic features corresponding to input data elements, and shape-token reasoning includes, for example, using the extracted semantic features and combining an autoregressive mechanism to predict output shape-token sequences corresponding to the input data elements one by one. The specific processes of hint reasoning and shape-token reasoning can be found in the aforementioned description of the neural network and will not be elaborated upon here.

[0123] For example, while prompt reasoning is being performed on N2-N1 input data elements through the first sub-network, prompt reasoning is also being performed on a batch of input data that has not been reasoned at all through the first sub-network; and after prompt reasoning is completed on both the N2-N1 input data elements and the batch of input data that has not been reasoned at all, shape symbol reasoning is simultaneously started on the N2-N1 input data elements and the batch of input data that has not been reasoned at all through the second sub-network.

[0124] For example, using the second round of inference as an example, the newly concatenated input data elements obtained through the aforementioned process are fed into the first sub-network in batches for prompt inference, obtaining the semantic features corresponding to each new input data element. Simultaneously, the first sub-network performs prompt inference on a batch of completely uninferred input data (e.g., input data Batch_X+1) to obtain the semantic features of this batch of input data.

[0125] After prompt inference is completed for both the N2-N1 input data elements and the input data Batch_X+1, token inference for both begins simultaneously. For example, for the N2-N1 input data elements, the semantic features containing the inferred output token information can be used to directly advance the inference to the last token in the inferred output token, accelerating the inference.

[0126] For example, for the other X-1 batches of input data except the first batch of input data among the N2 input data elements processed in the i+1 round of reasoning, the reasoning is suspended when the number of output tokens inferred in the i+1 round of reasoning reaches the corresponding reasoning suspension threshold. The specific process is as described in the relevant description of step S201 and will not be repeated here.

[0127] Afterwards, in the second stage, the first batch of input data in the (i+1) round of reasoning is continued to be reasoned until all reasoning of the first batch of input data is completed, thereby completing the (i+1) round of reasoning.

[0128] The above process is repeated until all the reasoning of the data to be processed is completed.

[0129] For example, if i+1 is equal to M, for the Mth round of reasoning, in some embodiments, there may be no more completely unreasoned input data that needs to be reasoned. At this time, the Mth round of reasoning may not include the first stage, and only step S30 is executed to complete all reasoning of the N2-N1 input data elements that have not been reasoned in the i-th round of reasoning.

[0130] In the data processing method provided by at least one embodiment of the present disclosure, in the early stages of a round of reasoning, such as the first stage, a larger batch size is used, for example, by performing reasoning on multiple batches of input data simultaneously. For systems with redundant computing power, this can improve computing power utilization in this portion of reasoning, thereby improving overall reasoning efficiency. In the later stages of a round of reasoning, such as the second stage, the batch size is reduced to N1 to complete the entire reasoning of a batch of input data. As a result, the system computing power is fully utilized in each round of reasoning, and partial reasoning of N2-N1 additional input data elements is performed, thereby accelerating the overall reasoning process and improving system computing efficiency and computing power utilization efficiency.

[0131] FIG3 is a schematic diagram of the execution process of the data processing method provided in one embodiment of the present disclosure.

[0132] For example, in the example shown in Figure 3, assuming that the batch size * (step_t + P) corresponding to each batch of input data must meet the upper limit imposed by storage space, the maximum batch size can be N1, where step_t represents the number of tokens in the longest output token sequence supported by the neural network, and P represents the maximum length of input text corresponding to a batch of input data supported by the neural network. The batch size of each batch of input data is N1, meaning that each round of reasoning completes the entire reasoning of N1 input data elements.

[0133] For example, in the example of Figure 3, it is shown that all the reasoning of input data Batch_1, input data Batch_2, input data Batch_3, ..., input data Batch_M is completed through M rounds of reasoning, and the batch sizes of input data Batch_1 to input data Batch_M are all N1, N2 is twice that of N1, and X=2, that is, in the first stage, reasoning is performed on two batches of input data at the same time, and in the second stage, reasoning is performed on one batch of input data.

[0134] For example, the neural network used when executing the data processing method can refer to the structure and reasoning process of the neural network in Figure 1 above, which will not be repeated here.

[0135] In Figure 3, the boxes in the same row represent the reasoning process of a Batch (a batch of input data). For example, in the first round of reasoning, the boxes in the upper row represent the reasoning process of the input data Batch_2, and the boxes in the lower row represent the reasoning process of the input data Batch_1.

[0136] For example, for a Batch, the slashed box part represents the prompt reasoning completed by the first sub-network, and the remaining part represents the token reasoning completed by the second sub-network. In addition, the gray box in the remaining part represents the reasoning that has been performed, and the white box represents the part that has not yet been reasoned.

[0137] For example, for a Batch, p represents the maximum length of the input text corresponding to a batch of input data supported by the neural network. For a batch of input data, if the lengths of the input data elements are different, they are all padded to the maximum length p by padding. The specific process is as described above and will not be repeated here. L represents the longest number of reasoning steps to complete all the reasoning of a batch of input data. The longest number of reasoning steps is proportional to the length of the updated key tensor and the updated value tensor at the last moment of a round of reasoning. The longest number of reasoning steps L = p + step_t, step_t represents the number of symbols in the longest output symbol sequence supported by the neural network (that is, the longest number of reasoning steps for symbol reasoning); s represents the reasoning suspension threshold. In the example of Figure 3, the reasoning suspension thresholds corresponding to input data Batch_2 to input data Batch_M are all s, which means that when the number of output symbols obtained in this round of reasoning reaches s during the symbol reasoning process, the reasoning of the batch of input data is suspended.

[0138] The following describes in detail the execution process of the data processing method provided by at least one embodiment of the present disclosure in conjunction with FIG3 .

[0139] In the first round of inference, as shown in Figure 3, two batches of input data, both of size N1, are selected from the data to be processed and inference begins simultaneously. For example, input data Batch_1 and input data Batch_2 are selected as the N2 input data elements to be processed in the first round. Input data Batch_1 is selected as the first batch of input data in the first round of inference. This means that in the first round of inference, all inference on input data Batch_1 must be completed.

[0140] For example, in the first stage, input data Batch_1 and input data Batch_2 are each inferred. The inference process includes first performing p steps of prompt reasoning, and then performing shape-to-symbol reasoning based on the semantic features generated by the prompt reasoning. The specific process can refer to the relevant description of the aforementioned neural network.

[0141] During the shape symbol inference process, the update key tensor and update value tensor continue to grow, occupying more and more storage space. For example, when the inference of input data Batch_1 and input data Batch_2 reaches the sth step (outputting s output shapes), the inference pause threshold corresponding to input data Batch_2 is reached. At this time, the update key tensor and update value tensor of input data Batch_1 and input data Batch_2 have filled the storage space. The inference of input data Batch_2 is paused, and the inferred output shapes of input data Batch_2 are stored in the storage space, for example, as shape index values. The update key tensor and update value tensor generated by input data Batch_2 during the first round of inference are cleared, and the first stage ends.

[0142] The space occupied by the shape index value corresponding to the output shape obtained by the inference of the stored input data Batch_2 is extremely small and can be ignored. It does not affect the storage of the update key tensor and update value tensor generated by the input data Batch_1 in the second stage of inference.

[0143] In the second phase, as shown in Figure 3, inference on input data Batch_2 is paused, while inference on input data Batch_1 continues until inference on all input data elements in input data Batch_1 is complete, thus completing the first round of inference. For example, after the first round of inference is complete, the storage space occupied by the updated key tensor and updated value tensor of input data Batch_1 can be released to allow for the second round of inference.

[0144] In the first round of reasoning, all the input data Batch_1 was reasoned, and the input data Batch_2 was reasoned for s steps.

[0145] It should be noted that here, the first and second stages are two descriptive stages. In fact, the reasoning process does not need to be stopped in the first stage and restarted in the second stage. For example, for the input data Batch_1, its shape symbol reasoning process is continuous, and the relevant operations of pausing the reasoning of the input data Batch_2 can be completed before the input data Batch_1 executes the s+1th step.

[0146] In the second round of inference, as shown in Figure 3, in the first stage, we select Batch_3, an input data batch of size N1 that has not been inferred at all, and Batch_2, an input data batch that was not fully inferred in the first round of inference, from the data to be processed. For example, Batch_2 and Batch_3 are selected as the N2 input data elements processed in the first stage, and Batch_2 is selected as the first batch of input data in the second round of inference. This means that in the second round of inference, all inference on Batch_2 must be completed.

[0147] For example, in the first stage, input data Batch_2 and input data Batch_3 are each subjected to inference.

[0148] As shown in Figure 3, while the first sub-network is performing prompt reasoning on the input data Batch_2, the first sub-network is also performing prompt reasoning on the input data Batch_3. After the prompt reasoning on both the input data Batch_2 and the input data Batch_3 is completed, the second sub-network is used to perform shape symbol reasoning on both the input data Batch_2 and the input data Batch_3 at the same time.

[0149] For the input data Batch_3, the prompt reasoning is the same as the conventional process and will not be repeated here.

[0150] For the input data Batch_2, it is necessary to combine the reasoning results of the s-step reasoning executed in the first round of reasoning (the output shape symbol that has been inferred) to reason on the input data Batch_2. Specifically, as described in step S30, for each input data element in the input data Batch_2, the shape symbol index value corresponding to the inferred output shape symbol and the shape symbol index value corresponding to the input data element are spliced, and the splicing result is used as a new input data element, and these new input data elements are inferred through the neural network. The prompt reasoning process for the new input data element is the same as the conventional prompt reasoning process, but the length is (p+s). Therefore, the initial key tensor and initial value tensor obtained by the new input data element contain the relevant information of the reasoning results of the s-step reasoning executed in the first round of reasoning. When performing shape symbol reasoning on the input data Batch_2, it can be directly advanced to the s-step reasoning without repeating the reasoning of the first s steps executed in the first round of reasoning.

[0151] During the shape symbol reasoning process, the update key tensor and update value tensor continue to grow, occupying more and more storage space. For example, when the input data Batch_3 is reasoned to the sth step (outputting s output shapes), the reasoning pause threshold corresponding to the input data Batch_3 is reached. At this time, the update key tensor and update value tensor of the input data Batch_2 and the input data Batch_3 have filled the storage space, and the reasoning of the input data Batch_3 is suspended. The output shapes that have been reasoned for the input data Batch_3 are stored, for example, in the form of shape index values ​​in the storage space. The update key tensor and update value tensor generated by the input data Batch_3 in the second round of reasoning are cleared, and the first stage ends.

[0152] In the second stage, as shown in FIG3 , the reasoning of the input data Batch_3 is suspended, and the reasoning of the input data Batch_2 continues until the reasoning of all input data elements in the input data Batch_2 is completed, thereby completing the second round of reasoning.

[0153] In the second round of reasoning, all the input data Batch_2 was reasoned, and the input data Batch_3 was reasoned for s steps.

[0154] The subsequent reasoning processes from the third round to the M-1 round are similar to the second round. In each round, a batch of input data that has completed s-step shape symbol reasoning in the previous round and a batch of new input data that have not been reasoned at all are reasoned at the same time. The batch of input data that has completed s-step shape symbol reasoning completes all reasoning in this round, and the other batch of input data that has not been reasoned at all completes s-step shape symbol reasoning in this round, and releases the update key tensor and update value tensor of the other batch of input data. The reasoning results of the s-step shape symbol reasoning executed on the other batch of input data in this round are stored in the form of shape symbol index values. The specific process will not be repeated here.

[0155] For the Mth round of reasoning, since there is no more input data that has not been reasoned before, it is only necessary to complete the reasoning of the input data Batch_M that has completed s steps of shape symbol reasoning in the M-1th round of reasoning according to the aforementioned related process of Batch_2 in the second round of reasoning.

[0156] In the above embodiment, after removing the space occupied by the weight parameters of the neural network itself, the remaining part of the storage space is basically used to place the update key tensor and update value tensor generated by reasoning. Its volume is proportional to the completed reasoning part represented by the shaded or gray box part in the figure. As can be seen from Figure 3, the completed reasoning part of each round of reasoning except the first round of reasoning and the Mth round of reasoning is the largest.

[0157] For example, taking the second round of reasoning as an example, for a batch of input data with a total length of L to be output, when the reasoning reaches the S-th step of the prompt reasoning of the input data Batch_3, all storage space is used up, that is, it is consistent with the maximum storage space occupied when a batch of input data is normally reasoned to the L-th step. Therefore, 3S+2P=L, that is, the size of S is determined by the formula S=(L-2P) / 3.

[0158] It should be noted that in the above embodiment, the specific execution process of the data processing method provided by at least one embodiment of the present disclosure is described by taking X=2 as an example, but when X takes a larger value, the execution logic is similar and will not be repeated here.

[0159] In this embodiment, the batch size in the early stage of reasoning is increased by 2 times, that is, 2*N1 input data elements are inferred at the same time. For systems with redundant computing power, the computing power utilization can be improved in this part of the reasoning, and the utilization of tensor cores can be improved. In each round of reasoning, s more steps of reasoning are performed, thereby improving the overall reasoning efficiency. In the later stage of reasoning, the batch size is reduced to N1 to complete the entire reasoning of a batch of input data, thereby realizing reasoning acceleration.

[0160] At least one embodiment of the present disclosure further provides a data processing device. FIG4 is a schematic block diagram of a data processing device provided by at least one embodiment of the present disclosure.

[0161] As shown in Figure 4, data processing device 200 may include an inference unit 201 and a storage unit 202. It should be noted that the components and structure of data processing device 200 shown in Figure 4 are merely exemplary and non-limiting, and data processing device 200 may also have other components and structures as needed.

[0162] For example, these modules can be implemented by hardware (e.g., circuit) modules, software modules, or any combination thereof, and the following embodiments are the same and will not be described in detail. For example, these units can be implemented by a central processing unit (CPU), a data processing unit (GPU), a tensor processing unit (TPU), a field programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities, as well as corresponding computer instructions.

[0163] For example, the data processing device may be a processor that needs to perform reasoning operations, such as a general-purpose processor such as a central processing unit, a multi-core processor, a graphics processor, a general-purpose graphics processor, a digital signal processor, or a dedicated processor such as an AI processor, a tensor processing unit (TPU), or a field programmable gate array (FPGA). For example, the data processing device may be a component specifically provided in some GPUs or AI accelerators for optimizing calculations, such as a deep learning accelerator. Of course, the present disclosure is not limited thereto, and the data processing device may be any device that needs to execute a data processing method to perform corresponding reasoning operations.

[0164] For example, the inference unit 201 is configured to perform an inference operation on the data to be processed through a neural network.

[0165] For example, an inference operation includes performing M rounds of inference on the data to be processed using a neural network. Here, the data to be processed includes multiple batches of input data, each batch of input data includes N1 input data elements, and only the entire inference of one batch of input data is completed in each round of inference. M and N1 are positive integers greater than 1.

[0166] For example, M rounds of reasoning include the i-th round of reasoning and the i+1-th round of reasoning executed sequentially, where i is a positive integer and is less than M.

[0167] For example, the reasoning process of the i-th round of reasoning includes: in the first stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N2 input data elements, and in the second stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N1 input data elements to complete all reasoning of the N1 input data elements belonging to the same batch of input data; wherein, the first stage is earlier than the second stage, N2 is greater than N1, and the N1 input data elements belonging to the same batch of input data are a subset of the N2 input data elements;

[0168] For example, the reasoning process of the i+1th round of reasoning includes: combining the inferred output shapes of the N2-N1 input data elements that have not been reasoned in the i-th round of reasoning, and reasoning on the N2-N1 input data elements.

[0169] For example, the inference unit 201 includes a neural network 203 , and the neural network 203 includes a first sub-network and a second sub-network.

[0170] The first sub-network is configured to receive input data elements and extract semantic features corresponding to the input data elements in combination with a self-attention mechanism, wherein the semantic features include an initial key tensor and an initial value tensor.

[0171] The second sub-network is configured to predict the output token sequence corresponding to the input data elements one by one based on semantic features and combined with an autoregressive mechanism.

[0172] For example, in the process of predicting the output shape symbol sequence corresponding to the input data elements one by one in combination with the autoregressive mechanism, the initial key tensor and the initial value tensor are continuously updated according to the predicted output shapes to obtain the updated key tensor and the updated value tensor. The lengths of the updated key tensor and the updated value tensor are directly proportional to the number of output shapes corresponding to the predicted input data elements.

[0173] It should be noted that the structure and function of the neural network 203 in the inference unit 201 are the same as those of the neural network 100 in the embodiment of the above-mentioned data processing method, and will not be described in detail here.

[0174] For example, the storage unit 202 is configured to store the weight parameters of the neural network and the updated key tensors and updated value tensors generated during the inference operation. For example, the storage unit 202 can be implemented in the form of a memory, such as HBM (High Bandwidth Memory), etc. The present disclosure does not specifically limit the specific form of the storage unit.

[0175] For example, the inference unit 201 may include codes and programs stored in a memory; a processor may execute the codes and programs to implement some or all of the functions of the inference unit 201 as described above. For example, the inference unit 201 may be a dedicated hardware device used to implement some or all of the functions of the inference unit 201 as described above. For example, the inference unit 201 may be a circuit board or a combination of multiple circuit boards used to implement the functions as described above. In an embodiment of the present disclosure, the circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processors; and (3) firmware stored in the memory that is executable by the processor.

[0176] It should be noted that the inference unit 201 can be used to implement steps S10-S30 shown in Figure 1. Therefore, for a detailed description of the functions that the inference unit 201 can implement, reference can be made to the description of steps S10 to S30 in the embodiment of the above data processing method, and the repetitive parts will not be repeated.

[0177] The data processing device provided by at least one embodiment of the present disclosure adopts a larger batch size (simultaneously inferring more input data elements) to improve computing efficiency and fully utilize the computing power of the operators in the processor in the stage where the number of inference steps is small and the key tensor and value tensor occupy a small amount of storage space (i.e., the first stage) to fully utilize the computing power of the operators in the processor; as the inference continues (i.e., the second stage), the key tensor and value tensor gradually occupy a larger storage space, and the batch size is reduced to complete the inference, thereby accelerating the overall inference process, fully utilizing the system computing power, and improving inference efficiency and computing power utilization. The data processing device 200 can achieve technical effects similar to those of the aforementioned data processing method, which will not be repeated here.

[0178] It should be noted that in the embodiments of the present disclosure, the data processing device 200 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited and can be composed of analog devices according to circuit principles, or can be composed of digital chips, or constructed in other applicable ways.

[0179] Figure 5 is a schematic block diagram of an electronic device provided in accordance with an embodiment of the present disclosure. As shown in Figure 5 , the electronic device 300 is suitable for implementing the data processing method provided in accordance with an embodiment of the present disclosure. It should be noted that the components of the electronic device 300 shown in Figure 5 are merely exemplary and non-limiting. The electronic device 300 may also include other components as required by actual applications.

[0180] As shown in FIG5 , the electronic device 300 may include a processing device (eg, a central processing unit, a graphics processing unit, etc.) 301 , which may perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in a memory to implement various functions.

[0181] For example, when the computer-readable instructions are executed by the processing device 301, one or more steps of the data processing method described in any of the above embodiments may be executed. It should be noted that for a detailed description of the processing process of the data processing method, reference can be made to the relevant description in the above embodiments of the data processing method, and any repetitions will not be repeated.

[0182] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory. For example, computer-readable instructions may be loaded from storage device 308 into random access memory (RAM) 303 to execute the computer-readable instructions. Non-volatile memory may include, for example, read-only memory (ROM) 302, a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, a flash memory, etc. Various applications and various data, such as style images and various data used and / or generated by the applications, may also be stored in the computer-readable storage media.

[0183] For example, the processing device 301, the ROM 302, and the RAM 303 are connected to one another via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0184] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 308 including, for example, magnetic tape, hard disk, flash memory, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other electronic devices wirelessly or wired to exchange data. Although FIG9 shows an electronic device 300 with various devices, it should be understood that it is not required to implement or possess all of the devices shown, and the electronic device 300 may alternatively implement or possess more or fewer devices. For example, the processor 301 may control other components in the electronic device 300 to perform desired functions. The processor 301 may be a device with data processing and / or program execution capabilities, such as a central processing unit (CPU), a tensor processing unit (TPU), or a graphics processing unit (GPU). The central processing unit (CPU) may be of X86 or ARM architecture, etc. The GPU may be separately integrated directly onto the motherboard or built into the motherboard's northbridge chip. A GPU can also be built into a central processing unit (CPU).

[0185] FIG6 is a schematic diagram of a non-transitory computer-readable storage medium provided in at least one embodiment of the present disclosure. For example, as shown in FIG6 , storage medium 400 may be a non-transitory computer-readable storage medium, on which one or more computer-readable instructions 401 may be non-transitory stored. For example, when computer-readable instructions 401 are executed by a processor, the data processing method described above may be executed.

[0186] For example, the storage medium 400 may be applied to the above-mentioned electronic device. For example, the storage medium 400 may include a memory in the electronic device.

[0187] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a flash memory, or any combination of the above storage media, or other applicable storage media.

[0188] For example, the description of the storage medium 400 may refer to the description of the memory in the embodiment of the electronic device, and the repeated parts will be omitted.

[0189] Figure 7 is a schematic diagram of a hardware environment provided by at least one embodiment of the present disclosure. The electronic device provided by the present disclosure can be applied in an Internet system.

[0190] The computer system provided in FIG7 can be used to implement the functions of the data processing device and / or electronic device involved in this disclosure. Such computer systems may include personal computers, laptops, tablet computers, mobile phones, personal digital assistants, smart glasses, smart watches, smart rings, smart helmets, and any smart portable or wearable device. The specific system in this embodiment uses a functional block diagram to illustrate a hardware platform including a user interface. This computer device can be a general-purpose computer device or a computer device with a specific purpose. Both computer devices can be used to implement the data processing device and / or electronic device in this embodiment. The computer system may include any components that implement the information required to implement the data processing described herein. For example, the computer system can be implemented by the computer device through its hardware devices, software programs, firmware, and combinations thereof. For convenience, only one computer device is depicted in FIG7, but the relevant computer functions for implementing the information required for data processing described in this embodiment can be implemented in a distributed manner by a group of similar platforms, thereby distributing the processing load of the computer system.

[0191] As shown in FIG7 , the computer system may include a communication port 250 connected to a network for data communication. For example, the computer system can send and receive information and data via the communication port 250. That is, the communication port 250 enables the computer system to communicate with other electronic devices wirelessly or wired to exchange data. The computer system may also include a processor group 220 (i.e., the processor described above) for executing program instructions. The processor group 220 may be composed of at least one processor (e.g., a CPU). The computer system may include an internal communication bus 210. The computer system may include various forms of program storage units and data storage units (i.e., the memory or storage media described above), such as a hard disk 270, a read-only memory (ROM) 230, and a random access memory (RAM) 240, which can be used to store various data files used for computer processing and / or communication, as well as possible program instructions executed by the processor group 220. The computer system may also include an input / output component 260, which is used to implement input / output data flow between the computer system and other components (e.g., a user interface 280, etc.).

[0192] Typically, the following devices can be connected to the input / output component 260: input devices such as a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices such as a display (e.g., an LCD, an OLED display, etc.), a speaker, a vibrator, etc.; storage devices such as a tape, a hard disk, etc.; and a communication interface.

[0193] Although FIG. 7 shows a computer system having various devices, it should be understood that the computer system is not required to have all of the devices shown. Alternatively, the computer system may have more or fewer devices.

[0194] Those skilled in the art will appreciate that the contents disclosed in this disclosure may be subject to various modifications and improvements. For example, the various devices or components described above may be implemented through hardware, software, firmware, or a combination of some or all of the three.

[0195] In addition, although the present disclosure makes various references to certain units in the system according to embodiments of the present disclosure, any number of different units can be used and run on the client and / or server. The units are only illustrative, and different aspects of the system and method can use different units.

[0196] Flowcharts are used in this disclosure to illustrate the steps of the methods according to the embodiments of the present disclosure. It should be understood that the preceding or following steps do not necessarily need to be performed in exact order. Instead, the various steps may be performed in reverse order or simultaneously. Furthermore, other operations may be added to these processes.

[0197] Those skilled in the art will appreciate that all or part of the steps in the above method can be performed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk. Alternatively, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software functional modules. The present disclosure is not limited to any particular combination of hardware and software.

[0198] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and should not be interpreted in an idealized or highly formal sense unless expressly defined as such herein.

[0199] The above is an illustration of the present disclosure and should not be considered as a limitation thereof. Although several exemplary embodiments of the present disclosure have been described, it will be readily understood by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure as defined by the claims. It should be understood that the above is an illustration of the present disclosure and should not be considered as limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present disclosure is defined by the claims and their equivalents.< / eos> < / eos> < / bos> < / ss> < / pad>

Claims

1. A data processing method, comprising: The neural network is used to perform reasoning operations on the data to be processed, and the reasoning operations include: Using the neural network to perform M rounds of reasoning on the data to be processed in sequence, wherein the data to be processed includes multiple batches of input data, each batch of input data includes N1 input data elements, and in each round of reasoning, only all reasoning of one batch of input data is completed, and M and N1 are positive integers greater than 1; The M rounds of reasoning include the i-th round of reasoning and the i+1-th round of reasoning performed sequentially, where i is a positive integer and is less than M. The reasoning process of the i-th round of reasoning includes: In the first stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N2 input data elements. In the second stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N1 input data elements to complete all reasoning of the N1 input data elements belonging to the same batch of input data; The first stage is earlier than the second stage, N2 is greater than N1, and the N1 input data elements belonging to the same batch of input data are a subset of the N2 input data elements; the reasoning process of the i+1th round of reasoning includes: Reasoning is performed on the N2-N1 input data elements in the i-th round of reasoning in combination with the inferred output shapes of the N2-N1 input data elements that have not been reasoned yet.

2. The data processing method according to claim 1, wherein: The neural network includes a first sub-network and a second sub-network, The first sub-network is configured to receive an input data element and extract a semantic feature corresponding to the input data element in combination with a self-attention mechanism, wherein the semantic feature includes an initial key tensor and an initial value tensor, The second sub-network is configured to predict the output shape sequence corresponding to the input data element one by one based on the semantic features in combination with an autoregressive mechanism, In which, the output shape sequence corresponding to the input data elements is predicted one by one by combining the autoregressive mechanism In the process of listing, the initial key tensor and the initial value tensor are continuously updated according to the predicted output shapes to obtain updated key tensor and updated value tensor, and the lengths of the updated key tensor and the updated value tensor are directly proportional to the number of output shapes corresponding to the predicted input data elements.

3. The data processing method according to claim 2, wherein: N1 is determined by the number of tokens in the longest output token sequence supported by the neural network, the maximum length of input data elements supported by the neural network, and the size of the storage space used to store the update key tensor and the update value tensor.

4. The data processing method according to any one of claims 1 to 3, wherein: In the first stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N2 input data elements, including: Determine X batches of input data as the N2 input data elements, wherein the X batches of input data include a first batch of input data, the first batch of input data includes the N1 input data elements that have been completely inferred in the i-th round of inference, and X is a positive integer greater than 1; Inference is performed on the X batches of input data simultaneously in batches.

5. The data processing method according to claim 4, wherein: Determine X batches of input data as the N2 input data elements, including: In response to i being equal to 1, determining X batches of input data that have not been completely reasoned about as the X batches of input data; In response to i being greater than 1: Determine X-1 batches of input data that have not been completely reasoned in the i-1th round of reasoning, and a batch of input data that has not been completely reasoned, and use them together as the X batches of input data, wherein the first batch of input data is determined from the X-1 batches of input data that have not been completely reasoned in the i-1th round of reasoning.

6. The data processing method according to claim 4, wherein: X-1 batches of input data other than the first batch of input data in the X batches of input data have respective corresponding reasoning suspension thresholds, Performing reasoning on the X batches of input data simultaneously in batches, including: The X batches of input data are inferred simultaneously, wherein, for any batch of input data in the X-1 batches of input data, when the number of output shapes inferred from the any batch of input data reaches a corresponding inference suspension threshold, the inference of the any batch of input data is suspended.

7. The data processing method according to claim 6, wherein: In response to X=2, the inference suspension threshold is (L-2P) / 3, where P is the number of The maximum length of the input text corresponding to a batch of input data, L is the sum of the number of glyphs in the longest output glyph sequence supported by the neural network and P, and " / " represents division.

8. The data processing method according to claim 6 or 7, wherein: When the number of output tokens inferred from any batch of input data reaches a corresponding inference suspension threshold, suspending the inference of any batch of input data comprises: suspending the reasoning of any batch of input data; Storing output shapes inferred from any batch of input data; Clear the update key tensor and update value tensor generated by any batch of input data during the i-th round of reasoning.

9. The data processing method according to claim 8, wherein: Storing the output shape obtained by reasoning any batch of input data, including: Convert each output shape in the output shape obtained by inference from any batch of input data into a corresponding shape index value; Store the shape index values ​​corresponding to the output shapes inferred from any batch of input data.

10. The data processing method according to any one of claims 6 to 9, wherein: Performing reasoning on the X batches of input data simultaneously includes: In response to i being greater than 1, for the other X-1 batches of input data except for one batch of input data that has not been reasoned about at all among the X batches of input data: In combination with the inferred output shapes of the N2-N1 input data elements that have not been inferred in the i-1th round of reasoning, the other X-1 batches of input data are inferred.

11. The data processing method according to any one of claims 1 to 10, wherein: In the second stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N1 input data elements to complete all reasoning of the N1 input data elements belonging to the same batch of input data, including: The first batch of input data continues to be inferred until the inference of all input data elements in the first batch of input data is completed, wherein the first batch of input data includes the N1 input data elements that have been inferred in the i-th round of inference.

12. The data processing method according to any one of claims 1 to 11, wherein: In combination with the inferred output shapes of the N2-N1 input data elements that have not been inferred in the i-th round of reasoning, reasoning is performed on the N2-N1 input data elements, including: Get the inferred values ​​of the N2-N1 input data elements in the i-th round of reasoning Outputting shapes, wherein the inferred output shapes of the N2-N1 input data elements are stored in a storage space in the form of shape index values; Inference is performed on the N2-N1 input data elements in batches, wherein, for each of the N2-N1 input data elements, the output shape obtained by inference of the input data element and the input data element are input into the neural network for inference.

13. The data processing method according to claim 12, wherein: Inputting the obtained output shape obtained by inference of the input data element and the input data element into the neural network for inference, including: The shape symbol index value corresponding to the output shape symbol inferred from the input data element and the shape symbol index value corresponding to the input data element are concatenated, and the concatenated result is input into the neural network for inference.

14. The data processing method according to claim 12 or 13, wherein: The neural network includes a first sub-network for performing prompt reasoning and a second sub-network for performing shape reasoning, While performing prompt reasoning on the N2-N1 input data elements through the first sub-network, performing prompt reasoning on a batch of input data that has not been completely reasoned through the first sub-network, and After the prompt reasoning is completed for the N2-N1 input data elements and the batch of input data that has not been completely reasoned, the second sub-network is used to simultaneously start to perform shape reasoning on the N2-N1 input data elements and the batch of input data that has not been completely reasoned.

15. The data processing method according to any one of claims 1 to 14, wherein: N2 is an integer multiple of N1.

16. A data processing device, comprising: an inference unit, configured to perform inference operations on the processed data through a neural network, The reasoning operation includes: Using the neural network to perform M rounds of reasoning on the data to be processed in sequence, wherein the data to be processed includes multiple batches of input data, each batch of input data includes N1 input data elements, and in each round of reasoning, only all reasoning of one batch of input data is completed, and M and N1 are positive integers greater than 1; The M rounds of reasoning include the i-th round of reasoning and the i+1-th round of reasoning performed sequentially, where i is a positive integer and is less than M. The reasoning process of the i-th round of reasoning includes: In the first stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N2 input data elements. In the second stage of the reasoning process of the i-th round of reasoning, reasoning is performed on N1 input data elements to complete all reasoning of the N1 input data elements belonging to the same batch of input data; The first stage is earlier than the second stage, N2 is greater than N1, and the N1 input data elements belonging to the same batch of input data are a subset of the N2 input data elements; the reasoning process of the i+1th round of reasoning includes: Reasoning is performed on the N2-N1 input data elements in the i-th round of reasoning in combination with the inferred output shapes of the N2-N1 input data elements that have not been reasoned yet.

17. The data processing device according to claim 16, further comprising a storage unit, The storage unit is configured to store weight parameters of the neural network and update key tensors and update value tensors generated during the inference operation.

18. An electronic device comprising: A memory non-transitorily stores computer executable instructions; a processor configured to execute the computer executable instructions, Wherein, when the computer executable instructions are executed by the processor, the data processing method according to any one of claims 1-15 is implemented.

19. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the data processing method according to any one of claims 1 to 15 is implemented.

Citation Information

Patent Citations

  • Data reasoning method, device, server and medium based on knowledge map

    CN109299283A

  • Neural network model training method and device, equipment and storage medium

    CN115238883A

  • Data processing method and device, electronic equipment and storage medium

    CN116579373A

  • Dynamically resizing minibatch in neural network execution

    US20200311536A1

  • Method and apparatus with neural network inference optimization implementation

    US20220083838A1