Data processing method, device and system, computing device and storage medium

Abstract

Embodiments of the invention provide a data processing method, device and system, a computing device and a storage medium, the data processing method is applied to a distributed device, the distributed device comprises a plurality of computing nodes, the plurality of computing nodes are deployed with task processing models, and the method comprises the steps of receiving a plurality of text sequences; segmenting the plurality of text sequences to obtain a plurality of text sequence units; sequentially polling and distributing the plurality of text sequence units to the plurality of computing nodes; and calling the task processing model according to the plurality of computing nodes, predicting the plurality of text sequence units, and obtaining a prediction result corresponding to each text sequence.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to data processing methods, devices and systems, computing devices and storage media. Background Technology

[0002] In practical applications, artificial intelligence models have shown good performance in artificial intelligence tasks such as natural language processing and multimodal reasoning. However, the number of parameters and activation values ​​of artificial intelligence models increases dramatically with the length of the input text sequence, which places higher demands on the memory capacity, bandwidth and computing power of the computing devices that deploy artificial intelligence models. This affects the computing power and reasoning performance of artificial intelligence models deployed on computing devices. Therefore, an effective technical solution is urgently needed to solve the above problems. Summary of the Invention

[0003] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a data processing system, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a data processing method is provided, applied to a distributed device, the distributed device including multiple computing nodes, the multiple computing nodes being deployed with a task processing model, the method comprising: Receive multiple text sequences; The multiple text sequences are segmented to obtain multiple text sequence units; The multiple text sequence units are sequentially and round-robin distributed to the multiple computing nodes; The task processing model is invoked by the multiple computing nodes to predict the multiple text sequence units and obtain the prediction results corresponding to each text sequence.

[0005] According to a second aspect of the embodiments of this specification, a data processing device is provided, including a scheduling node and a plurality of computing nodes, wherein the plurality of computing nodes are deployed with a task processing model, wherein... The scheduling node is configured to receive multiple text sequences; segment the multiple text sequences to obtain multiple text sequence units; and sequentially distribute the multiple text sequence units to the multiple computing nodes. The plurality of computing nodes are configured to invoke the task processing model to predict the plurality of text sequence units and obtain the prediction results corresponding to each text sequence.

[0006] According to a third aspect of the embodiments of this specification, a data processing system is provided, comprising: Processor core; A system-on-a-chip (SoC) includes at least one computing node, which can communicate with each other, and the SoC executes the steps of the above-described data processing method.

[0007] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0008] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0009] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0010] This specification provides a data processing method in one embodiment, applied to a distributed device. The distributed device includes multiple computing nodes, each of which is deployed with a task processing model. The method includes: receiving multiple text sequences; segmenting the multiple text sequences to obtain multiple text sequence units; sequentially distributing the multiple text sequence units to the multiple computing nodes; and, based on the multiple computing nodes, invoking the task processing model to predict the multiple text sequence units, thereby obtaining a prediction result corresponding to each text sequence.

[0011] In the above method, the task processing model can be deployed on multiple computing nodes of a distributed device. After receiving multiple text sequences, the multiple text sequences can be segmented to obtain multiple text sequence units, and these multiple text sequence units can be distributed to multiple computing nodes in a round-robin fashion. This achieves balanced distribution of multiple text sequences across multiple computing nodes, ensuring load balancing of the distributed device and balanced consumption of computing resources across multiple computing nodes. Based on the task processing model invoked by multiple computing nodes, predictions are made on multiple text sequence units to obtain prediction results for each text sequence. This enables parallel inference prediction of multiple text sequences, thereby ensuring the computing power and inference performance of the task processing model deployed on the distributed device. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a data processing method provided in one embodiment of this specification; Figure 2 This is a schematic diagram of the first allocation method in a data processing method provided in one embodiment of this specification; Figure 3 This is a schematic diagram of the second allocation method in a data processing method provided in one embodiment of this specification; Figure 4 This is a flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification. Figure 5 This is a schematic diagram of the structure of a data processing device provided in one embodiment of this specification. Detailed Implementation

[0013] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0014] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0015] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0016] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0017] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0018] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0019] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0020] Transformer: A deep learning architecture based on self-attention mechanism.

[0021] KV: Key-Value pairs. In the multi-head self-attention of Transformers, each token is transformed into three vectors through a linear transformation. The Key represents what queries this token can be matched with, and the Value represents the information actually contained in this token. During the autoregressive generation process, the K and V of historical tokens are cached as key-value pairs.

[0022] RoPE: Rotary Position Embedding, is a position encoding method used to enable Transformer-based models to perceive the order information of tokens in a sequence.

[0023] Distributed devices: Clusters of multiple computing nodes. This can be a single chip with multiple independent computing units and memory, or a distributed computing cluster composed of multiple computing chips.

[0024] Key-value cache memory pool: A memory pool in a distributed device where multiple computing nodes share the storage of key-value caches.

[0025] Text sequence unit: Sequence block, a key-value cache consisting of fixed-length tokens.

[0026] Shared prefix cache: When a Transformer-class model receives different inputs, it caches key-value pairs from the first token up to the first unequal token. This is used to quickly reuse cached content, thereby reducing the overhead of repeated calculations or data retrieval.

[0027] NUMA: Non-Uniform Memory Access, is a memory architecture design for multiprocessor systems.

[0028] In practical applications, large language models based on the Transformer structure have shown good performance in artificial intelligence tasks such as natural language processing and multimodal reasoning. However, the number of parameters and activation values ​​increase dramatically with the length of the inference sequence, posing a great challenge to the memory capacity, bandwidth and parallel computing capabilities of computing devices.

[0029] Common parallel inference methods include tensor parallelism, pipelined parallelism, and data parallelism. However, these existing parallel inference deployment methods largely rely on the model's hyperparameters, such as the number of self-attention heads and the number of model layers. This means the number of distributed devices often needs to be divided by these hyperparameters, limiting the flexibility of device deployment. Furthermore, with the development of large language model technology, the length of inference sequences is growing exponentially, leading to an increasing demand for inference with extremely long sequences on large models. Therefore, an effective technical solution is urgently needed to address these issues.

[0030] This specification provides a data processing method, and also relates to a data processing device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0031] See Figure 1 , Figure 1 A flowchart of a data processing method according to an embodiment of this specification is shown. The data processing method is applied to a distributed device, which includes multiple computing nodes and each computing node is equipped with a task processing model. The data processing method specifically includes the following steps.

[0032] Step 102: Receive multiple text sequences.

[0033] In this context, the text sequence can be understood as the text sequence to be processed. The text sequence can serve as input to the task processing model, which can be a model for processing text tasks, such as text generation, text parsing, or text retrieval. The task processing model can be a trained machine learning model, a deep learning model, or a large language model. Multiple computing nodes of the distributed device can invoke this task processing model to execute the text task. Optionally, the task processing model can be a model with more than a preset parameter threshold. Based on this, the task processing model can be deployed across multiple computing nodes of the distributed device. In practical applications, the distributed device can be a cluster of multiple computing nodes. It can be a chip with multiple independent computing units and memory, which can be a NUMA memory-based chip, where each independent computing unit is a computing node. Alternatively, it can be a distributed computing cluster composed of multiple computing chips, where each computing chip is a computing node.

[0034] Based on this, multiple text sequences to be processed can be received.

[0035] In practical applications, multiple text sequences can be segmented from an initial text sequence corresponding to a text processing request. This initial text sequence can be a text sequence with a length greater than a preset length threshold, such as an 8000-word article. Alternatively, a text processing request can correspond to a text sequence, and the multiple received text sequences can be determined based on the received text processing requests. For example, multiple text sequences could include: Text Sequence 1 "Generate an essay of no less than 800 words, titled 'My Dream'", Text Sequence 2 "Travel destinations to visit in October; summarize a travel guide for me", and Text Sequence 3 "Analyze whether there are any problems with this contract; contract text XXXX". Understandably, the text sequences can include prompt text input to the task processing model. Step 104: Segment the multiple text sequences to obtain multiple text sequence units.

[0036] In this context, a text sequence unit can be understood as a block of text obtained by segmenting a text sequence. This block can contain multiple character units, which can be understood as tokens. A token is the basic unit that the task processing model uses to process the text sequence. A token can be the smallest character unit recognized by the task processing model; for example, a token can be a word, a character, a punctuation mark, a number, or a single character. When the text sequence is Chinese, the token can be a character; when the text sequence is English, the token can be a word. For example, in the text sequence "Help me generate an essay of no less than 800 words, with the title 'My Dream'", "help" can be a token, and "," can also be a token.

[0037] In practical applications, when obtaining the input text of a task processing model, a tokenizer can be used to convert the input text into a text sequence containing multiple tokens.

[0038] Based on this, each text sequence can be segmented to obtain multiple text sequence units, and each text sequence unit can include multiple character units.

[0039] For example, if three text sequences are received, namely text sequence A, text sequence B, and text sequence C, then text sequence A can be segmented to obtain multiple text sequence units A1, A2, and A3; text sequence B can be segmented to obtain multiple text sequence units B1 and B2; and text sequence C can be segmented to obtain multiple text sequence units C1, C2, C3, and C4.

[0040] In specific implementation, the process of segmenting the multiple text sequences to obtain multiple text sequence units includes: The multiple text sequences are segmented according to a fixed length to obtain multiple text sequence units.

[0041] The fixed length can be understood as a pre-set fixed size of the text sequence unit. For example, the fixed length can be 10, which means that the length of each text sequence unit is 10, and that the text sequence unit contains 10 tokens.

[0042] Based on this, multiple text sequences can be segmented according to a pre-set fixed length, dividing each text sequence into multiple fixed-length text sequence units.

[0043] In practical applications, the fixed length can be determined based on the device attribute information of the distributed device, the model parameters of the task processing model, or manually set based on historical processing experience. This specification does not impose any limitations on this. Furthermore, the task processing model can be a large language model based on the Transformer architecture. Under the Transformer architecture, each text sequence unit (i.e., token block) can be used to store the key-value cache corresponding to the text sequence segment.

[0044] Continuing with the previous example, let's take text sequence A out of three text sequences as an example. Text sequence A can be divided into three text sequence units of fixed length 10: A1, A2, and A3. Understandably, a similar segmentation operation is performed on each text sequence, so it will not be repeated here.

[0045] In summary, by segmenting the text sequence according to a fixed length, each sequence adopts a fixed length as the granularity, which facilitates the subsequent allocation of multiple text sequence units to multiple computing nodes. This enables the subsequent polling and rearrangement of multiple text sequence units, and further achieves a balanced distribution among multiple computing nodes.

[0046] Furthermore, after receiving multiple text sequences, it also includes... Determine the sequence length information of each text sequence; The step of segmenting the multiple text sequences according to a fixed length to obtain multiple text sequence units includes: Based on the sequence length information of each text sequence and the fixed length, determine the number of segments corresponding to each text sequence; According to the stated number of segments, each text sequence is segmented to obtain multiple text sequence units; If a blank is found in any text sequence unit, padding data is added to the text sequence unit.

[0047] The sequence length information of a text sequence can be understood as its original length. For example, if the original length of a text sequence is 70, it means that the text sequence contains 70 tokens. The presence of blank spaces in a text sequence unit can be understood as a unit that cannot be filled with valid data. For example, if the original length of a text sequence is 50 and the fixed length is 20, then the number of segments is 3. The text sequence can be divided into 3 units. The first two units can be filled with valid data, while in the third unit, the first 10 segments contain valid data, but the last 10 segments contain no valid data. In this case, padding data can be added to these 10 segments. Padding data can be understood as invalid data without actual meaning, such as a value of 0.

[0048] Specifically, after receiving multiple text sequences, the original length of each text sequence can be determined, and the number of segments corresponding to each text sequence can be calculated based on the original length and fixed length of each text sequence. Each text sequence is then segmented according to the number of segments corresponding to each text sequence to obtain multiple text sequence units. Furthermore, if any text sequence unit contains blanks, padding data is added to the blank parts of that text sequence unit.

[0049] In practical applications, distributed devices, as inference frameworks, can include scheduling nodes (i.e., schedulers). Scheduling nodes can divide each input text sequence into fixed-size blocks (i.e., fixed-length text sequence units). The fixed size of these blocks can be determined based on the characteristics of different hardware. Furthermore, the last text sequence block of each text sequence may not be completely filled with valid data. Based on this, scheduling nodes can record the original length of each text sequence.

[0050] In summary, by determining the original length of each text sequence, fixed-length text sequence units are segmented for each text sequence, which facilitates the subsequent distribution of text sequence units to multiple computing nodes and further achieves load balancing for each computing node.

[0051] In an optional embodiment of this specification, after segmenting the plurality of text sequences to obtain a plurality of text sequence units, the method further includes: From the plurality of text sequence units, determine the text sequence unit corresponding to each text sequence; The text sequence units corresponding to the first text sequence are sequentially and round-robin distributed to the plurality of computing nodes, wherein the first text sequence is any one of the plurality of text sequences; The text sequence units corresponding to the next text sequence of the first text sequence are sequentially and round-robinly distributed to the plurality of computing nodes until the text sequence units corresponding to the first text sequence are distributed to the plurality of computing nodes.

[0052] Specifically, when allocating multiple text sequence units to multiple computing nodes, the multiple text sequence units corresponding to each text sequence can be allocated to multiple computing nodes in a round-robin fashion. That is to say, after allocating the text sequence units corresponding to the first text sequence to multiple computing nodes in a round-robin fashion, starting from the first computing node, the text sequence units corresponding to the next text sequence of the first text sequence are allocated to multiple computing nodes in a round-robin fashion, and so on, until all text sequence units corresponding to all text sequences are allocated to multiple computing nodes.

[0053] For example, see Figure 2 , Figure 2 A schematic diagram of a first allocation method in a data processing method according to an embodiment of this specification is shown. Figure 2 As shown, text sequence D has a length of 70, text sequence E has a length of 90, and text sequence F has a length of 110. These three text sequences are segmented to obtain multiple text sequence units. Text sequence D is segmented to obtain text sequence units D1, D2, D3, D4, and D5, where text sequence unit D5 contains padding data. Text sequence E is segmented to obtain text sequence units E1, E2, E3, E4, E5, and E6. Text sequence F is segmented to obtain text sequence units F1, F2, F3, F4, F5, F6, and F7. According to the above allocation method, for text sequence D, text sequence unit D1 is allocated to computing node 0, text sequence unit D2 is allocated to computing node 1, text sequence unit D3 is allocated to computing node 2, text sequence unit D4 is allocated to computing node 3, and text sequence unit D5 is allocated to computing node 0. This completes the round-robin allocation of text sequence D. Next, for text sequence E, text sequence unit E1 is assigned to computing node 0, text sequence unit E2 to computing node 1, text sequence unit E3 to computing node 2, text sequence unit E4 to computing node 3, text sequence unit E5 to computing node 0, and text sequence unit E6 to computing node 1, thus completing the round-robin allocation of text sequence E. Then, following a similar process, text sequence F is allocated, with text sequence unit F1 assigned to computing node 0, and so on. This process will not be repeated here.

[0054] In summary, by sequentially polling and allocating multiple text sequence units corresponding to each text sequence, the rearrangement of multiple text sequence units is achieved, thereby ensuring the balance of multiple text sequences being allocated to multiple computing nodes.

[0055] Step 106: Distribute the multiple text sequence units to the multiple computing nodes in a round-robin fashion.

[0056] Specifically, when allocating text sequences, a short text sequence may be assigned to a computing node in a distributed device, resulting in a significant waste of computing resources on other computing nodes. This makes it difficult to ensure load balancing in the distributed device and further affects the deployment efficiency and inference performance of artificial intelligence models. Based on this, when polling and allocating multiple text sequence units, we can also start directly from the first computing node and poll and allocate multiple text sequence units to multiple computing nodes in turn until all text sequence units have been allocated.

[0057] In practical applications, the scheduling node can reorganize the text sequence units of each text sequence, and deploy one text sequence unit at a time starting from the first computing node. After polling all computing nodes, the next text sequence unit is deployed to the first computing node again, until the last text sequence unit is deployed.

[0058] For example, see Figure 3 , Figure 3 A schematic diagram of a second allocation method in a data processing method according to an embodiment of this specification is shown, such as... Figure 3 As shown, after dividing text sequences D, E, and F into multiple text sequence units, starting from the first computing node 0, text sequence unit D1 is assigned to computing node 0, text sequence unit D2 is assigned to computing node 1, text sequence unit D3 is assigned to computing node 2, text sequence unit D4 is assigned to computing node 3, text sequence unit D5 is assigned to computing node 0, text sequence unit E1 is assigned to computing node 1, text sequence unit E2 is assigned to computing node 2, text sequence unit E3 is assigned to computing node 3, text sequence unit E4 is assigned to computing node 0, text sequence unit E5 is assigned to computing node 1, text sequence unit E6 is assigned to computing node 2, text sequence unit F1 is assigned to computing node 3, and so on, until all text sequence units of the three text sequences are assigned to computing nodes.

[0059] Furthermore, in this allocation method, the scheduling node can record the starting position of the computing node to which each text sequence unit is assigned, i.e., the position of the computing node to which the first text sequence unit of the text sequence is assigned. For example, the starting position of the computing node to which text sequence D is assigned is computing node 0, while the starting position of the computing node to which text sequence E is assigned is computing node 1, and the starting position of the computing node to which text sequence F is assigned is computing node 3. This allows the task processing model to determine the computing node assigned to each text sequence and the sub-text sequences of the text sequence included in each computing node when it is subsequently invoked for inference. In practical applications, the node identifier information of the computing node to which the first text sequence unit corresponding to each text sequence is assigned can be recorded.

[0060] Step 108: Based on the multiple computing nodes, call the task processing model to predict the multiple text sequence units and obtain the prediction results corresponding to each text sequence.

[0061] Specifically, after allocating multiple text sequence units to multiple computing nodes, the task processing model can be invoked according to the multiple computing nodes to perform self-attention calculation on each token included in the multiple task sequence units. After obtaining the self-attention calculation result, residual connection processing, normalization processing, nonlinear transformation processing, feature enhancement processing and other steps are continued to obtain the prediction result corresponding to each text sequence.

[0062] Understandably, the target computing node includes sub-text sequences corresponding to each text sequence. The target sub-text sequence is composed of target text sequence units corresponding to the target text sequence. The target text sequence unit is the text sequence unit corresponding to the target text sequence and assigned to the target computing node among the plurality of text sequence units. The target computing node is any one of the plurality of computing nodes, the target text sequence is any one of the plurality of text sequences, and the target sub-text sequence is any one of the sub-text sequences corresponding to each text sequence.

[0063] Specifically, for one of the multiple computing nodes (i.e., the target computing node), after the aforementioned text sequence unit allocation, the target computing node can include sub-text sequences corresponding to each text sequence. Then, within the target computing node, for one of the sub-text sequences (i.e., the target sub-text sequence), the task processing model can be invoked to perform self-attention calculation on the target sub-text sequence, obtaining the self-attention calculation result corresponding to that target sub-text sequence. The output of the target computing node is then the self-attention calculation result of the sub-text sequence assigned to it for each text sequence. It can be understood that when the task processing model predicts the target sub-text sequence, it may include steps such as performing self-attention calculation, residual connection processing, normalization processing, nonlinear transformation processing, and feature enhancement processing on the target sub-text sequence. The aforementioned self-attention calculation of the target sub-text sequence can be understood as part of the computational steps in the task processing model's prediction of the target sub-text sequence.

[0064] For one of the multiple text sequences (i.e., the target text sequence), this target text sequence is assigned to multiple computing nodes. Each computing node includes a sub-text sequence corresponding to the target text sequence. The self-attention calculation result corresponding to the target text sequence can be determined based on the self-attention calculation result of the sub-text sequences output by each computing node. Furthermore, the target text sequence is subjected to the other processing described above through a task processing model, and finally, the prediction result corresponding to the target text sequence is obtained.

[0065] For example, taking multiple text sequences, including text sequence A and text sequence B, and assigning them to two computing nodes as an example, text sequence A is divided into blocks to obtain multiple text sequence units A1, A2, and A3 corresponding to text sequence A, and text sequence B is divided into blocks to obtain multiple text sequence units B1 and B2 corresponding to text sequence B.

[0066] In computing node 1, text sequence units A1 and A3 of text sequence A and text sequence unit B1 of text sequence B are allocated; in computing node 2, text sequence unit A2 of text sequence A and text sequence unit B2 of text sequence B are allocated.

[0067] Therefore, computation node 1 includes sub-text sequence 1A and sub-text sequence 1B. Sub-text sequence 1A is composed of text sequence units A1 and A3, and sub-text sequence 1B is composed of text sequence unit B1. Computation node 1 can then perform self-attention calculation on sub-text sequence 1A and sub-text sequence 1B, obtaining the self-attention calculation results for sub-text sequence 1A and sub-text sequence 1B, respectively, output by computation node 1.

[0068] Computation node 2 includes sub-text sequence 2A and sub-text sequence 2B. Sub-text sequence 2A is composed of text sequence unit A2, and sub-text sequence 2B is composed of text sequence unit B2. Then, computation node 2 can perform self-attention calculation on sub-text sequence 2A and sub-text sequence 2B to obtain the self-attention calculation results corresponding to sub-text sequence 2A and sub-text sequence 2B output by computation node 2.

[0069] Furthermore, the self-attention calculation result corresponding to the text sequence A can be determined based on the self-attention calculation result of the sub-text sequence 1A output by computing node 1 and the self-attention calculation result corresponding to the sub-text sequence 2A output by computing node 2. Similarly, the self-attention calculation result corresponding to the text sequence B can be determined based on the self-attention calculation result of the sub-text sequence 1B output by computing node 1 and the self-attention calculation result of the sub-text sequence 2B output by computing node 2.

[0070] Furthermore, the task processing model is invoked again to perform residual connection processing, normalization processing, nonlinear transformation processing, and feature enhancement processing based on the self-attention calculation results, in order to obtain the prediction results corresponding to the target text sequence.

[0071] In specific implementation, the step of calling the task processing model based on the multiple computing nodes to predict the multiple text sequence units and obtain the prediction result corresponding to each text sequence includes: The task processing model is invoked by the multiple computing nodes to perform self-attention calculation on the sub-text sequences corresponding to each text sequence, thereby obtaining the self-attention calculation results output by each computing node for each sub-text sequence. The self-attention calculation results corresponding to each text sequence are determined based on the self-attention calculation results corresponding to each sub-text sequence.

[0072] Specifically, among multiple computing nodes, each computing node can call the task processing model to perform encoding and self-attention calculations on the sub-text sequences assigned to each computing node for each text sequence, and obtain the self-attention calculation results corresponding to each sub-text sequence output by each computing node. Based on the self-attention calculation results corresponding to each sub-text sequence output by each computing node, the self-attention calculation result corresponding to each text sequence is determined.

[0073] In practice, positional encoding and self-attention calculation can be performed on each sub-text sequence.

[0074] In practical applications, since task processing models involve positional encoding calculations such as RoPE, the scheduling node can determine the starting position information of each text sequence unit within each text sequence. This facilitates the correct calculation of positional encoding for each sub-text sequence by the subsequent task processing model. The scheduling node can then send this starting position information of each text sequence unit to multiple computing nodes. These computing nodes can then use this starting position information to perform positional encoding and self-attention calculations when subsequently calling the task processing model.

[0075] In summary, by utilizing multiple computing nodes to encode and calculate each sub-text sequence corresponding to each text sequence, sub-prediction results corresponding to each sub-text sequence are obtained, facilitating the subsequent acquisition of prediction results for each text sequence. Furthermore, by preserving the starting position information of each text sequence unit, the original position of each token can be restored during subsequent calculations, ensuring the correctness of position encoding calculations.

[0076] Further, the step of calling the task processing model according to the multiple computing nodes to perform self-attention calculation on the sub-text sequences corresponding to each text sequence, and obtaining the self-attention calculation results corresponding to each sub-text sequence output by each computing node, includes: The task processing model is invoked by the multiple computing nodes to perform self-attention calculation on each sub-text sequence corresponding to the target text sequence, thereby obtaining the self-attention calculation results output by each computing node for each sub-text sequence. The step of determining the self-attention calculation result corresponding to each text sequence based on the self-attention calculation result corresponding to each sub-text sequence includes: The self-attention calculation result corresponding to the target text sequence is determined based on the self-attention calculation results output by each computing node and corresponding to each sub-text sequence.

[0077] Specifically, taking the example of multiple computing nodes encoding and calculating the sub-text sequences corresponding to one of multiple text sequences (i.e., the target text sequence) as an example, after distributing the text sequence units corresponding to the target text sequence to multiple computing nodes, each computing node includes the sub-text sequences corresponding to the target text sequence. The combination of the sub-text sequences included in all computing nodes can obtain the target text sequence. That is to say, one computing node corresponds to one sub-text sequence, and one computing node outputs the self-attention calculation result of one sub-text sequence. Then, each computing node can perform self-attention calculation on each sub-text sequence corresponding to the target text sequence, obtaining the self-attention calculation result output by each computing node for each sub-text sequence. Based on the self-attention calculation results output by each computing node for each sub-text sequence, the self-attention calculation result corresponding to the target text sequence is determined.

[0078] Understandably, the above processing is performed on each of the multiple text sequences to obtain the self-attention calculation result corresponding to each text sequence.

[0079] In specific implementation, the step of calling the task processing model according to the multiple computing nodes to perform self-attention calculation on each sub-text sequence corresponding to the target text sequence, and obtaining the self-attention calculation results corresponding to each sub-text sequence output by each computing node, includes: The target computing node calls the task processing model to perform self-attention calculation on the target sub-text sequence and obtain the initial calculation result corresponding to the target sub-text sequence. The association sequence information corresponding to the associated sub-text sequence corresponding to the target text sequence in the association computing node is sent to the target computing node. Attention calculation is performed on the initial calculation result and the association sequence information by the target computing node to obtain the target self-attention calculation result corresponding to the target sub-text sequence output by the target computing node. The associated computing node is any computing node other than the target computing node among the plurality of computing nodes.

[0080] A related subtext sequence can be understood as a subtext sequence of the target text sequence assigned to the related computing node. This related subtext sequence and the target subtext sequence in the target computing node belong to the same target text sequence.

[0081] Specifically, taking the encoding calculation of a sub-text sequence (i.e., the target sub-text sequence) of a text sequence by one of the computing nodes (i.e. the target computing node) as an example, after the target computing node is assigned the sub-text sequence, it can perform self-attention calculation on each token in the sub-text sequence to obtain the self-attention calculation result corresponding to each token. The self-attention calculation result can be used as a key-value cache and cached in the key-value cache memory pool in the target computing node.

[0082] In practical applications, the key-value cache for a token can be the Key vector (key) and Value vector (value) from the token's self-attention calculation result. Combining the above... Figure 2 and Figure 3 When performing self-attention calculation on each token in the sub-text sequence, the self-attention calculation can be performed on the first token first to obtain the self-attention calculation result corresponding to the first token, and this self-attention calculation result is determined as the key-value cache corresponding to the first token. When calculating the self-attention calculation result corresponding to the second token, the key-value cache corresponding to the first token can be obtained, and self-attention calculation can be performed on the key-value cache corresponding to the first token and the second token to obtain the self-attention calculation result corresponding to the second token, and this self-attention calculation result is determined as the key-value cache corresponding to the second token. This process is repeated to calculate the key-value cache corresponding to each token and cache it in the key-value cache memory pool in the target node.

[0083] Understandably, after each computing node is assigned to a sub-text sequence, it can perform the above encoding calculation on each token in each sub-text sequence to generate a key-value cache corresponding to each token and cache it in the key-value cache memory pool of each computing node. Then, the association sequence information in the associated computing node can be understood as the key-value cache (i.e., the self-attention calculation result) calculated by the associated computing node for each token in the sub-text sequence.

[0084] Based on this, the target computing node can invoke the task processing model to perform self-attention calculation on each token in the target sub-text sequence, obtaining the self-attention calculation result corresponding to each token in the target sub-text sequence as the initial calculation result (i.e., key-value cache). The target computing node communicates with associated computing nodes, retrieving the key-value cache corresponding to the associated sub-text sequence of the target text sequence from the associated computing nodes. The target computing node continues to perform self-attention calculation on the initial calculation result and the key-value cache, finally obtaining the target self-attention calculation result corresponding to the target sub-text sequence. When there are multiple associated computing nodes, the target computing node can retrieve the key-value cache corresponding to the associated sub-text sequence of the target text sequence from each associated computing node, and the target computing node continues to perform self-attention calculation on the initial calculation result and the key-value cache, finally obtaining the target self-attention calculation result corresponding to the target sub-text sequence.

[0085] In one embodiment of this specification, the process of calculating the initial calculation result in the target computing node and the process of the target computing node obtaining the associated sequence information from the associated computing node can be performed in parallel.

[0086] In practical applications, the key-value cache generated by each token in each text sequence unit within each computing node can be deployed on the key-value cache memory pool of the current computing node. The specific location can be determined based on the memory block identifier allocated by the scheduling node. Based on this, the key-value cache can be deployed in the memory block according to the memory block identifier. When performing self-attention calculation on the tokens in each text sequence, the computing node can first obtain all memory blocks of the current text sequence in its local key-value cache memory pool and perform local self-attention calculation. At the same time, the current computing node can obtain all memory blocks of the current text sequence in the key-value cache memory pool of the next computing node based on inter-device communication, so as to perform the next self-attention calculation. By parallelizing the above calculation and communication processes, communication latency can be masked. After all computing nodes have polled the memory blocks on all other computing nodes in parallel and completed the calculation, the self-attention calculation of the local sub-text sequence (i.e., multiple text sequence units) is completed in each computing node.

[0087] For example, for a target text sequence A, sub-text sequence A1 is assigned to computing node 1, sub-text sequence A2 is assigned to computing node 2, and sub-text sequence A3 is assigned to computing node 3. Computing node 1 can perform self-attention calculation on each token included in sub-text sequence A1, and obtain the self-attention calculation result of each token as key-value cache 1. At the same time, it obtains the key-value cache 2 of all tokens in sub-text sequence A2 from computing node 2, and the key-value cache 3 of all tokens in sub-text sequence A3 from computing node 3. Computing node 1 continues to perform self-attention calculation on key-value cache 1, key-value cache 2, and key-value cache 3 to obtain the self-attention calculation result corresponding to sub-text sequence A1 output by computing node 1. Correspondingly, computing node 2 and computing node 3 perform similar operations. Computing node 2 outputs the self-attention calculation result corresponding to sub-text sequence A2, and computing node 3 outputs the self-attention calculation result corresponding to sub-text sequence A3. Based on the self-attention calculation results corresponding to sub-text sequence A1, sub-text sequence A2, and sub-text sequence A3, the self-attention calculation result of the target text sequence A is determined.

[0088] In another embodiment of this specification, each computing node can perform self-attention calculation on each sub-text sequence based on the key-value cache in its own key-value cache memory pool to obtain the self-attention calculation result of each sub-text sequence; then, multiple computing nodes communicate with each other and share the self-attention calculation results of each sub-text sequence according to the text sequence to which each sub-text sequence belongs. Each computing node calculates the complete self-attention calculation result of each text sequence based on the self-attention calculation result.

[0089] For example, taking text sequence A, sub-text sequence A1 assigned to computing node 1, sub-text sequence A2 assigned to computing node 2, and sub-text sequence A3 assigned to computing node 3 as examples, computing node 1 can perform self-attention calculation on sub-text sequence A1 based on the key-value cache in its own key-value cache memory pool, and obtain the self-attention calculation result corresponding to sub-text sequence A1; in parallel, computing nodes 2 and 3 also perform self-attention calculation on sub-text sequence A2 and sub-text sequence A3 respectively, and obtain the self-attention calculation result of sub-text sequence A2 and the self-attention calculation result of sub-text sequence A3. After computation nodes 1, 2, and 3 have completed their calculations, they communicate with each other. For text sequence A, the self-attention calculation results of its three sub-text sequences A1, A2, and A3 are shared among these nodes. Specifically, node 1 sends the self-attention calculation result corresponding to sub-text sequence A1 to nodes 2 and 3; node 2 sends the self-attention calculation result corresponding to sub-text sequence A2 to nodes 1 and 3; and node 3 sends the self-attention calculation result corresponding to sub-text sequence A3 to nodes 1 and 2. Based on this, each computation node obtains the self-attention calculation result of text sequence A. Given the self-attention calculation results of sub-text sequences, for example, taking computation node 1 as an example, computation node 1 calculates the self-attention calculation result of sub-text sequence A1 itself, and receives the self-attention calculation results of sub-text sequence A2 sent by computation node 2 and the self-attention calculation results of sub-text sequence A3 sent by computation node 3. Based on this, computation node 1 can calculate the complete self-attention calculation result of text sequence A according to the self-attention calculation results of sub-text sequence A1, sub-text sequence A2 and sub-text sequence A3. Correspondingly, computation nodes 2 and 3 can also perform the above calculation, so computation nodes 2 and 3 can also calculate the complete self-attention calculation result of text sequence A.

[0090] Furthermore, since each compute node stores the key-value cache generated by each token in its key-value cache memory pool, when processing the next text sequence, the next text sequence is allocated to multiple compute nodes according to the above process. Within each compute node, if it is determined that the sub-text sequence allocated to that compute node shares a partial prefix with a sub-text sequence already cached in the key-value cache memory pool, the key-value cache of the token corresponding to that prefix can be directly retrieved from the key-value cache memory pool, reducing computational resource consumption. For example, if the sub-text sequence allocated to a compute node is "Help me XXX", and the key-value cache memory pool of that compute node contains a key-value cache for the token "Help me", then the shared prefix is ​​"Help me". Therefore, the key-value cache for the token "Help me" can be directly retrieved from the key-value cache memory pool (i.e., the shared prefix cache), without repeated computation.

[0091] In one optional embodiment of this specification, when performing position encoding on a sub-text sequence based on a task processing model, the position encoding cache of each token in the sub-text sequence can be deployed in a round-robin fashion in the same manner as described above, and each text sequence unit reads the corresponding position encoding cache in each computing node according to the starting position information for calculation.

[0092] In summary, by unifying the management of key-value cache memory pools across multiple computing nodes, memory fragmentation is reduced. Each text sequence unit is deployed on multiple computing nodes in a round-robin manner, ensuring that the computing power among multiple computing nodes is as balanced as possible. This effectively supports inference of any number and length of text sequences in sequence parallel scenarios.

[0093] In practical applications, receiving multiple text sequences includes: In response to multiple text processing requests sent by the client, determine multiple text sequences corresponding to the multiple text processing requests; After the step of calling the task processing model according to the multiple computing nodes to predict the multiple text sequence units and obtain the prediction result corresponding to each text sequence, it further includes: The prediction results corresponding to each text sequence are sent to the client.

[0094] Multiple text processing requests can be sent by a single client or by multiple clients within the same time period, with each text processing request corresponding to a text sequence.

[0095] Specifically, in response to multiple text processing requests sent by the client, the system can determine the text sequence corresponding to each text processing request, obtain the prediction result corresponding to each text sequence based on the above processing, and then send the prediction result corresponding to each text sequence to the corresponding client.

[0096] In summary, the above method deploys the task processing model across multiple computing nodes in a distributed device. After receiving multiple text sequences, the text sequences are segmented to obtain multiple text sequence units, which are then distributed sequentially to multiple computing nodes in a round-robin fashion. This ensures a balanced distribution of the text sequences across the distributed device, guaranteeing load balancing and balanced consumption of computing resources across the nodes. The task processing model is then invoked by the multiple computing nodes to predict the text sequence units, obtaining prediction results for each text sequence. This enables parallel inference and prediction of multiple text sequences, thereby ensuring the computational power and inference performance of the task processing model deployed in the distributed device. The following is in conjunction with the appendix Figure 4 Taking the application of the data processing method provided in this specification in a distributed device as an example, the data processing method will be further explained. Figure 4 A flowchart illustrating the processing procedure of a data processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0097] Step 402: Identify multiple text sequences.

[0098] For example, determine text sequence 1, text sequence 2, and text sequence 3.

[0099] Step 404: Segment the multiple text sequences to obtain multiple text sequence units.

[0100] For example, text sequence 1 is segmented to obtain multiple text sequence units corresponding to text sequence 1; text sequence 2 is segmented to obtain multiple text sequence units corresponding to text sequence 2; and text sequence 3 is segmented to obtain multiple text sequence units corresponding to text sequence 3.

[0101] Step 406: Distribute multiple text sequence units to each computing node in a round-robin fashion to achieve recombination.

[0102] For example, computing node 1 is allocated text sequence units 11 and 14 of text sequence 1, text sequence units 21 and 24 of text sequence 2, and text sequence unit 31 of text sequence 3; computing node 2 is allocated text sequence unit 12 of text sequence 1, text sequence unit 22 of text sequence 2, and text sequence unit 32 of text sequence 3, etc.

[0103] From the multiple text sequence units corresponding to each text sequence, determine the text sequence units to be assigned to each computing node. The text sequence units assigned to each computing node constitute a sub-text sequence. That is, a computing node includes a sub-text sequence corresponding to each text sequence.

[0104] Step 408: Based on the task processing model invoked by multiple computing nodes, self-attention calculation is performed on multiple text sequence units to obtain the self-attention calculation results corresponding to each text sequence.

[0105] Specifically, all computing nodes can call the task processing model in parallel to calculate the position encoding of the sub-text sequence in each computing node and update the key-value cache memory pool. All computing nodes calculate the self-attention calculation result of each sub-text sequence in parallel, and obtain the self-attention calculation result corresponding to each text sequence based on the self-attention result of each sub-text sequence.

[0106] Taking one of the multiple computing nodes as an example, this computing node includes multiple sub-text sequences. Taking one of the sub-text sequences as an example, self-attention calculation is performed on the sub-text sequence to obtain the initial calculation result. The association sequence information of the text sequence corresponding to this sub-text sequence in other computing nodes is obtained from other computing nodes. Self-attention calculation is continued on the initial calculation result and the association sequence information to obtain the self-attention calculation result. Based on this, the computing node outputs the self-attention calculation result corresponding to each sub-text sequence.

[0107] For one of the text sequences, obtain the self-attention calculation result of the sub-text sequence corresponding to the text sequence output by each computing node, and obtain the self-attention calculation result corresponding to the text sequence.

[0108] Furthermore, after obtaining the self-attention calculation results corresponding to each text sequence, the task processing model can continue to perform subsequent residual connection processing, normalization processing, nonlinear transformation processing, feature enhancement processing, and other steps based on the self-attention calculation results corresponding to the text sequences to obtain the prediction results corresponding to each text sequence.

[0109] In summary, the above method deploys the task processing model across multiple computing nodes in a distributed device. After receiving multiple text sequences, the text sequences are segmented to obtain multiple text sequence units, which are then distributed sequentially to multiple computing nodes in a round-robin fashion. This ensures a balanced distribution of the text sequences across the distributed device, guaranteeing load balancing and balanced consumption of computing resources across the nodes. The task processing model is then invoked by the multiple computing nodes to predict the text sequence units, obtaining prediction results for each text sequence. This enables parallel inference and prediction of multiple text sequences, thereby ensuring the computational power and inference performance of the task processing model deployed in the distributed device. Corresponding to the above method embodiments, this specification also provides data processing device embodiments. Figure 5 A schematic diagram of the structure of a data processing device according to one embodiment of this specification is shown. Figure 5 As shown, the device includes a scheduling node 502 and multiple computing nodes 504, wherein the multiple computing nodes 504 are deployed with a task processing model, wherein... The scheduling node 502 is configured to receive multiple text sequences; segment the multiple text sequences to obtain multiple text sequence units; and sequentially distribute the multiple text sequence units to the multiple computing nodes 504. The plurality of computing nodes 504 are configured to invoke the task processing model to predict the plurality of text sequence units and obtain the prediction results corresponding to each text sequence.

[0110] In one optional embodiment, the target computing node includes sub-text sequences corresponding to each text sequence. The target sub-text sequence is composed of target text sequence units corresponding to the target text sequence. The target text sequence unit is the text sequence unit corresponding to the target text sequence and assigned to the target computing node among the plurality of text sequence units. The target computing node is any one of the plurality of computing nodes, the target text sequence is any one of the plurality of text sequences, and the target sub-text sequence is any one of the sub-text sequences corresponding to each text sequence.

[0111] In an optional embodiment, the plurality of computing nodes 504 are further configured to: The task processing model is invoked by the multiple computing nodes to perform self-attention calculation on the sub-text sequences corresponding to each text sequence, thereby obtaining the self-attention calculation results output by each computing node for each sub-text sequence. The self-attention calculation results corresponding to each text sequence are determined based on the self-attention calculation results corresponding to each sub-text sequence.

[0112] In an optional embodiment, the plurality of computing nodes 504 are further configured to: The task processing model is invoked by the multiple computing nodes to perform self-attention calculation on each sub-text sequence corresponding to the target text sequence, thereby obtaining the self-attention calculation results output by each computing node for each sub-text sequence. The step of determining the self-attention calculation result corresponding to each text sequence based on the self-attention calculation result corresponding to each sub-text sequence includes: The self-attention calculation result corresponding to the target text sequence is determined based on the self-attention calculation results output by each computing node and corresponding to each sub-text sequence.

[0113] In an optional embodiment, the plurality of computing nodes 504 are further configured to: The target computing node calls the task processing model to perform self-attention calculation on the target sub-text sequence and obtain the initial calculation result corresponding to the target sub-text sequence. The association sequence information corresponding to the associated sub-text sequence corresponding to the target text sequence in the association computing node is sent to the target computing node. Attention calculation is performed on the initial calculation result and the association sequence information by the target computing node to obtain the target self-attention calculation result corresponding to the target sub-text sequence output by the target computing node. The associated computing node is any computing node other than the target computing node among the plurality of computing nodes.

[0114] In an optional embodiment, the scheduling node 502 is further configured to: The multiple text sequences are segmented according to a fixed length to obtain multiple text sequence units.

[0115] In an optional embodiment, the scheduling node 502 is further configured to: Determine the sequence length information of each text sequence; Based on the sequence length information of each text sequence and the fixed length, determine the number of segments corresponding to each text sequence; According to the stated number of segments, each text sequence is segmented to obtain multiple text sequence units; If a blank is found in any text sequence unit, padding data is added to the text sequence unit.

[0116] In an optional embodiment, the scheduling node 502 is further configured to: From the plurality of text sequence units, determine the text sequence unit corresponding to each text sequence; The text sequence units corresponding to the first text sequence are sequentially and round-robin distributed to the plurality of computing nodes, wherein the first text sequence is any one of the plurality of text sequences; The text sequence units corresponding to the next text sequence of the first text sequence are sequentially and round-robinly distributed to the plurality of computing nodes until the text sequence units corresponding to the first text sequence are distributed to the plurality of computing nodes.

[0117] In an optional embodiment, the scheduling node 502 is further configured to: In response to multiple text processing requests sent by the client, determine multiple text sequences corresponding to the multiple text processing requests; The prediction results corresponding to each text sequence are sent to the client.

[0118] In summary, the above-mentioned device can deploy the task processing model across multiple computing nodes. After receiving multiple text sequences, it can segment these sequences to obtain multiple text sequence units, which are then distributed to the multiple computing nodes in a round-robin fashion. This ensures a balanced distribution of the text sequences across the distributed device, balancing the computational resources consumed by the multiple computing nodes. The task processing model is then invoked by the multiple computing nodes to predict the text sequence units, obtaining the prediction results for each text sequence. This enables parallel inference and prediction of multiple text sequences, thereby guaranteeing the computational power and inference performance of the task processing model deployed in the distributed device. The above is an illustrative scheme of a data processing device according to this embodiment. It should be noted that the technical solution of this data processing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing device, please refer to the description of the technical solution of the data processing method described above.

[0119] Corresponding to the above method embodiments, this specification also provides a data processing system, including: Processor core; A system-on-a-chip (SoC) includes at least one computing node, which can communicate with each other, and the SoC executes the steps of the above-described data processing method.

[0120] The processor core is responsible for logic control, task scheduling, and serial computation. Its functions include generating computation task instructions and allocating tasks to computing nodes on the on-chip system. Its design focuses on low latency and complex instruction processing. The on-chip system is responsible for executing computation tasks for data-parallel processing, achieving high throughput through large-scale thread concurrency.

[0121] The processor core and the system-on-a-chip (SoC) can interact with each other. The processor core sends tasks to the SoC, which executes the tasks in parallel on each computing node and returns the results to the processor core.

[0122] In practical applications, within artificial intelligence systems, operators occupy an intermediate layer between the on-chip system's underlying drivers and the framework model interface, connecting algorithmic semantics with system resources. From bottom to top, the overall architecture of an AI system can be divided into the following layers: system layer, compiler layer, intermediate layer, and framework layer. The intermediate layer is where operators are developed. Based on the characteristics of the algorithm model, it defines the computational semantics and execution strategies of operators, provides standard or custom operator interfaces to deep learning frameworks, and supports forward and backward propagation. Simultaneously, it calls the scheduling and construction interfaces provided by the compiler, mapping abstract semantics into on-chip executable logic. Operators are the basic computational units that constitute neural networks, processing tensor data and implementing various computational logics commonly used in machine learning, including data transformation, conditional control, and mathematical operations.

[0123] Modern mainstream machine learning frameworks, such as PyTorch or TensorFlow, typically include: multiple programming language interfaces for users to write model logic and for the system to schedule tasks; model building modules; automatic differentiation mechanisms; and backend execution engines. Execution from the user layer to the system layer involves calling interfaces in high-level languages ​​(such as Python) and implementing operators in low-level languages ​​(such as C or C++). Machine learning frameworks include various C and C++ interfaces, allowing users to define and register C++ implemented operators for developing high-performance models, data processing functions, optimizers, and a range of other framework extensions.

[0124] In one embodiment of this specification, the on-chip system may include multiple computing nodes, each with independent local memory. The computing nodes communicate with each other to form a NUMA architecture. An upper-layer framework (such as PyTorch) can segment and allocate the input text sequence based on scheduling functions, generating an absolute position index for each token and a target computing node identifier. The RoPE operator is configured to calculate the rotation code for each text sequence unit based on this absolute position index. A key-value cache write operator can be used to write each text sequence unit to different computing nodes in a polling order. An Attention operator can be used to aggregate the segmented computation results through communication between computing nodes to obtain the final computation result corresponding to the text sequence.

[0125] The above is an illustrative scheme of a data processing system according to this embodiment. It should be noted that the technical solution of this data processing system and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing system, please refer to the description of the technical solution of the data processing method described above.

[0126] This specification provides a computing device according to one embodiment. The components of the computing device include, but are not limited to, a memory and a processor. The processor is connected to the memory via a bus, and a database is used to store data. The processor executes a computer program / instruction that, when executed by the processor, implements the steps of the data processing method described above.

[0127] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.

[0128] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0129] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.

[0130] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0131] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0132] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0133] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0134] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0135] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0136] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method applied to a distributed device, the distributed device comprising multiple computing nodes, the multiple computing nodes being deployed with a task processing model, the method comprising: Receive multiple text sequences; The multiple text sequences are segmented to obtain multiple text sequence units; The multiple text sequence units are sequentially and round-robin distributed to the multiple computing nodes; The task processing model is invoked by the multiple computing nodes to predict the multiple text sequence units and obtain the prediction results corresponding to each text sequence.

2. The method according to claim 1, wherein the target computing node includes sub-text sequences corresponding to each text sequence, the target sub-text sequence is composed of target text sequence units corresponding to the target text sequence, the target text sequence unit is the text sequence unit corresponding to the target text sequence and assigned to the target computing node among the plurality of text sequence units, the target computing node is any one of the plurality of computing nodes, the target text sequence is any one of the plurality of text sequences, and the target sub-text sequence is any one of the sub-text sequences corresponding to each text sequence.

3. The method according to claim 2, wherein the step of calling the task processing model according to the plurality of computing nodes to predict the plurality of text sequence units and obtain the prediction result corresponding to each text sequence includes: The task processing model is invoked by the multiple computing nodes to perform self-attention calculation on the sub-text sequences corresponding to each text sequence, thereby obtaining the self-attention calculation results output by each computing node for each sub-text sequence. The self-attention calculation results corresponding to each text sequence are determined based on the self-attention calculation results corresponding to each sub-text sequence.

4. The method according to claim 3, wherein the step of calling the task processing model according to the plurality of computing nodes to perform self-attention calculation on the sub-text sequences corresponding to each text sequence, and obtaining the self-attention calculation results output by each computing node for each sub-text sequence, includes: The task processing model is invoked by the multiple computing nodes to perform self-attention calculation on each sub-text sequence corresponding to the target text sequence, thereby obtaining the self-attention calculation results output by each computing node for each sub-text sequence. The step of determining the self-attention calculation result corresponding to each text sequence based on the self-attention calculation result corresponding to each sub-text sequence includes: The self-attention calculation result corresponding to the target text sequence is determined based on the self-attention calculation results output by each computing node and corresponding to each sub-text sequence.

5. The method according to claim 4, wherein the step of calling the task processing model according to the plurality of computing nodes to perform self-attention calculation on each sub-text sequence corresponding to the target text sequence, and obtaining the self-attention calculation result corresponding to each sub-text sequence output by each computing node, includes: The target computing node calls the task processing model to perform self-attention calculation on the target sub-text sequence and obtain the initial calculation result corresponding to the target sub-text sequence. The association sequence information corresponding to the associated sub-text sequence corresponding to the target text sequence in the association computing node is sent to the target computing node. Attention calculation is performed on the initial calculation result and the association sequence information by the target computing node to obtain the target self-attention calculation result corresponding to the target sub-text sequence output by the target computing node. The associated computing node is any computing node other than the target computing node among the plurality of computing nodes.

6. The method according to claim 1, wherein segmenting the plurality of text sequences to obtain a plurality of text sequence units comprises: The multiple text sequences are segmented according to a fixed length to obtain multiple text sequence units.

7. The method according to claim 6, further comprising, after receiving the plurality of text sequences: Determine the sequence length information of each text sequence; The step of segmenting the multiple text sequences according to a fixed length to obtain multiple text sequence units includes: Based on the sequence length information of each text sequence and the fixed length, determine the number of segments corresponding to each text sequence; According to the stated number of segments, each text sequence is segmented to obtain multiple text sequence units; If a blank is found in any text sequence unit, padding data is added to the text sequence unit.

8. The method according to any one of claims 1-7, wherein after segmenting the plurality of text sequences to obtain a plurality of text sequence units, the method further comprises: From the plurality of text sequence units, determine the text sequence unit corresponding to each text sequence; The text sequence units corresponding to the first text sequence are sequentially and round-robin distributed to the plurality of computing nodes, wherein the first text sequence is any one of the plurality of text sequences; The text sequence units corresponding to the next text sequence of the first text sequence are sequentially and round-robinly distributed to the plurality of computing nodes until the text sequence units corresponding to the first text sequence are distributed to the plurality of computing nodes.

9. A data processing system, comprising: Processor core; A system-on-a-chip (SoC) comprising at least one computing node capable of communicating with each other, wherein the SoC executes the steps of the method according to any one of claims 1-8.

10. A data processing device, comprising a scheduling node and multiple computing nodes, wherein the multiple computing nodes are deployed with a task processing model, wherein, The scheduling node is configured to receive multiple text sequences; segment the multiple text sequences to obtain multiple text sequence units; and sequentially distribute the multiple text sequence units to the multiple computing nodes. The plurality of computing nodes are configured to invoke the task processing model to predict the plurality of text sequence units and obtain the prediction results corresponding to each text sequence.

11. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 8.

12. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

13. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

Images