Data processing method and related apparatus

Abstract

Provided in the embodiments of the present application are a data processing method and a related apparatus, which are applied to the field of artificial intelligence and used for partitioning, into a plurality of computing units, a plurality of computing tasks corresponding to a plurality of sub-sequences, so as to achieve load balance of the computing units and implement attention operations. The method comprises: on the basis of the number of computing units, partitioning a sequence into a plurality of sub-sequences; partitioning a plurality of computing tasks of the plurality of sub-sequences into a plurality of computing units, wherein the computing tasks refer to performing attention operations on the basis of the sub-sequences, and the number of computing tasks in each computing unit is the same; and on the basis of a preset communication rule, performing an operation on each computing task in each computing unit, wherein the preset communication rule is used for indicating that a first computing unit among the plurality of computing units communicates with any second computing unit among the plurality of computing units.

Description

A data processing method and related apparatus

[0001] This application claims priority to Chinese Patent Application No. 202410875289.5, filed on June 29, 2024, entitled “A Data Processing Method and Related Apparatus”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and more particularly to a data processing method and related apparatus. Background Technology

[0003] With the explosive growth of large-scale language models (LLMs) and the increasing demand for multimodal inputs such as text, speech, and video, the sequence lengths supported by LLM models are also increasing rapidly, from the kilo (K) level to the mega (M) level. However, training large-scale language models on ultra-long sequence data faces multiple challenges. For example, when training large language models on ultra-long sequences, the memory required increases rapidly as the sequence length increases because the memory size occupied by the multi-head self-attention matrix is ​​quadratically related to the sequence length.

[0004] Currently, long sequences are typically divided into multiple subsequences, which are then distributed across multiple computational units for parallel processing. However, since the subsequences in each computational unit are different, the amount of computation required for each subsequence may vary, leading to uneven load distribution across computational units. To address this uneven load distribution issue, academia and industry have proposed solutions such as LightSeq and Context Parallel.

[0005] However, since sequence parallelism is typically deployed on an even number of computing units, the LightSeq scheme cannot achieve complete load balancing. For example, with 8 computing units, the computational tasks are distributed as follows: originally 1:2:3:4:5:6:7:8, but after LightSeq, it becomes 4:4:4:4:5:5:5:5. Furthermore, the Context Parallel scheme uses ring attention to transmit the key and value matrices, followed by block attention operations. This ring communication can lead to redundant data transmission paths, resulting in additional communication overhead. Summary of the Invention

[0006] This application provides a data processing method and related apparatus for dividing multiple computational tasks corresponding to multiple subsequences into multiple computational units, so as to achieve load balancing of the computational units and complete attention operations.

[0007] In a first aspect, this application provides a data processing method, which includes: dividing a sequence into multiple subsequences according to the number of computing units; subsequently, dividing multiple computing tasks corresponding to the multiple subsequences into multiple computing units, wherein the computing task is to perform attention operation based on the subsequences, and the number of computing tasks in each computing unit is the same; after dividing the multiple computing tasks, performing operations on the computing tasks in the computing units according to preset communication rules, wherein the preset communication rules are used to instruct a first computing unit in the multiple computing units to communicate with any second computing unit in the multiple computing units.

[0008] In this embodiment, the sequence is divided into multiple subsequences, each of which may include one or more computational tasks. The computational tasks corresponding to the multiple subsequences can be rearranged to ensure that the computational load in each computational unit is the same. Since the number of computational tasks in each computational unit is the same, and the computational workload of each task is the same, the computational load of the computational unit is thus identical. When performing attention operations on the computational tasks, the computational unit can communicate directly point-to-point according to preset communication rules to avoid redundant transmission paths during data transmission, thereby reducing additional communication overhead.

[0009] In one possible implementation, after dividing the sequence into multiple subsequences according to the number of computing units, the method may further include: dividing the multiple subsequences into multiple computing units, wherein a computing unit includes two or more subsequences.

[0010] In this embodiment, the sequence is divided into fine-grained parts, with each computational unit including two or more subsequences. This reduces the amount of computation per subsequence but increases the number of computations. By arranging more computations and communications, computation and communication are maximized and masked, thereby reducing the overall computation time.

[0011] In one possible implementation, after dividing the sequence into multiple subsequences according to the number of computing units, the method may further include: determining the computing task of each subsequence according to the order of the multiple subsequences in the sequence.

[0012] In this embodiment, since there is a causal relationship between the sequences, and the preceding sequence does not need to calculate the correlation with the following sequence, the amount of computational tasks required for each subsequence is different. The computational tasks of each subsequence can be determined according to the order of the subsequences in the sequence, which facilitates the subsequent rearrangement based on the statistical computational tasks to achieve load balancing of the computing units.

[0013] In one possible implementation, the aforementioned operation on the computation task in the computing unit according to the preset communication rules may include: determining the query matrix, key matrix, and value matrix required for the computation task according to the computation task of the computing unit; obtaining the query matrix, key matrix, and value matrix according to the preset communication rules; and performing attention operation on the query matrix, key matrix, and value matrix.

[0014] In this embodiment of the application, a preset communication rule based on non-ring direct transmission can be used. This preset communication rule enables direct communication between computing units. Therefore, when computing units perform computing tasks, they can directly obtain the Q matrix or K matrix and V matrix from other computing units to avoid additional communication transmission and thus reduce the amount of communication.

[0015] In one possible implementation, after performing calculations on the computational tasks in the computing unit according to preset communication rules, the method may further include: updating the output result corresponding to each subsequence based on the calculation result obtained after the computational tasks, wherein the output result is used to represent the value of each element in the subsequence after being processed by the attention mechanism.

[0016] Secondly, this application provides a data processing apparatus, comprising: a partitioning module, configured to partition a sequence into multiple sub-sequences according to the number of computing units; the aforementioned partitioning module is further configured to partition multiple computing tasks of the multiple sub-sequences into multiple computing units, wherein the computing tasks are attention operations based on the sub-sequences, and the number of computing tasks in each computing unit is the same; and a computation module, configured to perform computations on the computing tasks in the computing units according to preset communication rules, wherein the preset communication rules are used to instruct a first computing unit among the multiple computing units to communicate with any second computing unit among the multiple computing units.

[0017] In one possible implementation, after dividing the sequence into multiple subsequences according to the number of computing units, the aforementioned partitioning module is further configured to: divide the multiple subsequences into multiple computing units, wherein a computing unit includes two or more subsequences.

[0018] In one possible implementation, after dividing the sequence into multiple subsequences according to the number of computing units, the apparatus may further include: a determination module for determining the computing task of each subsequence according to the order of the multiple subsequences in the sequence.

[0019] In one possible implementation, the aforementioned computation module is specifically used for: determining the query matrix, key matrix, and value matrix required for the computation task based on the computation task of the computation unit; obtaining the query matrix, key matrix, and value matrix according to preset communication rules; and performing attention operations on the query matrix, key matrix, and value matrix.

[0020] In one possible implementation, after the aforementioned computational task in the computing unit is processed according to the preset communication rules, the device may further include: an update module, used to update the output result corresponding to each subsequence based on the computational result obtained after the computational task, the output result being used to represent the value of each element in the subsequence after processing by the attention mechanism.

[0021] Thirdly, this application provides a data processing apparatus, which includes: a processor, a memory, an input / output device, and a bus; the memory stores computer instructions; when the processor executes the computer instructions in the memory, the memory stores computer instructions; when the processor executes the computer instructions in the memory, it is used to implement the first aspect and any one of the implementations of the first aspect.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium including computer program instructions, which, when executed by a cluster of computing devices, enable the cluster of computing devices to perform a method as described in the first aspect or any possible implementation thereof.

[0023] Fifthly, embodiments of this application provide a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform a method as described in the first aspect or any possible implementation thereof.

[0024] The technical effects of the second to fifth aspects or any of their possible implementations can be found in the first aspect or the related possible implementations of the first aspect, and will not be repeated here. Attached Figure Description

[0025] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0026] Figure 2 is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0027] Figure 3 shows the task arrangement diagram after rearranging multiple computing tasks;

[0028] Figure 4 shows the task execution sequence of multiple computing tasks;

[0029] Figure 5 is a schematic diagram of the process of parallel attention operation and communication in long sequences;

[0030] Figure 6 shows a performance comparison of different schemes when communication is restricted to a bound.

[0031] Figure 7 shows a performance comparison of different schemes when calculating the bound;

[0032] Figure 8 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0033] Figure 9 is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0035] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to have the functions of perception, reasoning, and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0036] First, the overall workflow of an artificial intelligence system is described. The following sections elaborate on the aforementioned AI framework from two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0037] (1) Infrastructure

[0038] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0039] (2) Data

[0040] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0041] (3) Data processing

[0042] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0043] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0044] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0045] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0046] (4) General ability

[0047] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0048] (5) Smart Products and Industry Applications

[0049] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0050] To better understand the solutions of the embodiments of this application, the relevant terms and concepts that may be involved in the embodiments of this application will be introduced first.

[0051] (1) Large Language Model (LLM)

[0052] LLM (Language Modeling) refers to a language model with hundreds of millions (or more) parameters trained on massive amounts of text data. It's a deep learning-based natural language processing model. These models can process large amounts of text data to learn the grammatical and semantic rules of natural language. LLMs can be applied to text generation, machine translation, question answering systems, text summarization, and sentiment analysis, offering advantages such as strong generative capabilities, high adaptability, accurate prediction, and scalability. For example, in movie recommendation scenarios, large language models can generate descriptions of movie scenes, including genre, main actors, and plot, enabling the system to better recommend similar films. Large language models can also generate recommendation reasons; for instance, e-commerce websites can use large language models to generate reasons for recommending products, such as product quality, price, and features, allowing users to better understand the value of the product.

[0053] (2) Attention mechanism

[0054] Attention mechanisms are techniques that allow models to focus on and learn important information and its relevance. In the fields of machine learning and artificial intelligence, attention mechanisms are widely used in natural language processing, computer vision, and speech recognition to help models better handle sequential data and extract key information.

[0055] (3) Sequence Parallelism (SP)

[0056] Sequence parallelism is a strategy that divides long sequences of data into multiple subsequences and processes these subsequences in parallel across different computing devices or operating domains. It has wide applications in deep learning, especially when processing long sequences of data. For example, in natural language processing, Transformer models typically need to process long sequences of text. By using sequence parallelism, these long sequences can be divided into multiple subsequences and processed in parallel across multiple GPUs or other computing devices, thereby accelerating the model training process.

[0057] (4) Model FLOPs Utilization (MFU)

[0058] Model computing power utilization rate refers to the ratio of matrix computing power consumed by a model in one forward and backward calculation to machine computing power, where matrix computing power is the ratio of matrix calculation amount to the duration of one iteration.

[0059] (5) Token

[0060] A token refers to a single element in a text sequence. In natural language processing, a token is the basic unit of text preprocessing.

[0061] The method provided in this application can be applied to training scenarios of large language models, as well as training of multimodal models and long sequence inference scenarios, and is not specifically limited here.

[0062] To make the technical solution of this application clearer and easier to understand, the system architecture of this application will be described below with reference to the accompanying drawings.

[0063] Referring to Figure 1, a system architecture 100 is provided according to an embodiment of this application. This system architecture 100 may include a computing device cluster 110, which includes at least one computing device. The computing device may be a server, such as a cloud server, a central server, an edge server, or a local server in a local data center. In some possible embodiments, the computing device may also be a desktop computer, a laptop computer, or other terminal device.

[0064] Specifically, any computing device in the computing device cluster can implement the data processing method of this application embodiment by installing applications or plugins. Then, the computing device can divide the sequence into multiple sub-sequences according to the number of computing units, and determine multiple computing tasks for each sub-sequence. After obtaining the multiple computing tasks to be processed for each sub-sequence, the multiple computing tasks can be sent to multiple computing units, which then complete the attention operation tasks corresponding to the sub-sequences. The computing unit (also called an accelerator card) used to perform the computational tasks can be deployed in any or multiple computing devices in the computing device cluster. The accelerator card can be a Graphics Processing Unit (GPU), a Neural Network Processing Unit (NPU), an Ascend processor, etc., and is not specifically limited here.

[0065] It is worth noting that the system architecture shown in Figure 1 is only an example and is not intended to limit its specific implementation to this example. For example, in other possible system architectures, system architecture 100 may also include the client, which provides sequence data, etc., to the computing device.

[0066] As the sequence lengths supported by large language models gradually increase, large language models will face multiple challenges when training on ultra-long sequences. For example, since the memory occupied by the multi-head self-attention matrix calculated based on ultra-long sequences is quadratically related to the sequence length, the sequence length will significantly affect memory usage. The communication brought about by sequence parallelism greatly reduces the utilization of model computing power. In addition, when multiple GPUs perform sequence parallel operations, the different amount of computing tasks on each GPU leads to uneven load among the multiple GPUs.

[0067] Currently, since sequence parallelism is typically deployed on an even number of computing units, the LightSeq scheme cannot achieve complete load balancing. To address the uneven load problem, existing solutions have proposed Context Parallel, which concatenates the beginning and end of the sequence to achieve load balancing. Then, it transmits the key and value matrices of each computing unit based on ring attention for attention operations. However, rule-based communication based on ring attention introduces additional communication overhead due to redundant data transmission paths, leading to an increase in the total communication volume.

[0068] To address the current problem, this solution proposes a data processing method that employs a fine-grained partitioning approach to achieve load balancing across each computing unit. Furthermore, it utilizes a pre-defined communication rule that does not involve direct ring transmission, enabling the query matrix, key matrix, and value matrix to be transmitted between computing units to perform attention operations, thereby avoiding redundant communication.

[0069] The method flow provided in this application will be described below in conjunction with the aforementioned system architecture.

[0070] Referring to Figure 2, a flowchart of a data processing method provided in this application is shown below.

[0071] Step 201: Divide the sequence into multiple subsequences according to the number of computational units;

[0072] To address the increased memory requirements of training large language models with long data sequences, long sequences are typically divided into multiple subsequences, which are then processed separately by multiple computational units. These subsequences can be text sequences, structured data sequences, audio sequences, or image description sequences, etc., and are not specifically limited here.

[0073] The sequence can be divided evenly to obtain multiple subsequences of the same length, or it can be divided randomly to obtain multiple subsequences of different lengths; the specific method is not limited here.

[0074] Model computational efficiency (MFU) is the ratio of matrix computational power to machine computational power consumed in a single forward-backward computation of the model. Matrix computational power is the ratio of matrix computation to iteration time. Dividing long sequences into multiple subsequences and parallelizing them across multiple computational units increases communication between these units, thus extending iteration time and reducing MFU. To improve MFU, computation and communication can be maximized, ensuring communication occurs as much as possible during computation, thereby reducing iteration time delays caused by communication.

[0075] In this embodiment, a fine-grained partitioning method can be used to subdivide the sequence, increasing the number of attention operations performed on the subsequences. The computational cost per subsequence is reduced, ensuring the total computational cost remains constant. By orchestrating more computations and communications, computational and communication coverage is maximized. For example, when the number of computational units is N, fine-grained partitioning can divide the text sequence into k·N subsequences, where k is an integer greater than or equal to 2. For instance, the text sequence can be divided into 2N subsequences or 3N subsequences; the specific division is not limited here. Each subsequence can include multiple tokens. However, considering computational complexity, dividing the sequence into 2N subsequences based on the number of computational units N is sufficient.

[0076] Specifically, after dividing the sequence into multiple subsequences, these subsequences can be further divided into multiple computational units. The number of subsequences can be k times the number of computational units, where k is an integer greater than or equal to 2. This allows a computational unit to include two or more subsequences, enabling fine-grained division of the sequence. Consequently, more computations and communications can be orchestrated based on this fine-grained segmentation, maximizing computational and communication coverage.

[0077] After dividing multiple subsequences into multiple computational units, each token in the subsequence can be vectorized to obtain a subsequence matrix based on the subsequences in the computational unit. Then, the subsequence matrix can be operated on with different parameter matrices to obtain the query matrix (Q matrix), key matrix (K matrix), and value matrix (V matrix) corresponding to the subsequence, so that attention operations can be performed based on the Q matrix, K matrix, and V matrix.

[0078] The formula for calculating the Q matrix satisfies Q = X·W Q Similarly, the formula for calculating the K matrix can satisfy K = X·W K The formula for calculating matrix V satisfies V = X·W V X can be represented as any subsequence vector, W Q W K W V These are the parameter matrices corresponding to the Q, K, and V matrices, respectively.

[0079] Step 202: Divide the multiple computational tasks of multiple subsequences into multiple computational units;

[0080] After dividing the sequence into multiple subsequences, the computational task to be performed can be determined based on these subsequences. This task involves performing attention operations on each element within each subsequence. The core idea of ​​the attention operation is to assign a weight (also called an attention score) to each element in the input sequence. This weight represents the importance of the element in the current sequence. Based on the attention operation results, the model can focus on and learn the more important information with higher weights.

[0081] In one possible implementation, for sequences with causal relationships, subsequences within the sequence only need to undergo correlation analysis and attention operations with their preceding subsequences. For example, in a text sequence, subsequent tokens only need to calculate attention with the tokens preceding them. Therefore, the computational task required for each subsequence can be determined based on its order within the sequence, facilitating subsequent rearrangement based on this computational task to achieve balanced computational load for each computational unit.

[0082] For example, a text sequence is divided into four subsequences, numbered according to their order in the sequence, which can be represented as token1, token2, token3, and token4. Tokens correspond to the Qi, Ki, and Vi matrices. Token1 only needs to perform attention operations based on Q1, K1, and V1 to obtain O. 11 While token2's Q2 requires attention operations with K2 and V2 to obtain O, 22 It also needs to perform attention operations with K1 and V1 of token1.

[0083] After obtaining multiple computing tasks, these tasks can be divided into multiple computing units based on the number of computing units, so that each computing unit has an equal number of computing tasks to execute. Since the computational workload of each computing task is equal, load balancing can be achieved when the total number of computing tasks in each computing unit is equal.

[0084] For example, as shown in Figure 3, which is a task arrangement diagram after rearranging multiple computing tasks, when the number of computing units (NPUs) is 4 and the text sequence includes 8 tokens, the text sequence can be divided into 8 parts according to the number of computing units and numbered according to their order in the text sequence, namely token1, token2, token3, token4, token5, token6, token7, and token8. Token1 and token2 are assigned to NPU1, token3 and token4 to NPU2, token5 and token6 to NPU3, and token7 and token8 to NPU4. If the computing task corresponding to a token is completed by the NPU containing that token, the computing load of each NPU is 4i-1, and the computing load of each NPU is unevenly distributed. Therefore, multiple computing tasks can be rearranged to make the number of computing tasks in each NPU the same, thereby balancing the computing load of the NPUs. Figure 3 is only one possible implementation and is not limited to this example. In other possible implementations, there may be other task arrangements so that the number of computing tasks in each NPU is the same.

[0085] Step 203: Perform calculations on the computational tasks in the computing unit according to the preset communication rules.

[0086] After arranging multiple computational tasks corresponding to multiple subsequences, the computational units can perform operations on the assigned tasks according to preset communication rules. These preset communication rules instruct the first computational unit among the multiple computational units to communicate directly with any second computational unit among the multiple computational units. When performing computational tasks, the computational unit can directly obtain the required Q-matrix, K-matrix, or V-matrix sent by other computational units based on these preset communication rules.

[0087] In this embodiment, based on preset communication rules, computing units can directly transmit data, avoiding redundant data transmission paths caused by ring communication, thereby reducing the amount of data transmission.

[0088] Specifically, based on the computation task in the computing unit, the Q matrix, K matrix, and V matrix required for the computation task can be determined first. Then, based on preset communication rules, the Q matrix, K matrix, and V matrix can be obtained. After that, attention operations can be performed on the Q matrix, K matrix, and V matrix. For example, based on the task arrangement diagram shown in Figure 3, when NPU3 needs to calculate the attention between token5 and token1, according to the preset communication rules, NPU1 can directly send K1 and V1 to NPU3. Then, NPU3 performs attention operations based on the obtained K1 and V1 and Q5 stored in NPU3.

[0089] The formula for attention operation can satisfy the following equation:

[0090] Where Q represents the query matrix, K represents the key matrix, V represents the value matrix, and d k Indicates the feature dimension.

[0091] After the computation unit completes the attention operation, it can update the output O of each subsequence based on the result of the attention operation for each subsequence. This output represents the value of each element in the subsequence after processing by the attention mechanism. If a subsequence undergoes multiple attention operations, the results of these operations can be weighted and averaged to obtain the final output of the subsequence. For example, the aforementioned token3 needs to consider not only the autocorrelation of each element in token3 but also its correlation with token1 and token2. Therefore, token3 also needs to undergo attention operations with token1 and token2 respectively to obtain O. 31 and O 32 Combined with O 33 A weighted average is then performed to obtain the final output result corresponding to token3.

[0092] The following is a detailed description of one possible implementation method in the embodiments of this application.

[0093] For example, there are 4 computing units. The sequence includes 8 tokens. The sequence can be divided into 8 parts, and the 8 tokens can be numbered in order. Token1 and token2 are assigned to NPU1, token3 and token4 are assigned to NPU2, token5 and token6 are assigned to NPU3, and token7 and token8 are assigned to NPU4. Tokeni corresponds to Qi, Ki, and Vi.

[0094] Based on the task arrangement shown in Figure 3, Figure 4 is a diagram showing the execution sequence of computational tasks based on the task arrangement shown in Figure 3, and Figure 5 is a flowchart of long-sequence parallel attention operations and communication. Figure 5 illustrates the communication between multiple computational units and the computational process of each unit on the computational tasks. Specifically, Figure 5...<i,j> Represents the computation task FA(Q) i ,K j V j ), for Q i ,K j V j Perform Flash Attention operation, denoted as FA(Q) i ,K j V j Flash Attention is a hardware-friendly and efficient way to compute attention.

[0095] Specifically, in the first round, each computing unit first performs attention operations based on the sub-sequences assigned by the computing unit. At the same time, it can also send the Q matrix or K matrix and V matrix stored in the computing unit to other computing units so that the other computing units can perform attention operations based on the obtained Q matrix or K matrix and V matrix.

[0096] As shown in Figure 5, the first round:

[0097] Calculation unit 1 calculates FA(Q1,K1,V1) and FA(Q2,K2,V2), and then passes K1 and V1 to calculation unit 2;

[0098] Calculation unit 2 calculates FA(Q3,K3,V3) and FA(Q4,K4,V4), and then passes K3 and V3 to calculation unit 3;

[0099] Calculation unit 3 calculates FA(Q5,K5,V5) and FA(Q6,K6,V6), and passes K5 and V5 to calculation unit 4;

[0100] Calculation unit 4 calculates FA(Q7,K7,V7) and FA(Q8,K8,V8), and then passes Q7 and Q8 to calculation unit 1.

[0101] In the second round, computing unit 2 can execute computing tasks FA(Q3,K1,V1) and FA(Q4,K1,V1) based on K1 and V1 sent by computing unit 1, as well as its own Q3 and Q4, and can send K4 and V4 to computing unit 3. Similarly, the computing and communication status of other computing units can be obtained.

[0102] Round Two:

[0103] Calculation unit 1 calculates FA(Q7,K1,V1) and FA(Q8,K1,V1), and passes K2 and V2 to calculation unit 2;

[0104] Calculation unit 2 calculates FA(Q3,K1,V1) and FA(Q4,K1,V1), and passes K4 and V4 to calculation unit 3;

[0105] Calculation unit 3 calculates FA(Q5,K3,V3) and FA(Q6,K3,V3), and passes K6 and V6 to calculation unit 4;

[0106] Calculation unit 4 calculates FA(Q7,K5,V5) and FA(Q8,K5,V5).

[0107] It should be understood that the computation and communication details for the third, fourth, and fifth rounds can be obtained by referring to Figure 5.

[0108] Third round:

[0109] Calculation unit 1 calculates FA(Q7,K2,V2) and FA(Q8,K2,V2), and passes K2 and V2 to calculation unit 3;

[0110] Calculation unit 2 calculates FA(Q3,K2,V2) and FA(Q4,K2,V2). 42 ), and pass K4,V4 to computing unit 4;

[0111] Calculation unit 3 calculates FA(Q5,K4,V4) and FA(Q6,K4,V4), and then passes Q5 and Q6 to calculation unit 1;

[0112] Calculation unit 4 calculates FA(Q7,K6,V6) and FA(Q8,K6,V6), and then passes Q7 and Q8 to calculation unit 2;

[0113] Round 4:

[0114] Calculation unit 1 calculates FA(Q5,K1,V1) and FA(Q6,K1,V1), and sets O 71 O 81 The data is transmitted to computing unit 4;

[0115] Calculation unit 2 calculates FA(Q7,K3,V3) and FA(Q8,K3,V3);

[0116] Calculation unit 3 calculates FA(Q5,K2,V2) and FA(Q6,K2,V2);

[0117] Calculation unit 4 calculates FA(Q7,K4,V4) and FA(Q8,K4,V4);

[0118] In the fourth round, O 71 For O 71 and O 72 The output value obtained after merging is similarly O. 81 For O 81 and O 82 The output value obtained after merging.

[0119] Round 5:

[0120] Calculation unit 1 calculates FA(Q2,K1,V1) and sets O 51 O 61 Passed to computing unit 3;

[0121] Calculation unit 2 calculates FA(Q4,K3,V3) and sets O 73 O 83 Passed to computing unit 3;

[0122] Calculation unit 3 calculates FA(Q6,K5,V5);

[0123] Calculation unit 4 calculates FA(Q8,K7,V7).

[0124] In this embodiment, different computing units store the output O corresponding to different subsequences. The output corresponding to different subsequences should be updated in the computing unit where the subsequences are stored. Therefore, the O calculated by computing unit 1... 71 O 81 And O 72 O 82 It can be sent to calculation unit 4, and the O calculated by calculation unit 2 73 O 83 It can be sent to computing unit 4, which will then process o. 71 o 72 o 73 o 74 o 75 o 76 and o 77 A weighted average is performed to update the output o7 corresponding to token7. It should be understood that the way calculation unit 3 updates the output is similar to that of calculation unit 4, and will not be described again here.

[0125] Based on the task arrangement diagram shown in Figure 3, according to the preset communication rules, the output results o1, o2, ..., o8 corresponding to token1, token2, ..., token8 can be obtained respectively. The output result corresponding to each token can represent the value of each token after being processed by the attention mechanism.

[0126] For example, a comparative analysis is performed on the data processing method proposed in the embodiments of this application for the use of existing schemes for sequence parallel processing. The comparison results are shown in Figures 6 and 7, where the numbers in Figures 6 and 7 are used to represent the computation time and communication time.

[0127] Figure 6 compares the performance of different schemes when communication bounds are limited. As shown in Figure 6, when communication bounds are limited (fast computation, slow communication), this scheme outperforms both LightSeq (LS) and Context Parallel (CP) schemes, with a performance improvement of over 20% compared to Context Parallel. Figure 7 compares the performance of different schemes when calculating bounds. As shown in Figure 7, when calculating bounds (slow computation, fast communication), this scheme achieves a 25% performance improvement compared to the LightSeq scheme.

[0128] The method flow provided in this application has been described above. The apparatus provided in this application will now be described based on the aforementioned method flow.

[0129] Referring to Figure 8, a schematic diagram of the structure of a data processing device provided in this application is shown below.

[0130] The partitioning module 801 is used to divide the sequence into multiple subsequences based on the number of computing units;

[0131] The aforementioned partitioning module 801 is also used to partition multiple computation tasks of multiple subsequences into multiple computation units, wherein the computation task is to perform attention operation based on the subsequences, and the number of computation tasks in each computation unit is the same.

[0132] The computation module 802 is used to perform computation on the computation tasks in the computation unit according to the preset communication rules. The preset communication rules are used to instruct the first computation unit in the plurality of computation units to communicate with any second computation unit in the plurality of computation units.

[0133] In one possible implementation, after dividing the sequence into multiple subsequences according to the number of computing units, the aforementioned partitioning module 801 is further configured to: divide the multiple subsequences into multiple computing units, wherein a computing unit includes two or more subsequences.

[0134] In one possible implementation, after dividing the sequence into multiple subsequences according to the number of computing units, the device may further include: a determination module 803, used to determine the computing task of each subsequence according to the order of the multiple subsequences in the sequence.

[0135] In one possible implementation, the aforementioned computation module 802 is specifically used for: determining the query matrix, key matrix, and value matrix required for the computation task based on the computation task of the computation unit; obtaining the query matrix, key matrix, and value matrix according to preset communication rules; and performing attention operations on the query matrix, key matrix, and value matrix.

[0136] In one possible implementation, after the aforementioned computational task in the computing unit is processed according to the preset communication rules, the device may further include: an update module 804, used to update the output result corresponding to each subsequence based on the computational result obtained after the computational task, the output result being used to represent the value of each element in the subsequence after processing by the attention mechanism.

[0137] This application also provides a computing device 900. As shown in FIG9, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other via the bus 902. The computing device 900 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 900.

[0138] Bus 902 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc. The Unified Bus is also known as the Lingqu Bus. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 9 uses only one line, but this does not indicate that there is only one bus or one type of bus. Bus 904 can include pathways for transmitting information between various components of the computing device 900 (e.g., memory 906, processor 904, communication interface 908). The Unified Bus can also be called the Lingqu Bus.

[0139] The processor 904 may include any one or more computing devices such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, or an accelerator card.

[0140] Memory 906 may include volatile memory, such as random access memory (RAM). Processor 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 906 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0141] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 906, or two or more types of storage media can be configured to realize the function of memory 906. This application does not limit this.

[0142] The memory 906 stores executable program code, which the processor 904 executes to implement the functions of the aforementioned partitioning module, arithmetic module, determination module, and update module, thereby realizing the data processing method. In other words, the memory 906 stores instructions for executing the data processing method.

[0143] The communication interface 908 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 900 and other devices or communication networks.

[0144] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the steps of the method described in the embodiment shown in FIG2 above.

[0145] This application also provides a computer program product that, when run on a computer, causes the computer to perform the method steps described in the embodiment shown in FIG2 above.

[0146] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0147] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the systems, devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0149] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0151] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0153] Finally, it should be noted that the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: The sequence is divided into multiple subsequences based on the number of computational units; The computational tasks of the multiple subsequences are divided into multiple computational units, and the computational tasks are attention operations based on the subsequences. The number of computational tasks in each computational unit is the same. According to preset communication rules, the computational tasks in the computing units are performed. The preset communication rules are used to instruct the first computing unit among the plurality of computing units to communicate with any second computing unit among the plurality of computing units.

2. The method according to claim 1, characterized in that, After dividing the sequence into multiple subsequences based on the number of computing units, the method further includes: The multiple subsequences are divided into multiple computing units, and each computing unit includes two or more subsequences.

3. The method according to claim 1 or 2, characterized in that, After dividing the sequence into multiple subsequences based on the number of computing units, the method further includes: The computational task for each subsequence is determined based on the order of the multiple subsequences in the sequence.

4. The method according to any one of claims 1 to 3, characterized in that, The step of performing calculations on the computational tasks in the computing unit according to preset communication rules includes: Based on the computation task of the computation unit, determine the query matrix, key matrix, and value matrix required to perform the computation task; According to the preset communication rules, the query matrix, the key matrix, and the value matrix are obtained; Attention operations are performed on the query matrix, the key matrix, and the value matrix.

5. The method according to any one of claims 1 to 4, characterized in that, After performing calculations on the computational tasks in the computing unit according to preset communication rules, the method further includes: Based on the computation results obtained after the computation task, the output results corresponding to each subsequence are updated. The output results are used to represent the value of each element in the subsequence after processing by the attention mechanism.

6. A data processing apparatus, characterized in that, include: The partitioning module is used to divide the sequence into multiple subsequences based on the number of computational units; The partitioning module is further configured to divide the multiple computation tasks of the multiple subsequences into multiple computation units, wherein the computation task is to perform attention operation based on the subsequences, and the number of computation tasks in each computation unit is the same; The computation module is used to perform computation on the computational tasks in the computational unit according to a preset communication rule, wherein the preset communication rule is used to instruct the first computational unit in the plurality of computational units to communicate with any one of the second computational units in the plurality of computational units.

7. The apparatus according to claim 6, characterized in that, After dividing the sequence into multiple subsequences according to the number of computing units, the division module is further configured to: divide the multiple subsequences into the multiple computing units, wherein one computing unit includes two or more subsequences.

8. The apparatus according to claim 6 or 7, characterized in that, After dividing the sequence into multiple subsequences based on the number of computing units, the apparatus further includes: The determination module is used to determine the computational task for each subsequence based on the order of the multiple subsequences in the sequence.

9. The apparatus according to any one of claims 6 to 8, characterized in that, The computing module is specifically used for: Based on the computation task of the computation unit, determine the query matrix, key matrix, and value matrix required to perform the computation task; According to the preset communication rules, the query matrix, the key matrix, and the value matrix are obtained; Attention operations are performed on the query matrix, the key matrix, and the value matrix.

10. The apparatus according to any one of claims 6 to 9, characterized in that, After performing calculations on the computational tasks in the computing unit according to preset communication rules, the device further includes: The update module is used to update the output result corresponding to each subsequence based on the computation result obtained after the computation task, wherein the output result is used to represent the value of each element in the subsequence after processing by the attention mechanism.

11. A data processing apparatus, characterized in that, include: A processor and a memory, wherein the processor is coupled to the memory; The memory is used to store programs; The processor is configured to execute a program in the memory, such that the method as described in any one of claims 1 to 5 is performed.

12. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 5.

13. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 5.

Images