Attention calculation method, machine learning system and related equipment

Abstract

The invention provides an attention calculation method, a machine learning system and related equipment, and the method comprises the following steps: obtaining a to-be-processed initial sequence and an initial vector by each node, dividing the initial vector into a plurality of sub-vectors, carrying out the first time of attention calculation according to the initial sequence and the initial vector, and meanwhile, carrying out the second time of attention calculation according to the sub-vectors; the sub-vectors are continuously sent to the next node by taking the sub-vectors as units, and the sub-vectors continuously sent by the previous node are received, so that after the first attention calculation is finished, attention calculation can be continuously performed according to the received sub-vectors of the previous node, and calculation is performed while receiving is performed; the sub-vectors of all the nodes are calculated until the sub-vectors of all the nodes are calculated, so that calculation results corresponding to the nodes are obtained, idle waiting of processing modules of the nodes is avoided, resource waste is avoided, attention calculation efficiency is improved, and overall training efficiency or reasoning efficiency of the model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence (AI), and in particular to an attention calculation method, a machine learning system and related equipment. BACKGROUND

[0002] A large language model (LLM) refers to a natural language processing model with a large scale and a large number of parameters. The model uses deep learning technology to understand and generate human language. Generally, the network layer of the large language model can include an input layer, an attention layer and an output layer. The input layer is used to convert input text of natural language into a plurality of word vectors that can be processed by the model. The attention layer is used to perform attention calculation according to the plurality of word vectors of the input text to obtain an attention vector, which is used to describe the importance of different word vectors in the inference process of the model, so that the large language model can focus on important information in the input text, thereby generating more reasonable output text. The output layer is used to generate an output result according to the plurality of word vectors and the attention vector. The attention layer is the core mechanism of the large language model, which enables the large language model to be trained on a large amount of text data. The trained large language model can not only generate natural language questions, but also deeply understand text semantics and process various natural language tasks, such as text summarization, question answering and translation.

[0003] However, the computational complexity of the attention layer of the large language model when processing the input sequence is closely related to the length of the input sequence. If the input text of the model is long, it will result in huge computational and storage overhead, affecting the computational efficiency of the attention layer, and further affecting the training efficiency of the model. In order to alleviate the computational and storage pressure of the large language model in the long sequence training scenario, relevant scholars have proposed a ring attention mechanism. Ring attention is an improved attention mechanism. The mechanism will split the input sequence, K vector and V vector of the attention layer according to the number of nodes, and then distribute them to each node. Each node obtains a part of the input sequence, a part of the K vector and a part of the V vector. The network structure between multiple nodes is a ring network structure. Each node sends the K vector and V vector obtained by the node to the next node, and performs attention calculation according to the K vector and V vector sent by the previous node and the input sequence on the node. After multiple sending, receiving and calculation, each node can calculate the attention calculation result of the input sequence on the node based on the K vector and V vector on all nodes. The attention calculation results of each node are summarized to obtain the attention calculation result corresponding to the complete input sequence, thereby realizing distributed processing of the attention mechanism.

[0004] However, after each node completes the calculation, if the K vector and the V vector sent by the previous node have not been received, the node needs to be idle and wait until the K vector and the V vector of the previous node are received, and then the attention calculation can continue. After multiple communications, a large amount of idle waiting time occurs in the node, which causes resource waste, affects the attention calculation efficiency of the model, and thus affects the training efficiency or inference efficiency of the model. SUMMARY

[0005] The application provides an attention calculation method, a machine learning system and related equipment, which are used to solve the problem that when a large language model performs attention calculation, the nodes of the training model are prone to a large amount of idle waiting time, which causes resource waste and affects the attention calculation efficiency.

[0006] In a first aspect, an attention calculation method is provided. The method is applied to a machine learning system, and the machine learning system includes a plurality of nodes. The plurality of nodes form a ring network communication structure. The method includes the following steps: a first node in the plurality of nodes acquires an initial sequence and an initial vector to be processed by an attention layer, the first node splits the initial vector to obtain a plurality of first sub-vectors, the first node performs first attention calculation according to the initial sequence and the initial vector, sends the plurality of first sub-vectors to a second node, receives a plurality of second sub-vectors sent by a third node, and the first node performs multiple attention calculations according to the initial sequence and the plurality of second sub-vectors to obtain an attention calculation result of the first node.

[0007] According to the method described in the first aspect, when the model performs attention calculation of each layer, after each node acquires the initial sequence and the initial vector allocated to it, the node first divides the initial vector into a plurality of first sub-vectors, then performs first attention calculation according to the initial sequence and the initial vector, and constantly sends the first sub-vectors to the next node and constantly receives the second sub-vectors sent by the previous node in units of sub-vectors. After the first attention calculation is completed, each node can perform attention calculation according to the second sub-vectors of the previous node that have been received. In this way, the sub-vectors of all nodes are calculated until the calculation result of the attention layer is obtained. Since each node has received the sub-vectors required for the next attention calculation after the previous attention calculation is completed, each node does not need to be idle and wait, which avoids resource waste, improves the attention calculation efficiency, and further improves the overall training efficiency or inference efficiency of the model.

[0008] In one possible implementation, the first node can first determine the sending order of each first sub-vector, then send multiple first sub-vectors to the second node according to the sending order, receive multiple second sub-vectors sent by the third node, obtain the receiving order of each second sub-vector, send multiple second sub-vectors to the second node according to the receiving order, and then perform multiple attention calculations according to the receiving order based on the initial sequence and multiple second sub-vectors.

[0009] In the above implementation, each node sends multiple first sub-vectors sequentially to the next node, one sub-vector at a time. This ensures that the first node continuously receives second sub-vectors sent by the third node. After the first attention calculation is completed, even if the first node has not yet received the complete initial vector from the third node, at least the first second sub-vector has been received. The first node does not need to wait for transmission but can directly perform attention calculations on a sub-vector-by-sub-vector basis, avoiding node idle waiting and improving attention calculation efficiency. Furthermore, since each node sends sub-vectors in a fixed order, the attention calculation order is also fixed, ensuring that each node's sub-vector is calculated only after its dependent sub-vectors have been calculated, thus avoiding calculation errors.

[0010] In one possible implementation, when the first node determines the sending order of each first sub-vector, it can first divide the initial vector into multiple first sub-vectors according to the length of the initial vector, determine the sending order of each first sub-vector based on the position of each first sub-vector in the initial vector, and then perform multiple attention calculations based on the initial sequence and multiple second sub-vectors. The calculation results obtained from each attention calculation are merged and updated with the calculation results of the previous attention calculations.

[0011] In simple terms, assume the initial sequence Q assigned to the node is... i Initial vector K i and the initial vector V i It is a four-dimensional matrix (s, b, x, h) or (s, b, x, h), where s is the length of the initial sequence assigned to the current node, b is the batch size of the current node, x is the number of heads in the multi-head attention mechanism, and h is the hidden layer dimension of each attention head. If the vector is split according to its length, and the number of splits is m, then after splitting, due to the initial sequence Q under this strategy... i It does not participate in the partitioning, so its four-dimensional matrix remains (s,b,x,h) or (b,s,x,h), with the initial vector K. i and initial vector V i Participating in the segmentation, its subvector K ij and V ijThe four-dimensional matrix is ​​(s / m,b,x,h) or (b,s / m,x,h).

[0012] The above implementation divides the initial vector into multiple first sub-vectors based on its length. The order in which these sub-vectors are sent, or the order of attention calculation, is determined by their position within the initial vector. After each sub-vector is calculated, its attention result is merged and updated with the previous sub-vector's attention calculation result. Since attention calculation itself requires calculation based on different positions within the initial vector, dividing the initial vector in this way and calculating them separately will not affect the attention calculation result.

[0013] In another possible implementation, when the first node determines the sending order of each first sub-vector, it can first divide the initial vector into multiple first sub-vectors according to the number of attention heads. Based on the order of the index numbers of the attention heads corresponding to each first sub-vector, the sending order of each first sub-vector is determined. Then, attention calculations are performed multiple times according to the initial sequence and multiple second sub-vectors. The calculation results obtained from every m attention calculations are merged and updated with the calculation results of previous attention calculations, where m is the number of first sub-vectors.

[0014] In simple terms, assume the initial sequence Q assigned to the node is... i Initial vector K i and the initial vector V i It is a four-dimensional matrix (s, b, x, h) or (s, b, x, h), where s is the length of the initial sequence assigned to the current node, b is the batch size of the current node, x is the number of heads in the multi-head attention mechanism, and h is the hidden layer dimension of each attention head. If the segmentation is based on the number of attention heads, and the number of segments is m, then after the segmentation, due to the initial sequence Q under this strategy... i It also needs to participate in the segmentation, so sequence Q i The corresponding subsequence Q ij The four-dimensional matrix is ​​(s,b,x / m,h) or (b,s,x / m,h), and the initial vector K i and initial vector V i Participating in the segmentation, its subvector K ij and V ij The four-dimensional matrix is ​​(s,b,x / m,h) or (b,s,x / m,h).

[0015] The above implementation divides the initial vector according to the number of attention heads in the attention layer to obtain multiple first sub-vectors. The sending order or the calculation order of attention is determined according to the index number of the attention head corresponding to each sub-vector. Since the multiple attention heads in the attention layer process the attention calculation results in parallel, the calculation results of the attention layer can be obtained by concatenating the calculation results of each attention head. Therefore, the attention calculation can be completed by m sub-vectors and then concatenated in a unified manner. Dividing the initial vector in this way and then calculating it separately will not affect the result of the attention calculation.

[0016] It should be noted that when splitting the initial vector according to its length, starting from the second attention calculation, the result needs to be merged and updated with the previous result after each calculation, which is achieved using the softmax function. When splitting the initial vector according to the number of attention heads, starting from the second attention calculation, the result is merged and updated with the previous result only after every m calculations, which is achieved by vector concatenation.

[0017] It should be understood that when the initial vector is split according to its length, it's equivalent to different nodes calculating the information at different positions of the initial vector. Therefore, after obtaining each calculation result, the softmax function needs to be used to integrate and update it, and the same initial vector is used each time. However, when the initial vector is split according to the number of attention heads, each sub-vector is split according to the number of attention heads. The attention mechanism itself requires each attention head to be calculated in parallel, and then the results of the attention heads are concatenated. Therefore, the calculation results of all attention heads can be obtained and then concatenated uniformly, without needing to concatenate them after each calculation.

[0018] In one possible implementation, the method further includes the following steps: the first node divides the initial sequence into multiple subsequences according to the number of attention heads, determines the sending order of each subsequence based on the order of the index numbers of the attention heads corresponding to each subsequence, and performs multiple attention calculations according to the sending order of each subsequence and the receiving order of each second sub-vector.

[0019] In simple terms, when splitting the initial vector according to its length, starting from the second attention calculation, the sequence involved in each calculation is fixed, that is, the initial sequence local to the node. When splitting the initial vector according to the number of attention heads, starting from the second attention calculation, the sequence involved in each calculation is the local subsequence Q of the node. ij The order changes as the sub-vectors involved in the calculation change, and the order is also determined by the processing order.

[0020] In the above implementation, when segmenting the initial vector according to the number of attention heads, the initial sequence is also segmented according to the number of attention heads. This is because in the attention mechanism of large language models, each attention head in each attention layer is used to focus on a part of the input sequence, and multiple attention heads process the complete input sequence in parallel. Therefore, when segmenting the initial vector according to the number of attention heads, the input sequence needs to be segmented, and different attention heads are assigned corresponding input sequences to complete the corresponding attention calculations, thereby ensuring that the attention calculation results obtained after segmenting the initial vector are correct.

[0021] In one possible implementation, the first node includes a processing module and a communication module. The processing module is used to perform the first attention calculation and multiple attention calculations, while the communication module is used to send vectors to the second node and receive vectors sent by the first node. The processing module and the communication module process tasks in parallel.

[0022] In the above implementation, the first node can achieve parallel sending, receiving, and processing. In this way, the processing module continuously performs attention calculations on a per-subvector basis, while the communication unit continuously receives subvectors sent by the previous node and continuously sends subvectors to the next node until the first node receives subvectors from all nodes and has completed the corresponding attention calculations, thereby obtaining the attention calculation result of the first node. Throughout the entire processing flow, the processing module of the first node never enters an idle state, thus avoiding resource waste.

[0023] In one possible implementation, the machine learning system is used to train or infer a large language model, which includes a transformer model. The initial vectors include keyword vectors K and value vectors V in the attention layer of the large language model, and the initial sequence includes the input sequence of the attention layer.

[0024] Secondly, a computing node is provided, which is deployed in a machine learning system. The machine learning system includes multiple computing nodes, and the network structure between the multiple computing nodes is a ring network structure. The computing node includes: an acquisition unit for acquiring the initial sequence and initial vector to be processed in the attention layer; a segmentation unit for segmenting the initial vector to obtain multiple first sub-vectors; a computing unit for performing the first attention calculation based on the initial sequence and initial vectors; a communication module for sending multiple first sub-vectors to a second node and receiving multiple second sub-vectors sent by a third node; the computing unit is also used to perform multiple attention calculations based on the initial sequence and multiple second sub-vectors to obtain the attention calculation result of the first node.

[0025] Implementing the computation nodes described in the second aspect, when the model performs attention calculations at each layer, each computation node, after obtaining the initial sequence and initial vector assigned to it, first divides the initial vector into multiple first sub-vectors. Then, while performing the first attention calculation based on the initial sequence and initial vector, it continuously sends the first sub-vector to the next node in units of sub-vectors and continuously receives the second sub-vector sent by the previous node. This ensures that after the first attention calculation is completed, each node can perform attention calculations based on the second sub-vectors received from the previous node, and so on, until all nodes' sub-vectors have been calculated, thereby obtaining the calculation result of that attention layer. Since each node has already received the sub-vectors required for the next attention calculation after the previous attention calculation is completed, each node will not be idle or waiting, avoiding resource waste, improving attention calculation efficiency, and thus improving the overall training or inference efficiency of the model.

[0026] In one possible implementation, a segmentation unit is used to determine the sending order of each first sub-vector; a communication module is used to send multiple first sub-vectors to the second node according to the sending order, receive multiple second sub-vectors sent by the third node, obtain the receiving order of each second sub-vector, and send multiple second sub-vectors to the second node according to the receiving order; and a calculation unit is used to perform multiple attention calculations according to the receiving order, based on the initial sequence and multiple second sub-vectors.

[0027] In one possible implementation, a segmentation unit is used to segment the initial vector into multiple first sub-vectors according to the length of the initial vector, and to determine the sending order of each first sub-vector based on the position of each first sub-vector in the initial vector. A calculation unit is used to perform multiple attention calculations based on the initial sequence and multiple second sub-vectors, and to merge and update the calculation results obtained from each attention calculation with the calculation results of the previous attention calculations.

[0028] In one possible implementation, a segmentation unit is used to segment the initial vector into multiple first sub-vectors according to the number of attention heads, and to determine the sending order of each first sub-vector based on the order of the index numbers of the attention heads corresponding to each first sub-vector. A calculation unit is used to perform multiple attention calculations based on the initial sequence and multiple second sub-vectors, and to merge and update the calculation results obtained from every m attention calculations with the calculation results of previous attention calculations, where m is the number of first sub-vectors.

[0029] In one possible implementation, a segmentation unit is used to segment the initial sequence into multiple subsequences according to the number of attention heads, and to determine the sending order of each subsequence based on the order of the index numbers of the attention heads corresponding to each subsequence. A calculation unit is used to perform multiple attention calculations according to the sending order of each subsequence and the receiving order of each second sub-vector.

[0030] In one possible implementation, the first node includes a processing module and a communication module. The processing module is used to perform the first attention calculation and multiple attention calculations, while the communication module is used to send vectors to the second node and receive vectors sent by the first node. The processing module and the communication module process tasks in parallel.

[0031] In one possible implementation, the machine learning system is used to train or infer a large language model, which includes a transformer model. The initial vectors include keyword vectors K and value vectors V in the attention layer of the large language model, and the initial sequence includes the input sequence of the attention layer.

[0032] Thirdly, a computing device is provided, the computing device including a processor and a memory, the memory for storing instructions and the processor for executing the instructions, such that the computing device implements the method described in the first aspect.

[0033] Fourthly, a chip is provided, which includes a power supply unit and a processing module. The power supply unit supplies power to the processing module, enabling the processing module to implement the method described in the first aspect.

[0034] Fifthly, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and the instructions are executed by a computing device or a cluster of computing devices to implement the method described in the first aspect.

[0035] In a sixth aspect, a computing device cluster is provided, the computing device cluster including at least one computing device, each of the at least one computing device including a processor and a memory, the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to implement the method as described in the first aspect.

[0036] In a seventh aspect, a computer program product comprising instructions is provided, the computer program product including instructions capable of running on a computing device or stored in any available medium, and when the computer program product is run on a computing device or a cluster of computing devices, causing the computing device or cluster of computing devices to perform the method described in the first aspect.

[0037] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the steps of ring attention in an application scenario.

[0039] Figure 2 This is an architecture diagram of a machine learning system provided in this application;

[0040] Figure 3 This is an example diagram of a machine learning system deployed in a cloud environment according to this application;

[0041] Figure 4 This is a flowchart illustrating the steps of the attention calculation method provided in this application;

[0042] Figure 5 This is a schematic diagram of the attention calculation steps of the attention layer under a partitioning strategy provided in this application;

[0043] Figure 6 This is a schematic diagram of the attention calculation steps for the next attention layer under another partitioning strategy provided in this application;

[0044] Figure 7 This is a schematic diagram of the structure of a computing node provided in this application;

[0045] Figure 8 This is a schematic diagram of the structure of a computing device provided in this application;

[0046] Figure 9 This is an example diagram of a computing device cluster provided in this application. Detailed Implementation

[0047] First, the application scenarios involved in this application will be explained.

[0048] Large language models (LLMs) are deep learning models trained on massive amounts of text data. They are typically based on a transformer architecture and use attention mechanisms to capture contextual information, enabling them to understand and generate natural language. Generally, an LLM consists of an input layer, an attention layer, and an output layer. The input layer converts the input text into multiple word vectors that the model can process. The attention layer performs attention calculations on these word vectors to obtain attention vectors, which describe the importance of different word vectors in the model's inference process. This allows the LLM to focus on important information in the input text, resulting in more reasonable output text. The output layer generates the output based on the word vectors and the attention vectors. The attention layer is the core mechanism of LLMs, enabling them to be trained on large amounts of text data. A well-trained LLM can not only generate natural language questions but also deeply understand text semantics and handle various natural language tasks, such as text summarization, question answering, and translation.

[0049] Typically, large language models consist of multiple attention layers connected end-to-end. Each attention layer takes the output of the previous one as its input, and the first attention layer takes a sequence of word vectors extracted by the input layer as its input. After each attention layer obtains the input sequence Q to be processed, it interpolates Q with the first matrix W. k Perform matrix multiplication to obtain the key (key, K) vector, and then multiply it with the second matrix W. v Perform matrix multiplication to obtain a value vector (value, V), where the first matrix W k The second matrix W v The first matrix W of different attention layers can be learned during the model training process. k The second matrix W v These are different, used to extract different types of features from the input vector. Each attention layer obtains an attention score for each position in the input sequence by calculating the dot product between the input sequence Q and the vector K. The attention score reflects the importance of each position in the input sequence during the inference process. Then, a softmax function is applied to convert all attention scores into a probability distribution. Based on this probability distribution, the V vector is weighted and summed so that the V1 vector corresponding to each position in the input sequence is multiplied and summed with its corresponding attention score to obtain the attention vector output by this layer. The attention vector output by each attention layer can be used as input for the next attention layer, allowing the next attention layer to repeat the above process, generating the Q, K, and V vectors of the next attention layer, performing the corresponding attention calculations to obtain the attention vector, and so on, until the last attention layer outputs its attention vector. The attention vector output by the last attention layer is obtained by fusing the outputs of multiple attention layers, integrating global contextual information. This attention vector is a highly abstract representation containing a comprehensive understanding of the most important features of the input data. This attention vector enters the output layer, which can output more accurate results.

[0050] While such large language models offer excellent natural language generation performance, the continuous development of machine learning techniques has made supporting extremely long input sequences a crucial feature. This is because long input sequences allow models to remember and utilize contextual information from further back, generating more coherent and logically rigorous text. However, as described above, each attention layer in a large language model requires attention calculation based on the input sequence Q. Therefore, the computational complexity of a large language model is closely related to the length of the input sequence. Such extremely long input sequences lead to significant computational and storage overhead, impacting the computational efficiency of the attention layers and consequently affecting the model's training efficiency.

[0051] To alleviate the computational and storage pressure of training long sequences in large language models, Ring Attention was developed. Ring Attention is an improved attention mechanism that divides the input sequence, K-vector, and V-vector of the attention layer according to the number of nodes and distributes them evenly across all nodes. This ensures that each node receives a portion of the input sequence, a portion of the K-vector, and a portion of the V-vector. The network structure among multiple nodes is a ring network. Each node sends its K-vector and V-vector to the next node and performs attention calculations based on the K-vector and V-vector sent by the previous node and the input sequence at its current node. After multiple rounds of sending, receiving, and calculation, each node can calculate the attention result corresponding to its input sequence based on the K-vector and V-vector from all nodes. The attention calculation results from each node are then aggregated to obtain the attention result corresponding to the complete input sequence, thus achieving distributed processing of the attention mechanism.

[0052] For example, Figure 1 This is a flowchart illustrating the steps of ring attention in a specific application scenario. For example... Figure 1 As shown, assume that there are 3 nodes participating in ring attention, namely node 1, node 2 and node 3. There is peer-to-peer (P2P) communication between the nodes. Specifically, node 1 sends data to node 2, node 2 sends data to node 3, and node 3 sends data to node 1.

[0053] After the model begins training, when the task reaches the attention layer, the input sequence Q, key (key, K) vector, and value (value, V) for each attention layer can be segmented according to the number of nodes and then distributed to each node. This results in each node receiving an initial sequence Qi, an initial K vector, and an initial V vector. Specifically, node 1 is assigned Q1, K1, V1; node 2 is assigned Q2, K2, V2; and node 3 is assigned Q3, K3, V3.

[0054] Because computation can be handled by the node's processing module and communication by its communication module, computation and communication are processed in parallel on each node. Taking node 1 as an example, node 1 performs attention calculation based on its assigned Q1, K1, and V1. Simultaneously, node 1 sends K1 and V1 to node 2 and receives K3 and V3 from node 3. Similarly, node 2 performs attention calculation based on its assigned Q2, K2, and V2. Simultaneously, node 2 sends K2 and V2 to node 3 and receives K1 and V1 from node 1. Similarly, node 3 performs attention calculation based on its assigned Q3, K3, and V3. Simultaneously, node 3 sends K3 and V3 to node 1 and receives K2 and V2 from node 2.

[0055] In this way, after each node has completed its attention calculation and received the key-value pairs (KV) from the previous node, it can perform a second round of attention calculation based on its local Q and the received KV from the previous node. Simultaneously, it sends the newly received KV from the previous node to the next node, waiting for the previous node to send new KV. Taking node 1 as an example again, after completing its attention calculation based on Q1, K1, and V1, and receiving K3 and V3 from node 3, node 1 can perform its own attention calculation based on Q1, K3, and V3, and simultaneously send K3 and V3 to node 2, waiting for node 3 to send K2 and V2. Similarly, nodes 2 and 3 can handle similar operations, which will not be repeated here.

[0056] It should be noted that after each attention calculation, the result of this attention calculation needs to be merged and updated with the previous attention calculation results. This is usually achieved through a softmax operation. Then, the next attention calculation is performed. Since the first attention calculation does not have the previous attention calculation results, it does not need to be merged and updated. However, starting from the second attention calculation, each attention calculation result needs to be merged and updated with the previous attention calculation results, and so on, until the final result is obtained.

[0057] After three rounds of computation and communication, each node can calculate the attention calculation result corresponding to 1 / 3 of the length of the input sequence on its own node based on the K vector and V vector on the three nodes. After the three nodes summarize the calculation results, they can obtain the attention calculation result corresponding to the complete input sequence, thus achieving distributed processing to implement the attention mechanism.

[0058] However, in the above process, each node needs to wait until the previous node has finished transmitting its K and V vectors before it can begin the next attention calculation. Taking node 1 as an example, when node 1 finishes the attention calculation based on its assigned Q1, K1, and V1, it has not yet received K3 and V3 sent by node 3. At this time, it needs to wait idle for a period of time. Figure 1 The first dark rectangular block represents this. Attention calculation based on Q1, K3, and V3 can only begin after K3 and V3 are received. When the calculation is complete, K2 and V2 from node 3 have not yet been received, so there is another period of idle waiting. Figure 1 Represented by the second dark rectangle, similarly, nodes 2 and 3 will also have multiple idle waiting times as shown in node 1. If multiple communications are required, each node will have a large amount of idle waiting time, which will not only waste resources but also affect the efficiency of model attention calculation, thereby affecting training efficiency and inference efficiency.

[0059] In summary, when the model uses the ring attention mechanism for attention, each node needs to wait idle until it receives the K and V vectors sent by the previous node after completing the calculation. Only after receiving the calculation result from the previous node can the node continue to perform attention calculation based on the K and V vectors sent by the previous node. After multiple communications, the node will have a lot of idle waiting time, which will waste resources and affect the overall training efficiency.

[0060] To address the issue of excessive idle time at training nodes during attention computation using the ring attention mechanism, leading to resource waste and reduced efficiency, this application provides an attention computation method. During attention computation at each layer, after each node receives its assigned initial sequence Q, initial vector K, and initial vector V, it first divides the initial vectors K and V into multiple sub-vectors K and V. Then, while performing the first attention computation based on the initial sequence Q, initial vector K, and initial vector V, it continuously sends sub-vectors to the next node, unit by unit. This ensures that after the first attention computation is complete, each node can perform attention computation based on the received sub-vectors from the previous node. This process continues until all nodes have completed their sub-vector computations, thus obtaining the result for that attention layer. Since each node obtains the necessary sub-vectors for the next attention computation after the previous one, no node experiences idle time, avoiding resource waste and improving attention computation efficiency, thereby enhancing the overall training or inference efficiency of the model.

[0061] The attention calculation method, machine learning system, and related equipment provided in this application will be explained below with reference to the accompanying drawings.

[0062] Figure 2 This is an architecture diagram of a machine learning system provided in this application, such as... Figure 2 As shown, the architecture includes a machine learning system 1000 and clients 2000, wherein the machine learning system 1000 can establish communication connections with one or more clients 2000. Figure 2Taking a client 2000 as an example for illustration, this application does not impose specific limitations. The machine learning system 1000 and the client 2000 can communicate via a wired network or a wireless network. Wired networks include Ethernet, wired lines, cables, etc., while wireless networks include wireless local area networks (Wi-Fi), cellular networks, etc., and may also include the Internet, local area networks (LANs), etc., which simultaneously support wired and wireless connections. Of course, other network types may also be included, which are not specifically limited in this application.

[0063] Client 2000 is deployed on terminal devices or computing devices to enable human-computer interaction. Terminal devices include personal computers, smartphones, wearable devices, handheld processors, tablets, mobile laptops, augmented reality (AR) devices, virtual reality (VR) devices, smart conferencing devices, etc., without specific limitations. Computing devices can be bare metal servers (BMS), virtual machines, containers, or storage devices. BMS refers to a general-purpose physical server, such as an ARM server or an x86 server; a virtual machine refers to a complete computer system with full hardware system functionality simulated by software, running in a completely isolated environment. Any task that can be performed on a physical computer can also be performed in a virtual machine. When creating a virtual machine on a computing device, a portion of the physical machine's hard drive and memory capacity needs to be used as the virtual machine's hard drive and memory capacity. Each virtual machine has its own independent basic input / output system (BIOS), hard disk, and operating system, and can be operated like a physical machine. A container is a portable software unit that can combine an application and all its dependencies into a single software package. This package is not limited by the underlying host operating system, thus eliminating the need to build complex environments and simplifying the application development and deployment process.

[0064] The machine learning system 1000 is deployed on computing devices or a cluster of computing devices. The description of the computing devices can be found in the previous text and will not be repeated here.

[0065] Furthermore, the machine learning system 1000 includes multiple nodes (or computing nodes). Figure 2The following explanation uses nodes 1, 2, and 3 as examples. In the specific implementation, this application does not limit the number of nodes in the machine learning system 1000. The network structure between multiple nodes is a ring network structure, with each node having two connected nodes. Each node receives data from the previous node and sends data to the next node, thus forming a ring network. Figure 2 The diagram shows a closed ring structure. Each node includes a processing module and a communication module. The processing module is used to perform data processing tasks, and the communication module is used to communicate with other nodes.

[0066] Optionally, each node can be a chip with data processing and data communication capabilities, such as a processor or an accelerator card. A processor refers to a device implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system-on-chip (SoC), or any combination thereof; this application does not impose a specific limitation. An accelerator card refers to a device with good performance when performing machine learning tasks, which can improve the speed and efficiency of neural network training, such as a graphics processing unit (GPU), a neural network processing unit (NPU), a data processing unit (DPU), a tensor processing unit (TPU), or other AI processing chips; this application does not impose a specific limitation. When the node is the aforementioned chip, the processing module on the node can be the chip's computing unit, stream processor, etc., and the communication module can be various interfaces or controllers on the chip.

[0067] Optionally, each node can also be a computing device including the aforementioned chips, such as a bare-metal server (BMS), a virtual machine, or a container. A BMS refers to a general-purpose physical server, such as an ARM server or an x86 server; a virtual machine refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. Any task that can be performed on a physical computer can also be performed in a virtual machine. When creating a virtual machine on a computing device, a portion of the physical machine's hard drive and memory capacity needs to be used as the virtual machine's hard drive and memory capacity. Each virtual machine has an independent basic input / output system (BIOS), hard drive, and operating system, and can be operated like a physical machine. A container is a portable software unit that can combine an application and all its dependencies into a single software package. This package is not limited by the underlying host operating system, thus eliminating the need to build complex environments and simplifying the application development and deployment process. When the node is one of the aforementioned computing devices, such as a server, the processing module can be various processors, and the communication module can be the server's network interface card or bus interface.

[0068] The aforementioned nodes can be held by the user or provided to the user by the cloud service provider. When the nodes are provided to the user by the cloud service provider, the aforementioned nodes can also be referred to as instances. It should be understood that cloud service providers can provide users with GPU instances, TPU instances, virtual machine instances, or container instances specifically for machine learning, etc. Users can purchase usage rights of instances from cloud service providers according to their usage needs, and this application does not make specific limitations.

[0069] Optionally, multiple nodes can establish P2P connections through Huawei Cloud Communication Service (HCCS), NVLink technology, switch chips, etc., or through other communication methods; this application does not impose specific limitations. HCCS is a high-speed interconnect technology for cloud computing scenarios, providing high-speed, low-latency cloud communication services. HCCS enables direct connections between nodes while providing high bandwidth and reliability. NVLink technology is a high-speed interconnect technology for direct communication between GPUs, suitable for high-performance computing and machine learning. NVLink can directly connect multiple GPUs to achieve efficient transmission. It should be understood that the above examples are for illustrative purposes only and this application does not impose specific limitations.

[0070] Optionally, the client 2000 and the machine learning system 1000 can be deployed on the same computing device, or the client 2000 and the machine learning system 1000 can be deployed on different computing devices in the same computing device cluster, or the client 2000 and the machine learning system 1000 can be deployed on different computing devices in different computing device clusters. This application does not make any specific limitations.

[0071] Optionally, the client 2000 can be software or an application running on a user-controlled terminal device or computing device, such as a personal computer (PC) client, a web client accessed via a browser, an application (APP) client running on a mobile terminal, or a cloud platform console; this application does not impose specific limitations. Users holding the client 2000 can be developers training the model, such as programmers or AI engineers, or personnel using the model to perform inference, such as users using after-sales robots for after-sales consultation or chatbots for chatting; this application does not impose specific limitations. It should be understood that the model requires attention calculations during both the inference and training phases; therefore, the machine learning system 1000 provided in this application can be used for both model training and model inference; this application does not impose specific limitations.

[0072] Optionally, the client 2000 can be a model service platform or a machine learning framework client for handling machine learning-related tasks. Such clients provide a series of tools and services for the entire process of defining, training, deploying, inferring, and managing machine learning models. The model service platform can be ModelArts, OpenAI, etc., and the machine learning framework can be TensorFlow, PyTorch, MindSpore, etc. This application does not make any specific limitations.

[0073] Optionally, the client 2000 can also be a client of a cloud platform, used for users to purchase and rent various cloud services. The attention calculation method provided in this application can be one of these cloud services, which users can purchase separately to optimize the attention calculation efficiency during model training or inference. Alternatively, the cloud platform provides users with a comprehensive service, and the attention calculation method provided in this application can be a sub-service of this comprehensive cloud service. For example, if the comprehensive cloud service is an AI cloud service, the attention calculation method provided in this application can be a sub-service of this AI cloud service to optimize the attention calculation efficiency during model training and inference.

[0074] The preceding text has described in detail the possible deployment methods for Machine Learning System 1000 and Client 2000. In actual deployment, flexible deployment can be carried out based on specific application scenarios and business needs. The following section provides examples of specific deployment methods for Machine Learning System 1000 and Client 2000.

[0075] For example, suppose machine learning system 1000 and client 2000 are deployed on an enterprise's internal network. Machine learning system 1000 is located on a server or server cluster purchased by the enterprise, and client 2000 is a remote desktop connection tool or server management console software. Enterprise employees can remotely connect to the server or server cluster where machine learning system 1000 is located through client 2000 and use machine learning system 1000 to optimize the efficiency of attention calculation during model training or model inference.

[0076] For example, machine learning system 1000 can be deployed in a cloud environment. Figure 3 This is an example diagram of a machine learning system deployed in a cloud environment, as provided in this application. Figure 3 As shown, a user can initiate a purchase request for AI cloud services through client 2000. These AI cloud services include sub-services corresponding to the attention calculation method provided in this application. Specifically, after client 2000 sends the purchase request to the cloud platform, the cloud platform can grant client 2000 access to the machine learning system 1000 provided in this application, enabling the user to use the machine learning system 1000 to optimize the attention calculation efficiency during model training or inference.

[0077] The cloud platform also maintains various basic resources, including computing resources, storage resources, network resources, and security resources, to meet the computing needs of the machine learning system 1000 under different scales and loads. Furthermore, these computing resources can be dynamically scaled according to the usage needs of the machine learning system 1000 to ensure the stable operation of the machine learning system 1000 and provide users with reliable model inference services.

[0078] It should be understood that the above application scenarios are for illustrative purposes only. The actual machine learning system 1000 and client 2000 can be flexibly deployed according to actual business needs. These will not be illustrated one by one here.

[0079] In this embodiment, the client 2000 can send a user's training request or inference request to the machine learning system 1000. In response to the training request or inference request, the machine learning system 1000 begins the model training or inference process. When processing attention calculations for multiple attention layers of the model, each attention layer can... Figure 2As shown, the number of nodes is n, and the input sequence Q of the attention layer is divided into n initial sequences Q according to their lengths. i The K vector of this attention layer can be divided into n initial vectors K i The V vector of the attention layer can be divided into n initial vectors V. i Then, use n initial sequences Q i n initial K i Vectors and n initial V i The vector is distributed to n nodes, such that each node receives an initial sequence Qi and an initial sequence K. i Vector and an initial V i vector, Figure 2 Taking n=3 as an example, node 1 obtains Q1, K1 and V1, node 2 obtains Q2, K2 and V2, and node 3 obtains Q3, K3 and V3.

[0080] Then, each node divides its assigned initial vector into m sub-vectors, where the initial vector K is... i Divide into m subvectors K ij The initial vector Vi is divided into m subvectors V. ij , Figure 2 The example given is m=3, meaning the K1V1 vector of node 1 is divided into K... 11 V 11 K 12 V 12 and K 13 V 13 The K2V2 vector of node 2 is divided into K 21 V 21 K 22 V 22 and K 23 V 23 The K3V3 vector of node 3 is split into K... 31 V 31 K 32 V 32 and K 33 V 33 .

[0081] Next, each node begins working with its initially assigned sequence Qi and initial vector K. i and the initial vector V i The first attention calculation is performed. Simultaneously, communication occurs with other nodes in units of sub-vectors. Specifically, each node sequentially sends multiple sub-vectors to the next node and receives multiple sub-vectors sent by the previous node. Taking node 1 as an example, node 1 performs attention calculation based on Q1, K1, and V1, and simultaneously sends K to node 2. 11 V 11Receive K sent by node 3 31 V 31 .

[0082] Finally, after the first attention calculation, each node begins attention calculation and communication simultaneously, working on a sub-vector basis. Taking node 1 as an example, after completing the attention calculation based on Q1, K1, and V1, node 1 has already received K. 31 V 31 At this point, we can base it on Q1, K 31 and V 31 Attention calculations are performed, and sub-vectors are continuously sent to node 2 and received from node 3 in sequence, and so on.

[0083] In this way, after each node completes the first attention calculation, even if it has not received the complete initial K vector and initial V vector sent by the previous node, it will definitely have received the sub-vector sent by the previous node. At this time, it can continue to complete the attention calculation based on the sub-vector, without having to wait for the complete initial K vector and initial V vector sent by the previous node to perform the calculation, thus avoiding the problem of nodes waiting in vain.

[0084] The size of m can be set based on experience. It is necessary to ensure that the communication time of at least one sub-vector is less than the calculation time of the complete initial K vector and initial V vector. This is to ensure that the node does not have the problem of waiting in vacancies. The specific size of m can be flexibly set according to the actual business scenario. This application does not limit the size of m.

[0085] Similarly, nodes 2 and 3 can also perform parallel computation and communication on a sub-vector basis; see the details for reference. Figure 2 The description of node 1 will not be repeated here. Furthermore, each node can continuously send the initial K vector and initial V vector subvectors received from the previous node to the next node until each node obtains the initial K vector and initial V vector of all nodes. Taking node 1 as an example, after node 1 obtains the complete vectors of K3V3 and K2V2, it can summarize the results of the previous attention calculations to complete the attention calculation of this attention layer.

[0086] Similarly, other attention layers can also be arranged according to... Figure 2 The attention calculation method shown typically involves multiple attention layers connected end-to-end. The input of each attention layer is the output of the previous attention layer. Using the technical solution provided in this application, each attention layer can avoid issues such as empty nodes during attention calculation, thus avoiding resource waste and significantly improving the efficiency of attention calculation.

[0087] In summary, the machine learning system provided in this application, when performing attention calculations at each layer of the model, after each node obtains its assigned initial sequence Q, initial vector K, and initial vector V, first divides the initial vector K and initial vector V into multiple sub-vectors K and sub-vector V. Then, while performing the first attention calculation based on the initial sequence Q, initial vector K, and initial vector V, it continuously sends sub-vectors to the next node in units of sub-vectors and continuously receives sub-vectors sent by the previous node in units of sub-vectors. In this way, after the first attention calculation is completed, attention calculations can be continuously performed based on the sub-vectors received from the previous node, receiving sub-vectors and calculating simultaneously, until the sub-vectors of all nodes have been calculated, thereby obtaining the calculation result of the attention layer. Since each node has already obtained the sub-vectors required for the next attention calculation after the previous attention calculation is completed, the processing module of each node will not be idle or waiting, avoiding resource waste, improving attention calculation efficiency, and thus improving the overall training or inference efficiency of the model.

[0088] The above text combined Figure 2 and Figure 3 The machine learning system provided in this application has been explained and described below, in conjunction with... Figure 4 The attention calculation method provided in this application is explained and described. This method can be applied to, for example... Figure 2 and Figure 3 The machine learning system 1000 shown.

[0089] Figure 4 This is a flowchart illustrating the steps of the attention calculation method provided in this application, as follows: Figure 4 As shown, the method may include the following steps:

[0090] S310: Client 2000 sends a model training request to machine learning system 1000.

[0091] It should be noted that the machine learning system 1000 provided in this application is used to optimize the efficiency of attention calculation. Attention calculation occurs in both the training and inference phases of the model. Therefore, the machine learning system 1000 can handle both model training and inference requests. Figure 4 Taking the processing of model training requests as an example, the processing flow for model inference requests is similar, and will not be repeated here.

[0092] In specific implementations, the model training request is used to train models that include attention layers, such as large language models, including but not limited to generative pre-trained transformer (GPT) models, bidirectional encoder representations from transformers (BERT), extreme language network (XLNet), a lite BERT (ALBERT), etc. This application does not impose specific limitations.

[0093] S320: The first node 1001 of the machine learning system 1000 obtains the initial sequence Q1, initial vector K1, and initial vector V1 of the current attention layer assigned to the first node.

[0094] In its specific implementation, when the machine learning system 1000 receives a model training request, it can begin training the corresponding large language model. The network layers of the large language model can include an input layer, an attention layer, and an output layer. The input layer is used to convert the natural language input text into multiple word vectors that the model can process. The attention layer is used to perform attention calculation based on the multiple word vectors of the input text to obtain an attention vector. This attention vector is used to describe the importance of different word vectors in the model's inference process, enabling the large language model to focus on important information in the input text, thereby generating more reasonable output text. The output layer is used to generate the output result based on the multiple word vectors and the attention vector.

[0095] As discussed above, large language models typically consist of multiple attention layers connected end-to-end. Each attention layer takes the output of the previous layer as its input. Each layer obtains its corresponding K and V vectors based on the input sequence Q, and then performs attention calculations based on these vectors until the last attention layer outputs an attention vector. This final attention vector is a fusion of the outputs from multiple attention layers, integrating global contextual information. It is a highly abstract representation containing a comprehensive understanding of the most important features of the input data. This attention vector then enters the output layer, enabling a more accurate output.

[0096] In this embodiment, in response to a model training request sent by the client 2000, the machine learning system 1000 begins training a large language model. The machine learning system 1000 then calculates the attention vector for each attention layer sequentially, following the order of the attention layers. Each attention layer, based on ring attention technology, divides the input sequence Q, K vector, and V vector into multiple parts according to the number of nodes, and distributes them to multiple nodes for distributed attention computation. Each node obtains an initial sequence Q. i Initial vector K i and the initial vector V i .

[0097] In simple terms, the first node 1001 will obtain the initial sequence Q1, initial vector K1, and initial vector V1 assigned to it for the current attention layer. Similarly, the second node 1002 will obtain the initial sequence Q2, initial vector K2, and initial vector V2 assigned to it for the current attention layer, and the third node 1003 will obtain the initial sequence Q3, initial vector K3, and initial vector V3 assigned to it for the current attention layer. If there are more nodes, other nodes can also perform the same steps to obtain the initial Q, K, and V assigned to them.

[0098] exist Figure 4 In the example shown, the node preceding the first node 1001 is the third node 1003, and the next node is the second node 1002. In other words, the first node 1001 can be understood as... Figure 2 In the embodiment, node 1 and the second node 1002 can be understood as... Figure 2 In the embodiment, node 2 and the third node 1003 can be understood as... Figure 2 Node 3 in the embodiment.

[0099] It should be understood that Figure 4 The technical solution of this application is described using the first node 1001 as an example. Therefore, the steps of the second node 1002 and the third node 1003 obtaining the initial sequence Q, initial vector K and initial vector V assigned to them are not shown. However, in the actual processing, each node will execute S320. The example will not be repeated here.

[0100] S330: The first node 1001 of the machine learning system 1000 divides the initial vector K1 and the initial vector V1 into multiple sub-vectors according to the partitioning strategy, and each sub-vector corresponds to a sending order.

[0101] In the specific implementation, the first node 1001 is divided into m sub-vectors K1 according to the partitioning strategy. 11 ,K 12 ,…,K 1mThe initial vector V1 is divided into m sub-vectors V 11 V 12 ,…,V 1m Similarly, the second node 1002 divides the initial vector K2 into m sub-vectors K. 21 ,K 22 ,…,K 2m Divide the initial vector V2 into m sub-vectors V 21 V 22 ,…,V 2m The third node 1003 divides the initial vector K3 into m sub-vectors K. 31 ,K 32 ,…,K 3m The initial vector V3 is divided into m sub-vectors V 31 V 32 ,…,V 3m .

[0102] The partitioning strategy is used not only to segment the initial vector but also to assign a transmission order to each segmented sub-vector. This transmission order indicates the order in which the sub-vector is communicated. It should be understood that each node uses the same partitioning strategy to segment the initial vector K and initial vector V, and uses the same partitioning strategy to assign a transmission order to each segmented sub-vector. This consistency ensures that when transmitting sub-vectors between nodes, the transmission order of the sub-vectors is consistent across nodes. Correspondingly, the order of attention calculation is also consistent, thus ensuring that each node's sub-vector is calculated only after its dependent sub-vectors have been calculated, avoiding calculation errors.

[0103] Optionally, when dividing the initial vector into multiple sub-vectors according to the partitioning strategy, the initial vector can be evenly divided into m sub-vectors according to its length, and the sending order of the sub-vectors can be determined according to their positions in the initial vector. It should be understood that since attention calculation involves performing local calculations at each position in the input sequence and then summing the results of these local calculations to obtain the final result, there is no dependency between each local calculation, allowing for parallel processing. Therefore, partitioning the vector into sub-vectors according to its length will not affect the overall calculation.

[0104] For example, if vector K1 has a length of 3X and m = 3, then the initial vector K1 can be divided into 3 sub-vectors K. 11 K 12 K 13 Each subvector has a length of X, where K 11 It can be the portion of the K1 vector from position 0 to position X, where K... 12 It can be the portion of vector K1 from position X to position 2X, where K... 13It can be the portion of vector K1 from position 2X to position 3X, and correspondingly, the sending order from earliest to latest is K... 11 K 12 K 13 Similarly, vector V1 can also be partitioned in the same way. It should be understood that the above examples are for illustration only and are not intended to be specific limitations in this application.

[0105] Optionally, when dividing the network into multiple sub-vectors according to the partitioning strategy, the initial vector can be evenly divided into m sub-vectors according to the number of attention heads in the network layer, and the initial sequence can be divided into m sub-sequences. The processing order of the sub-sequences and the sending order of the sub-vectors can be determined according to the index order of the attention heads. It should be understood that an attention layer may include multiple attention heads, each of which performs attention calculations in its independent subspace. The calculation results obtained by multiple attention heads in parallel calculations are concatenated to obtain the output of the attention layer. The calculations between each attention head are independent and can be processed in parallel. Therefore, dividing the network into sub-vectors according to the number of attention heads will not affect the overall calculation result.

[0106] The index of an attention head is used to indicate its position among multiple attention heads, which is used to distinguish different attention heads so that specific operations and processing can be performed in the model. For example, a network layer includes 32 attention heads, which can be assigned an index from 0 to 31. For example, attention head 0 processes the first part of the input sequence Q, attention head 1 processes the second part of the input sequence Q, and so on.

[0107] For example, if the attention layer has 32 attention heads and m = 4, then the initial vector K1 is divided into 4 sub-vectors K. 11 K 12 K 13 K 14 Each subvector corresponds to 8 attention heads. Subvector K 11 For the first 8 attention heads, the subvector K 12 For attention heads 9-16, subvector K 13 For attention heads 17-24, subvector K 14 Corresponding to the 25th-32nd attention heads, the corresponding sending order from morning to evening is K. 11 K 12 K 13 K 14 Similarly, the initial sequence Q1 and the initial vector V1 can also be divided into multiple subsequences and subvectors in the same way. This will not be repeated here. It should be understood that the above examples are for illustration only and this application does not impose any specific limitations.

[0108] In summary, assuming the initial sequence Q is assigned to the nodes... i Initial vector K i and the initial vector V i It is a four-dimensional matrix (s,b,x,h) or (s,b,x,h), where s is the length of the initial sequence assigned to the current node, b is the batch size of the current node, x is the number of heads in the multi-head attention mechanism, and h is the hidden layer dimension of each attention head.

[0109] If the partitioning strategy used is to split according to the vector length, and the number of splits is m, then after the split, due to the initial sequence Q under this strategy... i It does not participate in the partitioning, so its four-dimensional matrix remains (s,b,x,h) or (b,s,x,h), with the initial vector K. i and initial vector V i Participating in the segmentation, its subvector K ij and V ij The four-dimensional matrix is ​​(s / m,b,x,h) or (b,s / m,x,h).

[0110] If the partitioning strategy used is to segment based on the number of attention heads, and the number of segments is m, then after segmentation, due to the initial sequence Q under this strategy... i It also needs to participate in the segmentation, so sequence Q i The corresponding subsequence Q ij The four-dimensional matrix is ​​(s,b,x / m,h) or (b,s,x / m,h), and the initial vector K i and initial vector V i Participating in the segmentation, its subvector K ij and V ij The four-dimensional matrix is ​​(s,b,x / m,h) or (b,s,x / m,h).

[0111] S340: The first node 1001 of the machine learning system 1000 performs the first attention calculation based on the initial sequence Q1, the initial vector K1, and the initial vector V1.

[0112] In the specific implementation, the algorithm used for attention calculation can be the Flash Attention algorithm, or other algorithms can be used. The specific choice depends on the actual processing situation, and this application does not impose any specific limitations.

[0113] It should be noted that when the first node 1001 executes S340, other nodes will also execute according to the initial sequence Q assigned to them. i Initial vector K i and the initial vector V i Then, perform the first attention calculation. Figure 4Taking the first node 1001 as an example, the attention processing flow of each node is described. In fact, other nodes are also executing S340 at the same time, which means that... Figure 4 The second node 1002 will perform the first attention calculation in S340 based on the assigned initial sequence Q2, initial vector K2, and initial vector V2. The third node 1003 will perform the first attention calculation in S340 based on the assigned initial sequence Q3, initial vector K3, and initial vector V3.

[0114] S350A: The first node 1001 of the machine learning system 1000 sends multiple sub-vectors to the next node in sequence.

[0115] In the specific implementation, the first node 1001 sends data to the next node in the order of sending, using sub-vectors as units; that is, it sends one sub-vector K each time. ij and one subvector V ij Subvectors sent in the order of priority will be transmitted first. For example, if K is sent first... 11 V 11 Send K again 12 V 12 Send K again 13 V 13 The sending order is determined by the partitioning strategy of S330. It should be noted that, regardless of the partitioning strategy used, the first node 1001 only sends the subvector, and the subsequence Q is not sent to the next node.

[0116] Similarly, other nodes will also execute S350A, sending multiple sub-vectors to the next node according to the order in which the sub-vectors were sent. In other words, Figure 4 The second node 1002 in S350A will send multiple sub-vectors to the third node 1003 in the sending order, and the third node 1003 will send multiple sub-vectors to the first node 1001 in the sending order.

[0117] S350B: The first node 1001 of the machine learning system 1000 sequentially receives multiple sub-vectors sent by the previous node.

[0118] In the specific implementation, the first node 1001 receives data sent by the previous node in units of sub-vectors, that is, it receives one sub-vector K each time. ij and one subvector V ij The previous node will also send sub-vectors to the first node 1001 in the order of transmission.

[0119] Similarly, other nodes will also execute S350B, receiving multiple sub-vectors sent by the previous node, meaning that... Figure 4The second node 1002 in the S350B will receive multiple sub-vectors sent by the first node 1001, and the third node 1003 will receive multiple sub-vectors sent by the second node 1002 in the S350B.

[0120] In specific implementation, S340, S350A, and S350B can be performed simultaneously. Simultaneous execution means that the three steps start executing at the same time, or the three steps do not start executing at the same time, but the execution time periods overlap. This application does not make specific limitations.

[0121] by Figure 2 For example, assume that the K1V1 vector on node 1 is divided into K... 11 V 11 K 12 V 12 and K 13 V 13 The K3V3 vectors on node 3 are all divided into K... 31 V 31 K 32 V 32 and K 33 V 33 Then, the processing module and communication module of each node begin to process tasks in parallel. The processing module executes S340 to complete the first attention calculation based on the initial sequence and initial vector. At the same time, the communication module executes S350A and S350B to start sending and receiving sub-vectors. Figure 2 In the example, during the first communication, node 1 sends K to node 2. 11 V 11 Receive K sent by node 3 31 V 31 During the second communication, Node 1 sends K to Node 2. 12 V 12 Receive K sent by node 3 32 V 32 During the third communication, Node 1 sends K to Node 2. 13 V 13 Receive K sent by node 3 32 V 32 This process continues until node 1 obtains the sub-vectors of all nodes and gets the attention calculation result corresponding to that node.

[0122] It should be understood that Figure 2 In the example, the sending and receiving times are the same each time. However, in actual processing, there may be network fluctuations or latency between nodes, and the sending and receiving times may not be the same. For example, node 1 may send K. 11 V 11 After it was finished, K 31 V 31Before all data has been received, node 1 can continue sending K. 12 V 12 No need to wait to receive K 31 V 31 Send it again. Figure 2 For ease of understanding, this explanation uses the example of the same sending and receiving time, but this application does not limit this.

[0123] S360: The first node 1001 of the machine learning system 1000 performs multiple attention calculations on a per-subvector basis, based on multiple subvectors sent by the previous node.

[0124] In the specific implementation, after the processing module of the first node 1001 finishes executing S340, it can start to repeatedly execute S360. However, the sub-vectors participating in the attention calculation are different each time S360 is executed. The sub-vectors participating in the calculation are determined according to the order in which the sub-vectors are received.

[0125] Meanwhile, while the processing module repeatedly executes S360, the communication module also repeatedly executes S350A and S350B. Because the processing module and the communication module can be executed in parallel, while the processing module repeatedly executes S360, the communication module can repeatedly execute S350A and S350B until the first node 1001 obtains the sub-vectors on all nodes and completes the corresponding attention calculation.

[0126] It should be noted that when each node sends a subvector to the next node, it first sends its own subvector, and then sends the subvector it received from the previous node, until each node has obtained the subvectors from all nodes.

[0127] Optionally, when the subvector partitioning strategy is to split according to the vector length, since the initial sequence is not split under this strategy, the initial sequence Q used for each attention calculation is... i Maintaining inconvenience, only subvector K ij and subvector V ij Furthermore, the order of reception changes continuously, and the sub-vectors with higher reception order will participate in the calculation first.

[0128] For example, suppose the initial vector K3 on the third node 1003 is divided into K according to the vector length. 31 K 32 and K 33 The initial vector V3 is divided into V 31 V 32 and V 33 The sending order of the subvectors is determined by their positions in the initial vector, and this order is K. 31 V 31 >K 32 V32 >K 33 V 33 Then, if the first node 1001 repeatedly executes S350A, it will sequentially receive K sent by the third node 1003. 31 V 31 K 32 V 32 and K 33 V 33 The first node 1001 repeatedly executes S360. The first execution is based on Q1 and K. 31 and V 31 Attention calculation is performed, and the second execution is based on Q1 and K. 32 and V 32 Attention calculation is performed, and the third execution is based on Q1 and K. 33 and V 33 Attention calculations are performed, and so on. The above examples are for illustrative purposes only and are not intended to be specific.

[0129] It should be noted that under this partitioning strategy, after the first node 1001 performs an attention calculation by executing S360 each time, the current calculation result can be merged and updated with the previous calculation result. Since each sub-vector is partitioned according to the vector length, the merging and updating can be implemented based on the softmax function. Then, S360 is executed again, and the calculation and updating are continuously performed. After the last execution of S360, the attention calculation result corresponding to the node can be obtained.

[0130] Optionally, when the sub-vector partitioning strategy is to split according to the number of attention heads, since the initial sequence is also split under this partitioning strategy, the sub-sequence used for each attention calculation will change continuously according to the processing order, and the sub-vector will change continuously according to the receiving order.

[0131] For example, suppose the initial sequence Q1 on the first node 1001 is divided into Q according to the number of attention heads. 11 Q 12 And Q 13 The K1 vector is divided into K 11 K 12 and K 13 The V1 vector is divided into V 11 V 12 and V 13 The initial sequence Q3 on the third node 1003 is divided into Q... 31 Q 32 And Q 33 The K3 vector is divided into K 31 K 32 and K 33 The V3 vector is divided into V31 V 32 and V 33 The processing order of the subsequences is determined according to the index order of the attention heads as Q. 31 >Q 32 >Q 33 The sub-vector receiving order is K 31 V 31 >K 32 V 32 >K 33 V 33 .

[0132] Then, if the first node 1001 repeatedly executes S350A, it will sequentially receive K sent by the third node 1003. 31 V 31 K 32 V 32 and K 33 V 33 Meanwhile, the first node 1001 repeatedly executes S360, with the first execution based on Q. 11 K 31 and V 31 Attention calculation is performed, and the second execution is based on Q. 12 K 32 and V 32 Attention calculation is performed, and the third execution is based on Q. 13 K 33 and V 33 Attention calculations are performed, and so on. The above examples are for illustrative purposes only and are not intended to be specific.

[0133] It should be noted that under this partitioning strategy, after the first node 1001 executes S360 every m times, the result of this calculation can be merged and updated with the results of previous calculations. Since each sub-vector is partitioned according to the attention head, the merging and updating can be achieved based on vector concatenation, where m is the number of sub-vector partitions. In other words, these m calculation results can be concatenated to obtain the calculation result corresponding to a complete initial vector. By repeatedly executing S360 and performing a merging and updating every m calculations, the attention calculation result corresponding to that node is obtained.

[0134] S370: The first node 1001 of the machine learning system 1000 determines whether the attention calculation of the subvectors of all nodes has been completed. If yes, proceed to S380; otherwise, proceed to S350A to S360.

[0135] In simple terms, the processing module of the first node 1001 continuously executes S360, and the communication module continuously executes S350A and S350B until the first node 1001 receives the sub-vectors of all nodes and completes the attention calculation of the sub-vectors of all nodes. Then the attention calculation of this attention layer is completed, and S380 is executed.

[0136] The above processes S340 to S370 involve parallel and repeated operations, making the process relatively complex. To facilitate a better understanding of this application, the following explanation is provided in conjunction with... Figure 5 and Figure 6 The steps for attention calculation using the two partitioning strategies provided in this application are illustrated with examples.

[0137] Figure 5 This is a schematic diagram of the attention calculation steps for an attention layer under a partitioning strategy provided in this application. Figure 5 The partitioning strategy used is to divide the vector into subvectors according to the length of the vector.

[0138] like Figure 5 As shown, the nodes participating in the training in this example include the first node 1001, the second node 1002, and the third node 1003. The previous node of the first node 1001 is the third node 1003, and the next node of the first node 1001 is the second node 1002.

[0139] First, each node executes S320. Specifically, the first node 1001 obtains the initial sequence Q1, the initial vector K1, and the initial vector V1; the second node 1002 obtains the initial sequence Q2, the initial vector K2, and the initial vector V2; and the third node 1003 obtains the initial sequence Q3, the initial vector K3, and the initial vector V3.

[0140] Secondly, each node executes S330. Specifically, the K1V1 vector on the first node 1001 is divided into K... 11 V 11 K 12 V 12 and K 13 V 13 The K2V2 vector on the second node 1002 is divided into K... 21 V 21 K 22 V 22 and K 23 V 23 The K3V3 vector on the third node 1003 is divided into K... 31 V 31 K 32 V 32 and K 33 V 33 .

[0141] Then, each node begins to execute S340, S350A, and S350B simultaneously. Specifically, the processing module of the first node 1001 begins to perform attention calculation based on (Q1, K1, V1), the processing module of the second node 1002 begins to perform attention calculation based on (Q2, K2, V2), and the processing module of the third node 1003 begins to perform attention calculation based on (Q3, K3, V3). Simultaneously, the communication module of the first node 1001 begins to send K to the second node 1002 sequentially. 11 V 11 K 12 V 12 and K 13 V 13 The communication module of the second node 1002 begins to send K sequentially to the third node 1003. 21 V 21 K 22 V 22 and K 23 V 23 The communication module of the third node 1003 begins to send K sequentially to the first node 1001. 31 V 31 K 32 V 32 and K 33 V 33 .

[0142] After each node sends its local initial vector to the next node, it continues sending the received sub-vectors from the previous node, until each node has received the sub-vectors from all nodes. That is, as follows... Figure 5 As shown, after the first node 1001 sends K... 13 V 13 After that, it will continue to send the received K to the second node 1002 in sequence. 31 V 31 K 32 V 32 and K 33 V 33 After the second node 1002 finishes sending K... 23 V 23 After that, it will continue to send the received K to the third node 1003 in sequence. 11 V 11 K 12 V 12 and K 13 V 13 After the third node 1003 finishes sending K... 33 V 33 After that, it will continue to send the received K to the first node 1001 in sequence.21 V 21 K 22 V 22 and K 23 V 33 This continues until all nodes have received the subvector from each node.

[0143] Next, after each node's processing module completes the first attention calculation, it begins to perform multiple attention calculations based on the received sub-vectors and its local initial sequence. The sub-vectors involved in each attention calculation are different. Specifically, the first node 1001 will perform attention calculations based on its initial sequence Q1, sequentially with the K and V sub-vectors received from the third node 1003. The second node 1002 will perform attention calculations based on its initial sequence Q2, sequentially with the K and V sub-vectors received from the first node 1001. The third node 1003 will perform attention calculations based on its initial sequence Q3, sequentially with the K and V sub-vectors received from the second node 1002. For more details, see [link to relevant documentation]. Figure 5 Examples of the processing flow of each node's processing module are provided, but not all examples are given here.

[0144] It should be noted that each node continuously receives the sub-vectors sent by the previous node and then continuously performs attention calculations. The previous node first sends out the sub-vector corresponding to its initial node vector, and then sends out the sub-vectors it receives from other nodes. Therefore, each node continuously performs attention calculations based on the sub-vectors of different nodes until it obtains the sub-vectors of all nodes and obtains the final attention calculation result of that node.

[0145] It should be noted that after each attention calculation, the results are merged and updated based on the previous attention calculation results. This merging and updating can be implemented using the softmax function. Specifically, the first node 1001 is based on (Q1, K... 31 V 31 After performing attention calculations, the attention calculation result for (Q1,K1,V1) can be compared with that for (Q1,K1,V1). 31 V 31 The attention calculation results of (Q1,K) are merged and updated, and then based on (Q1,K) 32 V 32 After performing attention calculations and obtaining the corresponding results, (Q1,K1,V1) and (Q1,K 31 V 31 ) and (Q1,K 32 V 32The calculation results of the first node (1002) and the second node (1003) are merged and updated, and this process is repeated until the final attention calculation result is obtained. Similarly, the second node (1002) and the third node (1003) can be merged and updated in the same way, as detailed in [link to documentation]. Figure 5 The descriptions in the examples will not be repeated here.

[0146] based on Figure 5 As can be seen, when each node's processing module performs the first attention calculation based on the initial vector, the communication module simultaneously sends the sub-vector of its current node to the next node and continuously receives the sub-vector sent by the previous node. In this way, after each node completes the first attention calculation, it can perform the second attention calculation based on the received sub-vector. At the same time, the communication module is still continuously sending and receiving. After the second attention calculation is completed, it can perform the third attention calculation based on the received sub-vector, and so on. This ensures that each node's processing module will not be idle and waiting, thus avoiding resource waste.

[0147] Figure 6 This is a schematic diagram of the attention calculation steps for the next attention layer under another partitioning strategy provided in this application. Figure 6 The partitioning strategy used is to divide the subvectors according to the number of attention heads.

[0148] Figure 6 and Figure 5 The same architecture is used, and the nodes participating in the training include the first node 1001, the second node 1002, and the third node 1003. The previous node of the first node 1001 is the third node 1003, and the next node of the first node 1001 is the second node 1002.

[0149] First, each node executes S320. Specifically, the first node 1001 obtains the initial sequence Q1, the initial vector K1, and the initial vector V1; the second node 1002 obtains the initial sequence Q2, the initial vector K2, and the initial vector V2; and the third node 1003 obtains the initial sequence Q3, the initial vector K3, and the initial vector V3.

[0150] Secondly, each node executes S330, specifically, splitting the initial sequence and initial vector on the node according to the number of attention heads. The Q1 on the first node 1001 is evenly divided into Q... 11 Q 12 And Q 13 The K1V1 vector is divided into K... 11 V 11 K 12 V 12 and K 13 V 13 Q2 on the second node 1002 is evenly divided into Q 21 Q22 And Q 23 The K2V2 vector is divided into K... 21 V 21 K 22 V 22 and K 23 V 23 Q3 on the third node 1003 is divided into Q 31 Q 32 And Q 33 The K3V3 vector is divided into K... 31 V 31 K 32 V 32 and K 33 V 33 .

[0151] Then, each node begins to execute S340, S350A, and S350B simultaneously. Specifically, the processing module of the first node 1001 begins to perform attention calculation based on (Q1, K1, V1), the processing module of the second node 1002 begins to perform attention calculation based on (Q2, K2, V2), and the processing module of the third node 1003 begins to perform attention calculation based on (Q3, K3, V3). Simultaneously, the communication module of the first node 1001 begins to send K to the second node 1002 sequentially. 11 V 11 K 12 V 12 and K 13 V 13 The communication module of the second node 1002 begins to send K sequentially to the third node 1003. 21 V 21 K 22 V 22 and K 23 V 23 The communication module of the third node 1003 begins to send K sequentially to the first node 1001. 31 V 31 K 32 V 32 and K 33 V 33 .

[0152] After each node sends its local initial vector to the next node, it continues sending the received sub-vectors from the previous node, until each node has received the sub-vectors from all nodes. That is, as follows... Figure 5 As shown, after the first node 1001 sends K... 13 V 13 After that, it will continue to send the received K to the second node 1002 in sequence. 31V1, K 32 V 32 and K 33 V 33 After the first node 1001 sends K... 13 V 13 After that, it will continue to send the received K to the second node 1002 in sequence. 31 V1, K 32 V 32 and K 33 V 33 After the first node 1001 sends K... 13 V 13 After that, it will continue to send the received K to the second node 1002 in sequence. 31 V1, K 32 V 32 and K 33 V 33 This continues until all nodes have received the subvector from each node.

[0153] Next, after each node's processing module completes the first attention calculation, it begins to perform multiple attention calculations based on the received sub-vectors and the local sub-sequences. The sub-vectors and sub-sequences involved in each attention calculation are different, and the attention calculations are performed according to the order in which the sub-vectors and sub-sequences were sent.

[0154] For example Figure 6 As shown, after the first node 1001 performs attention calculation based on (Q1, K1, V1), it first calculates (Q 11 ,K 31 V 31 ), then calculate (Q) 12 ,K 32 V 32 ), then calculate (Q) 13 ,K 33 V 33 After the second node 1002 performs attention calculation based on (Q2, K2, V2), it first calculates (Q... 21 ,K 11 V 11 ), then calculate (Q) 22 ,K 12 V 12 ), then calculate (Q) 23 ,K 13 V 13 After the third node 1003 performs attention calculation based on (Q3, K3, V3), it first calculates (Q... 31 ,K 21 V 21 ), then calculate (Q) 32 ,K22 V 22 ), then calculate (Q) 33 ,K 23 V 23 ).

[0155] It's important to note that each node continuously receives sub-vectors from the previous node and performs attention calculations. The previous node first sends out the sub-vector corresponding to its initial node vector, and then sends out the sub-vectors it received from other nodes. Therefore, each node continuously performs attention calculations based on the sub-vectors of different nodes until it obtains the sub-vectors of all other nodes, thus obtaining the final attention calculation result for that node. For example, after the first node (1001) completes its attention calculation (Q... 11 ,K 33 V 33 After that, it will continue to calculate (Q). 11 ,K 21 V 21 ), (Q 12 ,K 22 V 22 ), then calculate (Q) 13 ,K 13 V 13 At this point, the first node 1001 is based on the subvectors of all other nodes, and the calculation can be terminated.

[0156] It should be noted that for each node, attention calculations are performed m times based on sub-vectors and sub-sequences. The results of the current calculation can be merged and updated with the results of the previous calculations. Since each sub-vector is divided according to the attention head, the merging and updating can be achieved based on vector concatenation. m is the number of sub-vector divisions.

[0157] For example Figure 6 As shown, m=3, the first node 1001 performs attention calculations 3 times in sequence to obtain (Q 11 ,K 31 V 31 ), (Q 12 ,K 32 V 32 ), (Q 13 ,K 33 V 33 After calculating the attention result, the process begins merging and updating based on the previous attention calculation results (including the first calculation). After obtaining the updated result, the attention calculation continues to obtain (Q). 11 ,K 21 V 21 ), (Q 12 ,K 22 V 22 ), (Q 13,K 13 V 13 After the attention calculation result of the node is obtained, the merging and updating process begins based on the previous attention calculation results (including the update result of the previous merge update) until all subvectors of the node have participated in the attention calculation, thereby obtaining the final attention calculation result of the node.

[0158] based on Figure 6 As can be seen, when each node's processing module performs the first attention calculation based on the initial vector, the communication module simultaneously sends the sub-vector of its current node to the next node and continuously receives the sub-vector sent by the previous node. In this way, after each node completes the first attention calculation, it can perform the second attention calculation based on the received sub-vector. At the same time, the communication module is still continuously sending and receiving. After the second attention calculation is completed, it can perform the third attention calculation based on the received sub-vector, and so on. This ensures that each node's processing module will not be idle and waiting, thus avoiding resource waste.

[0159] It should be noted that, Figure 5 and Figure 6 The main differences in the processing flow under the two partitioning strategies are:

[0160] first, Figure 5 In the manner shown, starting from the second attention calculation, the sequence involved in each calculation is fixed, that is, the node's local initial sequence Q. i , Figure 6 In the manner shown, starting from the second attention calculation, each sequence involved in the calculation is a subsequence Q local to the node. ij The order changes as the sub-vectors involved in the calculation change, and the order is also determined by the processing order.

[0161] Secondly Figure 5 In the method shown, starting from the second attention calculation, each calculation needs to be merged and updated with the results of the previous calculations after completion. This is achieved using the softmax function. Figure 6 In the method shown, starting from the second attention calculation, the results are merged and updated with the previous calculation results only after every m calculations, which is achieved by vector concatenation.

[0162] It should be understood that Figure 5 In the method shown, each sub-vector is divided according to the length of the initial vector. This means that different positional information of the initial vector is calculated by different nodes. Therefore, after obtaining each calculation result, the softmax function needs to be used to integrate and update it. However, Figure 6In the method shown, each subvector is split according to the number of attention heads. The attention mechanism itself requires each attention head to be computed in parallel and then concatenated based on the computation results of the attention heads. Therefore, the computation results of all attention heads can be obtained and then concatenated in a unified manner, without needing to concatenate them every time they are computed.

[0163] It should be understood that Figure 5 and Figure 6 The examples shown are for illustrative purposes only. This application does not limit the number of nodes n or the number of sub-vectors m. The specific partitioning strategy and various parameters can be flexibly selected and set according to the actual business scenario. This application does not impose any specific limitations.

[0164] In specific implementation, each node can determine whether it has completed the attention calculation for all node subvectors based on the identifiers of the subvectors involved in the attention calculation, or it can determine it based on the number of calculations. If the number of nodes is n and the number of subvector segments is m, then after the first forward calculation, m(n-1) calculations are performed on a subvector basis to determine whether the node has completed the attention calculation for all node subvectors. Of course, other methods can also be used to determine this, and this application does not impose any specific limitations.

[0165] In the specific implementation, after each node obtains its corresponding attention calculation result, the attention calculation results of multiple nodes can be summarized to obtain the attention calculation result of the network layer. The specific algorithm used for attention calculation and summarization can be determined according to the actual business scenario, and this application does not impose any specific limitations.

[0166] S380: The first node 1001 of the machine learning system 1000 determines whether the attention calculation of all attention layers has been completed. If yes, proceed to S390; otherwise, proceed to S320-S370.

[0167] It should be understood that steps S320 to S370, Figure 5 and Figure 6 The examples all use a single attention layer as an example to illustrate how attention calculation is performed. Other attention layers are typically handled by multiple attention layers, which are connected end-to-end. The input of each attention layer is the output of the previous one, so each attention layer can perform attention calculation as described above. The attention vector output by the last attention layer is obtained by fusing the outputs of multiple attention layers, integrating global contextual information. This attention vector is a highly abstract representation containing a comprehensive understanding of the most important features of the input data. This attention vector enters the output layer, allowing for a more accurate output result.

[0168] It should be understood that by using the technical solution provided in this application, each attention layer can avoid problems such as empty nodes when performing attention calculations, thus avoiding resource waste and significantly improving the efficiency of attention calculations.

[0169] S390: The first node 1001 of the machine learning system 1000 obtains the attention calculation result and completes model training based on the attention calculation result.

[0170] Specifically, the final attention calculation result can be fed into the output layer, allowing the output layer to make corresponding decisions and thus complete model training. It should be understood that if the S310 client 2000 sends a model inference request, then the S390 can complete model inference based on the attention calculation result; this will not be illustrated again here.

[0171] It should be understood that the attention calculation method provided in this application not only does not introduce additional storage overhead, but also improves the efficiency of attention calculation. Assuming the number of nodes is n, the attention calculation time for each node (e.g., using the flashAttention algorithm) is t, and each initial vector is divided into m sub-vectors, then each attention layer compared to... Figure 1 The traditional approach shown has a time gain of (1-1 / p)×t. Assuming there are N attention layers, the final time gain is N(1-1 / p)t, which can greatly shorten the attention computation time, thereby reducing the model inference or model training time.

[0172] It should be understood that due to computing hardware and network bandwidth limitations, Figure 5 and Figure 6 The time benefits of the two partitioning strategies shown may differ slightly. The specific partitioning strategy should be selected based on the actual business scenario.

[0173] It should be understood that Figures 4-6 The attention calculation method of this application is described using the perspective of the first node 1001 as an example. The processing flow of other nodes in the machine learning system is the same as that of the first node 1001, and will not be illustrated here.

[0174] In summary, the attention calculation method provided in this application, when the model performs attention calculation at each layer, after each node obtains the initial sequence Q, initial vector K, and initial vector V assigned to it, each node first divides the initial vector K and initial vector V into multiple sub-vectors K and sub-vector V. Then, while performing the first attention calculation based on the initial sequence Q, initial vector K, and initial vector V, it continuously sends sub-vectors to the next node in units of sub-vectors and continuously receives sub-vectors sent by the previous node in units of sub-vectors. In this way, after the first attention calculation is completed, attention calculation can be continuously performed based on the sub-vectors received from the previous node, calculating while receiving sub-vectors, until the sub-vectors of all nodes have been calculated, thereby obtaining the calculation result of the attention layer. Since each node has already obtained the sub-vectors required for the next attention calculation after the previous attention calculation is completed, the processing module of each node will not be idle and waiting, avoiding resource waste, improving attention calculation efficiency, and thus improving the overall training efficiency or inference efficiency of the model.

[0175] The attention calculation method and machine learning system provided in this application have been described in detail above. The following section will combine... Figure 7 The structure of the nodes provided in this application is described.

[0176] Figure 7 This is a schematic diagram of the structure of a computing node provided in this application. The computing node 700 can be the first node 1001 mentioned above. The computing node 700 is... Figure 2 One node in the machine learning system 1000 shown, such as Figure 7 As shown, the computing node 700 includes an acquisition unit 710, a segmentation unit 720, a computing unit 730, and a communication unit 740. It should be understood that... Figure 7 This is an exemplary partitioning method. The computing node 700 can be divided into more or fewer unit modules, and this application does not make any specific limitations.

[0177] The acquisition unit 710, the segmentation unit 720, the calculation unit 730, and the communication unit 740 can all be implemented in software or in hardware. For example, the implementation of the acquisition unit 710 will be described below. Similarly, the implementation of the segmentation unit 720, the calculation unit 730, and the communication unit 740 can refer to the implementation of the acquisition unit 710.

[0178] As an example of a software functional unit, the acquisition unit 710 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, the acquisition unit 710 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0179] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0180] As an example of a hardware functional unit, the acquisition unit 710 may include at least one computing device, such as a server. Alternatively, the acquisition unit 710 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-a-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0181] The multiple computing devices included in the acquisition unit 710 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition unit 710 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition unit 710 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0182] It should be noted that, in other embodiments, the steps implemented by the acquisition unit 710, the segmentation unit 720, the calculation unit 730, and the communication unit 740 can be specified as needed. The acquisition unit 710, the segmentation unit 720, the calculation unit 730, and the communication unit 740 respectively implement different steps in the attention calculation method to realize all the functions of the machine learning system 1000.

[0183] The functions of the acquisition unit 710, the segmentation unit 720, the calculation unit 730, and the communication unit 740 are explained below.

[0184] The acquisition unit 710 is used to acquire the initial sequence and initial vector to be processed, specifically for implementing... Figure 4 S310, S320 and their optional steps in the embodiments.

[0185] The segmentation unit 720 is used to segment the initial vector to obtain multiple first sub-vectors, specifically for implementing... Figure 4 S330 and its optional steps in the embodiments.

[0186] Computation unit 730 is used to perform the first attention calculation based on the initial sequence and initial vector, specifically for implementing... Figure 4 S340 and its optional steps in the embodiments.

[0187] Communication unit 740 is used to send multiple first sub-vectors to the second node and receive multiple second sub-vectors sent by the third node, specifically for implementing... Figure 4 S350A, S350B and their optional steps in the embodiments.

[0188] The computing unit 730 is also used to perform multiple attention calculations based on the initial sequence and multiple second sub-vectors to obtain the attention calculation result of the first node, specifically for implementing... Figure 4 S360 to S390 and their optional steps in the embodiments.

[0189] In one possible implementation, the segmentation unit 720 is used to determine the transmission order of each first sub-vector, the communication unit 740 is used to send multiple first sub-vectors to the second node according to the transmission order, receive multiple second sub-vectors sent by the third node, obtain the reception order of each second sub-vector, and send multiple second sub-vectors to the second node according to the reception order, and the calculation unit 730 is used to perform multiple attention calculations according to the reception order, based on the initial sequence and multiple second sub-vectors.

[0190] In one possible implementation, the segmentation unit 720 is used to segment the initial vector into multiple first sub-vectors according to the length of the initial vector, and determine the sending order of each first sub-vector based on its position in the initial vector. The calculation unit 730 is used to perform multiple attention calculations based on the initial sequence and multiple second sub-vectors, and the calculation result obtained from each attention calculation is merged and updated with the calculation result of the previous attention calculation. Specific steps can be found in [reference needed]. Figure 4 S330 in the embodiment Figure 5 The relevant descriptions of the embodiments will not be repeated here.

[0191] In one possible implementation, the segmentation unit 720 is used to segment the initial vector into multiple first sub-vectors according to the number of attention heads, and to determine the sending order of each first sub-vector based on the index number of the attention head corresponding to each first sub-vector; the calculation unit 730 is used to perform multiple attention calculations based on the initial sequence and multiple second sub-vectors, and to merge and update the calculation results obtained from every m attention calculations with the calculation results of previous attention calculations, where m is the number of first sub-vectors. Specific steps can be found in [reference needed]. Figure 4 S330 in the embodiment Figure 6 The relevant descriptions of the embodiments will not be repeated here.

[0192] In one possible implementation, the segmentation unit 720 is used to segment the initial sequence into multiple subsequences according to the number of attention heads, and to determine the sending order of each subsequence based on the order of the index numbers of the attention heads corresponding to each subsequence. The calculation unit 730 is used to perform multiple attention calculations according to the sending order of each subsequence and the receiving order of each second sub-vector.

[0193] In one possible implementation, the first node includes a processing module and a communication module. The processing module performs the first attention calculation and multiple attention calculations, while the communication module sends vectors to the second node and receives vectors sent by the first node. The processing module and the communication module process tasks in parallel. Optionally, the processing module can be used to implement the functions described in the above-described computing unit, and the communication module can be used to implement the functions described in the above-described communication unit.

[0194] In one possible implementation, the machine learning system 1000 is used to train or infer a large language model, which includes a transformer model, and the initial vectors include keyword K vectors and value V vectors in the attention layer of the large language model.

[0195] In summary, the node provided in this application, when performing attention calculations at each layer of the model, after obtaining the assigned initial sequence Q, initial vector K, and initial vector V, first divides the initial vector K and initial vector V into multiple sub-vectors K and sub-vector V. Then, while performing the first attention calculation based on the initial sequence Q, initial vector K, and initial vector V, it continuously sends sub-vectors to the next node in units of sub-vectors and receives sub-vectors continuously sent by the previous node. In this way, after the first attention calculation is completed, attention calculations can be continuously performed based on the sub-vectors of the previous node that have been received, receiving and calculating simultaneously, until the sub-vectors of all nodes have been calculated, thereby obtaining the calculation result corresponding to that node. Since the sub-vectors required for the next attention calculation have been obtained after the previous attention calculation is completed, the node's processing module will not be idle or waiting, avoiding resource waste, improving attention calculation efficiency, and thus improving the overall training or inference efficiency of the model.

[0196] The attention calculation method, machine learning system, and nodes within the system provided in this application have been described in detail above. The following section will combine... Figure 8 and Figure 9 The computing device provided in this application will be explained.

[0197] Figure 8 This is a schematic diagram of the structure of a computing device provided in this application, such as... Figure 8 As shown, the computing device 800 includes: a bus 802, a processor 804, a memory 806, an accelerator card 807, and a communication interface 808. The processor 804, memory 806, and communication interface 808 communicate with each other via the bus 802. The computing device 800 can be a server. It should be understood that this application does not limit the number of processors, memory, and accelerator cards in the computing device 800. The computing device 800 can be used to deploy the aforementioned components. Figures 2-7 The nodes in the embodiment. Memory 806 is used to store instructions, and processor 804 is used to execute the instructions stored in memory 806. Memory 806 stores program code, and processor 804 can call the program code stored in memory 806 to perform the following operations:

[0198] The process involves obtaining the initial sequence and initial vector to be processed, splitting the initial vector to obtain multiple first sub-vectors, performing the first attention calculation based on the initial sequence and initial vectors, sending the multiple first sub-vectors to the second node, receiving the multiple second sub-vectors sent by the third node, performing multiple attention calculations based on the initial sequence and multiple second sub-vectors, and obtaining the attention calculation result of the first node.

[0199] It should be understood that the memory 806 stores executable program code, and the processor 804 executes the executable program code to achieve the above. Figures 4 to 6 The operational steps of the attention calculation method described in each embodiment are as follows. The program code may include one or more software units, which are... Figure 7 The embodiment includes an acquisition unit, a segmentation unit, a calculation unit, and a communication unit, wherein the acquisition unit is used to implement... Figure 4 In the embodiments, steps S310, S320, and their optional steps, the segmentation unit is used to implement... Figure 4 S330 and its optional steps in the embodiment, the computing unit is used to implement Figure 4 In the embodiments S340, S360-S390 and their optional steps, the communication unit is used to implement... Figure 4 The steps S350A, S350B and their optional steps in the embodiment will not be described in detail here.

[0200] The 802 bus can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The Unified Bus is also known as the Lingqu bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus 802 is represented by only one line, but this does not mean that there is only one bus or one type of bus. The bus 802 may include a path for transmitting information between various components of the computing device 800 (e.g., memory 806, processor 804, communication interface 808). The unified bus may also be called the Lingqu bus.

[0201] The processor 804 may include any one or more computing devices such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, or an accelerator card.

[0202] Memory 806 may include volatile memory, such as random access memory (RAM). Memory 806 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 806 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0203] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 806, or two or more types of storage media can be configured to realize the function of memory 806. This application does not limit this.

[0204] The communication interface 808 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 800 and other devices or communication networks.

[0205] As one possible implementation method, Figures 4 to 6 The method described herein can be implemented either by the processor in the computing device 800 executing program code in the memory, or by the accelerator card 807 in the computing device 800. The accelerator card 807 can be implemented by computing devices such as DPU, NPU, GPU, XPU, SoC, or AI chips; this application does not specifically limit its implementation. The accelerator card 807 includes a processing unit and a power supply circuit. The power supply circuit supplies power to the processing unit, enabling the processing unit to execute the above-described... Figures 4 to 6 The operational steps of the attention calculation method described in each embodiment are as follows.

[0206] As one possible implementation method, Figures 4 to 6The method described above can be implemented either by the processor in the computing device 800 executing program code in memory, or by the accelerator card 807, or by a combination of the processor 804 and the accelerator card 807 in the computing device 800. Figures 4 to 6 The attention calculation method described herein refers to the operational steps performed by the corresponding entity. Specifically, the program code stored in memory 806 includes... Figure 7 The shown unit modules enable the processor 804 to execute the program code to implement part of the attention calculation method, and the power supply circuit in the accelerator card 807 supplies power to the processing unit, enabling the processing unit to execute another part of the operation.

[0207] For example, memory 806 stores program code for acquisition unit, segmentation unit and communication unit, and the processing unit of accelerator card 807 is used to implement the functions of computing unit. The above examples are for illustration only and are not specifically limited in this application. Furthermore, the functional descriptions of the above multiple units can be found in the relevant content above, and will not be repeated here.

[0208] This application also provides a computing device cluster, such as Figure 9 As shown, Figure 9 This is an example diagram of a computing device cluster provided in this application, which may include at least one Figure 8 The computing device of the embodiment is used to deploy the machine learning system 1000 described above.

[0209] As one possible implementation, the same implementation can be stored in the memory 806 of one or more computing devices 800 in a computing device cluster. Figures 4 to 6 The instructions for the operational steps of the attention calculation method described herein. Alternatively, different implementations may be stored separately in the memory 806 of one or more computing devices 800. Figures 4 to 6 The instructions for the operational steps of the described attention calculation method, in other words, a combination of one or more computing devices 800 can jointly execute the instructions for the operational steps required to implement the attention calculation method.

[0210] As one possible implementation, the same implementation can be stored in the accelerator cards 807 of one or more computing devices 800 in a computing device cluster. Figures 4 to 6 The instructions describe the operational steps of the attention calculation method. Alternatively, different implementations may be stored in one or more accelerator cards 807 of the computing device 800. Figures 4 to 6 The instructions for the operational steps of the described attention calculation method, in other words, a combination of one or more computing devices 800 can jointly execute the instructions for the operational steps required to implement the attention calculation method.

[0211] As one possible implementation, the same implementation is stored in the memory 806 of one or more computing devices 800 in the computing device cluster. Figures 4 to 6 The instructions for a portion of the operation steps in the described attention calculation method are stored in the accelerator card 807, along with the instructions for implementing another portion of the operation steps. Alternatively, different implementations may be stored in the memory 806 of one or more computing devices 800. Figures 4 to 6 The instruction for one part of the operation steps of the described data method is stored in the accelerator card 807, which contains different instructions for implementing another part of the operation steps. In other words, a combination of one or more computing devices 800 can jointly execute the instructions for the operation steps required to implement the attention computing method.

[0212] This application embodiment also provides a chip, which includes a processing unit and a power supply circuit. The power supply circuit supplies power to the processing unit, enabling the processing unit to perform... Figures 4 to 6 The operation steps of the first node in the attention calculation method in the embodiment will not be described in detail here for the sake of brevity.

[0213] This application also provides a chip system comprising multiple chips as described above, which is used to deploy the machine learning system described above.

[0214] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute a reasoning method based on a large language model.

[0215] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a reasoning method based on a large language model, or instruct the computing device to execute a reasoning method.

[0216] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. An attention computation method characterized by comprising: The method is applied to a machine learning system comprising a plurality of nodes constituting a ring network communication structure therebetween, and the method comprises: A first node of the plurality of nodes acquires an initial sequence and an initial vector to be processed by an attention layer; The first node splits the initial vector to obtain a plurality of first sub-vectors; The first node performs first attention calculation according to the initial sequence and the initial vector, sends the plurality of first sub-vectors to a second node, and receives a plurality of second sub-vectors sent by a third node; The first node performs multiple attention calculations according to the initial sequence and the plurality of second sub-vectors to obtain an attention calculation result of the first node.

2. The method of claim 1, wherein, The sending of the plurality of first sub-vectors to the second node and the receiving of the plurality of second sub-vectors sent by the third node comprise: The first node determines a sending order of each first sub-vector; The first node sends the plurality of first sub-vectors to the second node and receives the plurality of second sub-vectors sent by the third node according to the sending order, obtains a receiving order of each second sub-vector, and sends the plurality of second sub-vectors to the second node according to the receiving order; The first node performs multiple attention calculations according to the initial sequence and the plurality of second sub-vectors comprises: The first node performs multiple attention calculations according to the initial sequence and the plurality of second sub-vectors according to the receiving order.

3. The method of claim 2, wherein, The first node determining a sending order of each first sub-vector comprises: The first node splits the initial vector into a plurality of first sub-vectors according to the length of the initial vector, and determines the sending order of each first sub-vector based on the position of each first sub-vector in the initial vector; The first node performing multiple attention calculations according to the initial sequence and the plurality of second sub-vectors comprises: The first node performs multiple attention calculations according to the initial sequence and the plurality of second sub-vectors, and the calculation result obtained each time is combined and updated with the calculation result of the previous attention calculation.

4. The method of claim 2, wherein, The first node determining a sending order of each first sub-vector comprises: The first node splits the initial vector into a plurality of first sub-vectors according to the number of each attention head, and determines the sending order of each first sub-vector based on the order of the index number of the corresponding attention head of each first sub-vector; The first node performing multiple attention calculations according to the initial sequence and the plurality of second sub-vectors comprises: The first node performs multiple attention calculations according to the initial sequence and the plurality of second sub-vectors, and the calculation result obtained every m attention calculation is combined and updated with the calculation result of the previous attention calculation, wherein m is the number of the first sub-vectors.

5. The method of claim 4, wherein, The method further comprises: The first node splits the initial sequence into a plurality of sub-sequences according to the number of attention heads, and determines the sending order of each sub-sequence based on the order of the index number of the corresponding attention head of each sub-sequence; The first node performing multiple attention calculations according to the initial sequence and the plurality of second sub-vectors to obtain an attention calculation result of the first node comprises: The first node performs multiple attention calculations according to the sending order of each sub-sequence and the receiving order of each second sub-vector.

6. The method according to any one of claims 1 to 5, characterized in that, The first node includes a processing module and a communication module, the processing module is configured to perform the first attention calculation and the multiple attention calculations, and the communication module is configured to send vectors to the second node and receive vectors sent by the first node, and the processing module and the communication module process tasks in parallel.

7. The method according to any one of claims 1 to 6, characterized in that, The machine learning system is used to implement training or inference of a large language model, the large language model includes a transformer model, the initial vector includes a key K vector and a value V vector in an attention layer of the large language model, and the initial sequence includes an input sequence of the attention layer.

8. A computing node, characterized in that, The computing node is deployed in a machine learning system, the machine learning system includes a plurality of computing nodes, a network structure between the plurality of computing nodes is a ring network structure, and the computing node includes: An acquisition unit is configured to acquire an initial sequence and an initial vector to be processed by an attention layer; A splitting unit is configured to split the initial vector to obtain a plurality of first sub-vectors; A calculation unit is configured to perform a first attention calculation according to the initial sequence and the initial vector; A communication module is configured to send the plurality of first sub-vectors to a second node and receive a plurality of second sub-vectors sent by a third node; The calculation unit is further configured to perform multiple attention calculations according to the initial sequence and the plurality of second sub-vectors to obtain an attention calculation result of the first node.

9. The computing node of claim 8, wherein The splitting unit is configured to determine a sending order of each first sub-vector; The communication module is configured to send the plurality of first sub-vectors to the second node in the sending order, receive the plurality of second sub-vectors sent by the third node, obtain a receiving order of each second sub-vector, and send the plurality of second sub-vectors to the second node in the receiving order; The calculation unit is configured to perform the multiple attention calculations according to the initial sequence and the plurality of second sub-vectors in the receiving order.

10. The computing node of claim 9, wherein The splitting unit is configured to split the initial vector into the plurality of first sub-vectors according to a length of the initial vector, and determine the sending order of each first sub-vector based on a position of each first sub-vector in the initial vector; The calculation unit is configured to perform the multiple attention calculations according to the initial sequence and the plurality of second sub-vectors, and combine and update a calculation result of each attention calculation with a calculation result of a previous attention calculation.

11. The computing node of claim 9, wherein The splitting unit is configured to split the initial vector into the plurality of first sub-vectors according to a number of attention heads, and determine the sending order of each first sub-vector based on an order of index numbers of the attention heads corresponding to each first sub-vector. The computing unit is configured to perform multiple attention computations according to the initial sequence and the plurality of second sub-vectors, and to update a result of each m-th attention computation based on a result of a previous attention computation, where m is the number of the first sub-vectors.

12. The computing node of claim 11, wherein, The splitting unit is configured to split the initial sequence into a plurality of sub-sequences according to a number of attention heads, and to determine a sending order of each sub-sequence based on an order of an index number of an attention head corresponding to the sub-sequence. The computing unit is configured to perform the multiple attention computations according to the sending order of each sub-sequence and a receiving order of each second sub-vector.

13. The compute node of any of claims 8 to 12, wherein, The first node comprises a processing module and a communication module, the processing module is configured to perform the first attention computation and the multiple attention computations, and the communication module is configured to send a vector to the second node and receive a vector sent by the first node, and the processing module and the communication module process tasks in parallel.

14. The compute node of any of claims 8 to 13, wherein, The machine learning system is configured to implement training or inference of a large language model, the large language model comprises a transformer model, the initial vector comprises a key K vector and a value V vector in an attention layer of the large language model, and the initial sequence comprises an input sequence of the attention layer.

15. A computing device, comprising: The computing device comprises a processor and a memory. The processor is configured to execute instructions stored in the memory to cause the computing device to perform the operation steps of the method of any one of claims 1 to 7.

16. A chip, characterized by The chip comprises a power supply unit and a processing module, the power supply unit is configured to supply power to the processing module, so that the processing module implements the operation steps of the method of any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that, The computer program instructions, when executed by a computing device, cause the computing device to perform the operation steps of the method of any one of claims 1 to 7.

18. A computer program product comprising instructions, characterized in that, The instructions, when executed by a computing device, cause the computing device to perform the operation steps of the method of any one of claims 1 to 7.

Images