Self-attention calculation method and device and related product

Abstract

The invention provides a self-attention calculation method and device and a related product, and the method comprises the steps: carrying out the rearrangement of a plurality of lexical elements in an input sequence, obtaining a first rearrangement sequence, an attention mask corresponding to the first rearrangement sequence, and a lattice mask, and enabling the attention mask to comprise a plurality of sub-attention masks, the value at any position in the lattice mask is used for indicating whether there is a dependency relationship between lexical elements in the sub attention mask corresponding to the position, and the number of values, which are first values, in the lattice mask corresponding to the first rearrangement sequence is smaller than or equal to the number of values, which are first values, in the lattice mask corresponding to the input sequence; generating task scheduling information according to the position of which the numerical value is the first value in the latticed mask; and performing self-attention calculation on the first rearrangement sequence according to the task scheduling information. By the adoption of the method and device, the problem that loads among the computing devices are unbalanced when the multiple computing devices conduct self-attention computing in parallel can be solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a self-attention computing method, device and related products. Background Technology

[0002] Self-attention is a technique that captures the relationships between different positions in an input sequence, helping large models more accurately capture key information. This has led to its widespread application in training large models based on transformer architectures. However, as the length of the input sequence increases, the computational complexity also rises. Therefore, for long sequences, sequence parallelism is typically used to improve computational efficiency. In this approach, the long sequence is divided into multiple shorter subsequences and processed in parallel on different computational nodes to optimize resource utilization.

[0003] In ring attention, computing nodes are organized into a ring topology, with each node connected to its neighboring nodes, and each node communicates in a fixed order. When performing self-attention computation, each node determines whether to perform attention computation in each computation round based on the attention mask corresponding to the input sequence. However, this fixed scheduling method of ring attention may lead to an imbalance in the computational load among different computing nodes in each computation round. Summary of the Invention

[0004] This application provides a self-attention calculation method, apparatus, and related products, which can solve the problem of uneven load among computing devices when self-attention calculation is performed in parallel on multiple computing devices.

[0005] Firstly, this application provides a self-attention calculation method, which includes:

[0006] Multiple words in the input sequence are rearranged to obtain a first rearranged sequence, an attention mask and a lattice mask corresponding to the first rearranged sequence. The attention mask includes multiple sub-attention masks. The value at any position in the lattice mask is used to indicate whether there is a dependency relationship between words in the corresponding sub-attention mask. The number of values ​​in the lattice mask corresponding to the first rearranged sequence that are the first value is less than or equal to the number of values ​​in the lattice mask corresponding to the input sequence that are the first value. The value in the lattice mask that is the first value is used to indicate that there is a computational task at the position corresponding to that value.

[0007] Based on the position of the first value in the lattice mask, generate task scheduling information;

[0008] Based on the task scheduling information, self-attention calculation is performed on the first rearranged sequence.

[0009] In the above scheme, the value at any position in the lattice mask is used to indicate whether there is a dependency relationship between the terms in the sub-attention mask corresponding to that position. When the value in the lattice mask is the first value, it indicates that there is a computational task at the position corresponding to that value. The number of times the value in the lattice mask corresponding to the first rearranged sequence is the first value is less than or equal to the number of times the value in the lattice mask corresponding to the input sequence is the first value. Therefore, by rearranging multiple terms in the input sequence, computational tasks can be reduced and computational efficiency can be improved. In addition, the task scheduling information in this embodiment is determined based on the position of the first value in the lattice mask. By modifying the task scheduling information, not only can invalid computational tasks be avoided on each computing device, reducing the waste of computing resources, but also the number of computational tasks on each computing device can be made as balanced as possible throughout the self-attention computation process, solving the problem of uneven load on different computing devices.

[0010] Based on the first aspect, in possible implementations, the specific process of rearranging multiple words in the input sequence to obtain the first rearranged sequence, the attention mask corresponding to the first rearranged sequence, and the lattice mask is as follows:

[0011] Clustering is performed on the attention mask corresponding to the input sequence to obtain multiple clustering results. Each clustering result records the information of the words in the input sequence that are classified into the same category.

[0012] Multiple words in the input sequence are rearranged according to the information indicated in each clustering result to obtain multiple rearranged sequences. Each clustering result has a corresponding rearranged sequence. The relative order of multiple words in the same category in the clustering result is the same as the relative order of the corresponding words in the rearranged sequence of the clustering result.

[0013] Generate attention masks corresponding to multiple rearranged sequences, wherein the dependencies between each word in the attention mask corresponding to the rearranged sequence are the same as the dependencies between each word in the attention mask corresponding to the input sequence;

[0014] Divide the rows and columns of each attention mask in the attention mask corresponding to multiple rearranged sequences into a specified number to obtain the lattice mask corresponding to multiple rearranged sequences;

[0015] The first lattice mask is determined from the lattice masks corresponding to multiple rearranged sequences. The rearranged sequence corresponding to the first lattice mask is the first rearranged sequence. The attention mask corresponding to the first lattice mask is the attention mask corresponding to the first rearranged sequence. The first lattice mask is the lattice mask corresponding to the first rearranged sequence.

[0016] In the above scheme, by first clustering the attention masks corresponding to the input sequences, terms with similar features or semantics can be grouped together. Then, based on the information indicated in each clustering result, the sequences are rearranged to obtain multiple rearranged sequences. In each rearranged sequence, terms with similar features are more clustered, making the computational task more focused. Introducing a lattice mask when obtaining the first rearranged sequence facilitates the selection of the better-performing rearranged sequence from multiple sequences, which in turn helps optimize the generation of subsequent task scheduling information.

[0017] By clustering the attention masks corresponding to the input sequence, terms with similar features or semantics are grouped together. Multiple terms in the input sequence are rearranged according to the information indicated in each clustering result, resulting in multiple rearranged sequences. This makes the first value more clustered in the rearranged sequences, thereby reducing the total number of computational tasks. Determining the first lattice mask from the lattice masks corresponding to the multiple rearranged sequences allows for the selection of the best-performing rearranged sequence, facilitating the generation of subsequent task scheduling information.

[0018] Based on the first aspect, in possible implementations, when there is a dependency between the terms in the sub-attention mask, the value of the sub-attention mask at the corresponding position on the corresponding lattice mask is the first value.

[0019] Based on the first aspect, in possible implementations, the process of determining the first lattice mask from the lattice masks corresponding to multiple rearranged sequences is as follows:

[0020] Calculate the first result for each of the multiple lattice masks, where the first result is the number of times the value in each lattice mask is the first value;

[0021] Calculate a second result for each of the multiple lattice masks, the second result being the variance of the number of computational tasks that each computation node can choose in each lattice mask;

[0022] Calculate the evaluation result for each lattice mask among multiple lattice masks. The evaluation result is the weighted average of the first result and the second result.

[0023] The grid mask corresponding to the smallest value among multiple evaluation results is used as the first grid mask.

[0024] It is understandable that in the above scheme, by comprehensively considering the weighted average of the first and second results, the grid mask with the smallest evaluation result is selected as the first grid mask, which can achieve a more reasonable task allocation, avoid too many or too few tasks, and improve the overall efficiency of the system.

[0025] In this scheme, the second result is the variance of the number of computational tasks that each computing node can choose in each lattice mask. A larger variance indicates an uneven distribution of computational tasks available to each computing node, while a smaller variance indicates a more balanced distribution. Using the weighted average of the first and second results as the evaluation result comprehensively considers both the number of computational tasks and the balance of task allocation. Selecting the lattice mask with the smallest evaluation result as the first lattice mask ensures a more balanced distribution of computational tasks across each computing node during self-attention computation of the first rearranged sequence corresponding to the first lattice mask, thus avoiding resource waste.

[0026] Based on the first aspect, in possible implementations, each computing device can select a computing task that is a query matrix and a key-value matrix, where at least one of them belongs to the current computing node, and the key-value matrix includes a key matrix and a value matrix.

[0027] It's understandable that when a compute node executes a computational task, if the query matrix, key matrix, and value matrix are all obtained from other compute nodes, the communication load will increase. In the above scheme, by limiting the computational tasks that each compute node can choose, it ensures that each compute node's computational task includes at least one local query matrix or a local key and value matrix. This reduces cross-compute node communication needs, lowers communication overhead, and shortens communication latency. Furthermore, it maximizes resource utilization and avoids wasted resources.

[0028] Based on the first aspect, in possible implementations, when each computing device selects a computing task in each computing round in the task scheduling information, it needs to satisfy both a first condition and a second condition. The first condition is that the computing task selected by each computing device is a computing task selected by each computing node that belongs to the current computing node, where at least one of the query matrix and the key-value matrix exists. The key-value matrix includes a key matrix and a value matrix. The second condition is that the communication volume of each computing device in each computing round is less than or equal to the communication unit threshold.

[0029] It is understandable that when each computing device selects a computing task in each computing round, and each computing node selects a computing task in which at least one of the query matrix and key-value matrix belongs to the current computing node, the communication requirements across computing nodes can be reduced. Since in this embodiment, each computing device performs computation and communication simultaneously in each computing round, if the communication volume is too large, a situation may arise where a computing node completes its computation task but its communication task is not yet complete, leading to a decrease in computational efficiency. Furthermore, by controlling the communication volume, excessive data transmission and communication burden can be avoided, resource utilization can be optimized, and the overall performance of the model can be improved.

[0030] Based on the first aspect, among possible implementations, the method also includes:

[0031] When self-attention computation is non-distributed computation, multiple words in the input sequence are rearranged to obtain the second rearranged sequence and the attention mask corresponding to the second rearranged sequence.

[0032] Determine the position information of one or more regions in the attention mask corresponding to the second rearrangement sequence. The region is a region in the attention mask corresponding to the second rearrangement sequence that satisfies a first length and a first width, and all values ​​in the region are the second values.

[0033] The location information of one or more regions and the attention mask corresponding to the second rearrangement sequence are input into the self-attention operator to perform self-attention calculation on the second rearrangement sequence. The location information of one or more regions is used to indicate that the corresponding position of the region in the second rearrangement sequence is not subject to self-attention calculation.

[0034] In the above scheme, firstly, multiple tokens in the input sequence are rearranged to obtain a second rearranged sequence and an attention mask corresponding to the second rearranged sequence. Then, the position information of one or more regions is determined from the attention mask corresponding to the second rearranged sequence, wherein the midpoint of the region is all the second value. The position information of the region can indicate that the corresponding position of the region in the second rearranged sequence should not be subject to self-attention calculation. This can reduce unnecessary calculations, reduce the waste of computing resources, and improve the performance of the model.

[0035] Based on the first aspect, in possible implementations, the specific process of rearranging multiple words in the input sequence to obtain the second rearranged sequence and the attention mask corresponding to the second rearranged sequence is as follows:

[0036] Cluster the attention mask corresponding to the input sequence to obtain the first clustering result. The first clustering result records the information of the words in the input sequence that are classified into the same category.

[0037] The multiple words in the input sequence are rearranged according to the information indicated in the first clustering result to obtain the second rearranged sequence. The relative order of multiple words in the same category in the first clustering result is the same as the relative order of the corresponding words in the second rearranged sequence.

[0038] Generate an attention mask corresponding to the second rearranged sequence. The dependencies between the terms in the attention mask corresponding to the second rearranged sequence are the same as those between the terms in the attention mask corresponding to the input sequence.

[0039] In the above scheme, the attention mask corresponding to the input sequence is first clustered to obtain a first clustering result, which records the information of words in the input sequence that are classified into the same category. This step can group words with similar features or semantics together. Next, based on the indication information in the first clustering result, multiple words in the input sequence are rearranged to obtain a second rearranged sequence. Finally, a corresponding attention mask is generated based on the second rearranged sequence, ensuring that the dependencies between words in the attention mask corresponding to the second rearranged sequence are consistent with the dependencies in the attention mask corresponding to the input sequence. This step ensures that the calculation result of the second rearranged sequence is the same as the calculation result of the input sequence.

[0040] Based on the first aspect, in possible implementations, the specific process of clustering the attention mask corresponding to the input sequence to obtain the first clustering result includes:

[0041] Cluster the attention mask corresponding to the input sequence to obtain multiple clustering results and the silhouette coefficients corresponding to the multiple clustering results;

[0042] The cluster result with the largest silhouette coefficient among multiple cluster results is taken as the first cluster result.

[0043] It should be understood that the silhouette coefficient is a metric used to evaluate the quality of clustering. It considers the compactness within clusters and the separation between clusters; a value closer to 1 indicates a better clustering effect. In the above scheme, the cluster with the largest silhouette coefficient among multiple clustering results is taken as the first clustering result. This allows for a greater clustering of semantically similar and structurally relevant terms when rearranging the input sequence.

[0044] Secondly, this application provides a self-attention computing device, comprising:

[0045] The acquisition module is used to rearrange multiple words in the input sequence to obtain a first rearranged sequence, an attention mask and a lattice mask corresponding to the first rearranged sequence. The attention mask includes multiple sub-attention masks. The value at any position in the lattice mask is used to indicate whether there is a dependency relationship between words in the corresponding sub-attention mask. The number of values ​​with the first value in the lattice mask corresponding to the first rearranged sequence is less than or equal to the number of values ​​with the first value in the lattice mask corresponding to the input sequence. The value with the first value in the lattice mask is used to indicate that there is a computation task at the position corresponding to that value.

[0046] The generation module is used to generate task scheduling information based on the position of the first value in the grid mask;

[0047] The calculation module is used to perform self-attention calculation on the first rearranged sequence based on task scheduling information.

[0048] Based on the second aspect, in possible implementations, the acquisition module is used for:

[0049] Clustering is performed on the attention mask corresponding to the input sequence to obtain multiple clustering results. Each clustering result records the information of the words in the input sequence that are classified into the same category.

[0050] Multiple words in the input sequence are rearranged according to the information indicated in each clustering result to obtain multiple rearranged sequences. Each clustering result has a corresponding rearranged sequence. The relative order of multiple words in the same category in the clustering result is the same as the relative order of the corresponding words in the rearranged sequence of the clustering result.

[0051] Generate attention masks corresponding to multiple rearranged sequences, wherein the dependencies between each word in the attention mask corresponding to the rearranged sequence are the same as the dependencies between each word in the attention mask corresponding to the input sequence;

[0052] Divide the rows and columns of each attention mask in the attention mask corresponding to multiple rearranged sequences into a specified number to obtain the lattice mask corresponding to multiple rearranged sequences;

[0053] The first lattice mask is determined from the lattice masks corresponding to multiple rearranged sequences. The rearranged sequence corresponding to the first lattice mask is the first rearranged sequence. The attention mask corresponding to the first lattice mask is the attention mask corresponding to the first rearranged sequence. The first lattice mask is the lattice mask corresponding to the first rearranged sequence.

[0054] Based on the second aspect, in possible implementations, when there is a dependency between the terms in the sub-attention mask, the value of the sub-attention mask at the corresponding position on the corresponding lattice mask is the first value.

[0055] Based on the second aspect, in possible implementations, the acquisition module is used for:

[0056] Calculate the first result for each of the multiple lattice masks, where the first result is the number of times the value in each lattice mask is the first value;

[0057] Calculate a second result for each of the multiple lattice masks, the second result being the variance of the number of computational tasks that each computation node can choose in each lattice mask;

[0058] Calculate the evaluation result for each lattice mask among multiple lattice masks. The evaluation result is the weighted average of the first result and the second result.

[0059] The grid mask corresponding to the smallest value among multiple evaluation results is used as the first grid mask.

[0060] Based on the second aspect, in a possible implementation, each computing node can choose a computing task that belongs to the current computing node, either a query matrix or a key-value matrix, wherein the key-value matrix includes a key matrix and a value matrix.

[0061] Based on the second aspect, in possible implementations, each computing node in the task scheduling information needs to satisfy both a first condition and a second condition when selecting a computing task in each computing round. The first condition is that the computing task selected by each computing node is a computing task belonging to the current computing node, where at least one of the query matrix and the key-value matrix exists. The key-value matrix includes a key matrix and a value matrix. The second condition is that the communication volume of each computing node in each computing round is less than or equal to the communication unit threshold.

[0062] Based on the second aspect, in a possible implementation, the acquisition module is used to rearrange multiple words in the input sequence to obtain the second rearranged sequence and the attention mask corresponding to the second rearranged sequence when the self-attention computation is non-distributed computation.

[0063] The module also includes a determination module, used to determine the position information of one or more regions in the attention mask corresponding to the second rearrangement sequence. The region is a region in the attention mask corresponding to the second rearrangement sequence that satisfies a first length and a first width, and all values ​​in the region are second values.

[0064] The calculation module is used to input the location information of one or more regions and the attention mask corresponding to the second rearrangement sequence into the self-attention operator to perform self-attention calculation on the second rearrangement sequence. The location information of one or more regions is used to indicate that the corresponding position of the region in the second rearrangement sequence does not perform self-attention calculation.

[0065] Based on the second aspect, in possible implementations, the acquisition module is used for:

[0066] Cluster the attention mask corresponding to the input sequence to obtain the first clustering result. The first clustering result records the information of the words in the input sequence that are classified into the same category.

[0067] The multiple words in the input sequence are rearranged according to the information indicated in the first clustering result to obtain the second rearranged sequence. The relative order of multiple words in the same category in the first clustering result is the same as the relative order of the corresponding words in the second rearranged sequence.

[0068] Generate an attention mask corresponding to the second rearranged sequence. The dependencies between the terms in the attention mask corresponding to the second rearranged sequence are the same as those between the terms in the attention mask corresponding to the input sequence.

[0069] Based on the second aspect, in a possible implementation, the acquisition module is used to cluster the attention mask corresponding to the input sequence to obtain multiple clustering results and contour coefficients corresponding to the multiple clustering results; and to take the clustering result corresponding to the contour coefficient with the largest value among the multiple clustering results as the first clustering result.

[0070] Thirdly, this application provides a computing device including a memory and a processor, the processor being configured to execute instructions stored in the memory to enable the computing device to implement the method described in the first aspect and any possible implementation thereof.

[0071] Fourthly, this application provides a computing device cluster including at least one computing device, the at least one computing device including a memory and a processor, the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster implements the method described in the first aspect and any possible implementation of the first aspect.

[0072] Fifthly, this application provides a computer storage medium including program instructions that, when executed on a cluster of computing devices, enable the cluster of computing devices to perform the method described in the first aspect and any possible implementation thereof.

[0073] Sixthly, this application provides a computer program product including program instructions that, when executed on a computing device cluster, cause the computing device cluster to perform the method described in the first aspect or any possible implementation thereof. The computer program product may be a software installation package. When the method provided by any possible design of the first aspect is required, the computer program product may be downloaded and executed on the computing device cluster to implement the method described in the first aspect and any possible implementation thereof. Attached Figure Description

[0074] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0075] Figure 1 A schematic diagram of the structure for scheduling computational tasks using a ring attention mechanism provided in this application;

[0076] Figure 2A flowchart illustrating a self-attention calculation method provided in this application;

[0077] Figure 3 A schematic diagram illustrating the acquisition of the corresponding lattice mask based on the attention mask provided in this application;

[0078] Figure 4 A schematic diagram illustrating the determination of selectable computational tasks for computational nodes based on a lattice mask, as provided in this application;

[0079] Figure 5 A schematic diagram showing the effect comparison between the input sequence provided in this application and the first rearranged sequence;

[0080] Figure 6 A flowchart illustrating another self-attention calculation method provided in this application;

[0081] Figure 7 A comparative diagram showing the attention mask corresponding to the input sequence and the attention mask corresponding to the second rearranged sequence provided in this application;

[0082] Figure 8 This application provides a schematic diagram of the structure of a self-attention computing device;

[0083] Figure 9 A schematic diagram of the structure of a computing device provided in this application;

[0084] Figure 10 This application provides a schematic diagram of the structure of a computing device cluster;

[0085] Figure 11 This is a schematic diagram of another computing device cluster provided in this application. Detailed Implementation

[0086] There are two main methods for self-attention computation: one is distributed computing (sequence parallelism), and the other is non-distributed computing. In distributed computing, the self-attention computation is performed in parallel across multiple computing devices. That is, the input sequence is divided into multiple subsequences, each assigned to a different computing device for processing. The results from each device are then aggregated to obtain the final self-attention result for the input sequence. In non-distributed computing, the self-attention computation is performed on a single computing device. The formula for self-attention computation is given in formula (1).

[0087]

[0088] Where X is the input sequence, Q is the query matrix corresponding to the input sequence, used to indicate the current focus or information requirement of the model, K is the key matrix corresponding to the input sequence, containing the identification information of each position in the input sequence, V is the value matrix corresponding to the input sequence, used to store the actual value or information content corresponding to the K matrix, the three matrices Q, K, and V are generated by X through three different linear transformations, and d is the dimension of the model.

[0089] In distributed computing, a circular attention mechanism is typically used to perform self-attention computation on the input sequence. This mechanism usually requires multiple computation rounds to complete the self-attention computation on the input sequence. The number of computation rounds equals the number of computing nodes. In each computation round, computation and communication are masked; that is, in one round, a computing node performs computation while simultaneously sending and receiving key and value matrices to meet the computational needs of the next round, improving computational efficiency. After receiving the key and value matrices, each computing node determines whether there is a dependency between its own subsequence and the subsequences of the corresponding computing nodes. If a dependency exists, the computing node inputs its query matrix along with the received key and value matrices into the block attention operator for computation and also sends the received key and value matrices to the next computing node according to a fixed scheduling method. If no dependency exists, the computing node simply sends the key and value matrices to the next computing node according to a fixed scheduling method. It should be noted that, by default, in the 0th computation round, the key matrix and value matrix received by each computation node are the key matrix and value matrix stored on that computation node.

[0090] The following example, using a single computation cycle, illustrates how the circular attention mechanism schedules computational tasks. (See also...) Figure 1 , Figure 1 This is a schematic diagram of the structure for scheduling computational tasks using a ring attention mechanism, as provided in this application. Figure 1 In this architecture, four computing nodes are organized into a ring topology, with each node connected to its neighboring nodes. Each computing node stores a subsequence along with its corresponding query matrix, key matrix, and value matrix. Specifically, Figure 1Computation node 0 stores subsequence 0, query matrix 0, key matrix 0, and value matrix 0; computation node 1 stores subsequence 1, query matrix 1, key matrix 1, and value matrix 1; computation node 2 stores subsequence 2, query matrix 2, key matrix 2, and value matrix 2; and computation node 3 stores subsequence 3, query matrix 3, key matrix 3, and value matrix 3. During task scheduling, each computation node sends the received key and value matrices to the next node to meet its computational needs in the next computation round. Specifically, in a computation round, computation node 0 sends the key and value matrices received on computation node 0 to computation node 1, computation node 1 sends the key and value matrices received on computation node 1 to computation node 2, computation node 2 sends the key and value matrices received on computation node 2 to computation node 3, and computation node 3 sends the key and value matrices received on computation node 3 to computation node 0.

[0091] However, due to the different dependencies between subsequences, in some computation rounds, some computation nodes may have computation tasks (there is a dependency between the subsequence on the computation node and the subsequence on the computation node corresponding to the received key matrix and value matrix), while some computation nodes may not have computation tasks (there is no dependency between the subsequence on the computation node and the subsequence on the computation node corresponding to the received key matrix and value matrix). This leads to the problem of unbalanced load among the computation nodes.

[0092] In distributed computing, self-attention operators, such as flash attention, perform self-attention computation on the input sequence based on the query matrix, key matrix, value matrix, and the corresponding attention mask. During computation, the flash attention operator calculates the self-attention between each word in the input sequence and other words, and only after this calculation is complete does it decide whether to discard the result based on the value in the attention mask. However, performing self-attention computation when there is no dependency between two words leads to a waste of computational resources.

[0093] The following sections introduce two self-attention computation methods to address the issues of uneven load distribution among computing nodes in distributed computing and wasted computing resources in non-distributed computing.

[0094] See Figure 2 , Figure 2 This application provides a flowchart illustrating a self-attention computation method, which is applied to distributed computing self-attention. The method includes:

[0095] S101: Cluster the attention mask corresponding to the input sequence multiple times according to the number of categories to obtain multiple clustering results.

[0096] The input sequence comprises multiple lexical units, where a lexical unit refers to the basic unit after segmentation of the input sequence. It may be a word, character, subword, etc., and its specific form depends on the segmentation method used and the task requirements. For example, the input sequence "I love to explore new technologies and learn new things" can be divided into ten lexical units: "I", "love", "to", "explore", "new", "technologies", "and", "learn", "new", and "things". These ten lexical units will be referred to as lexical unit 1 to lexical unit 10 below. It should be noted that in self-attention mechanisms, the input sequence and lexical units are usually represented by vectors. This application does not specifically limit the representation of the input sequence and lexical units.

[0097] An attention mask is a structure used in self-attention mechanisms, typically represented as an n×n binary matrix, where n is the number of terms in the input sequence. The attention mask indicates whether dependencies exist between terms in the input sequence. When performing self-attention computation on the input sequence, the model can use the attention mask to determine which positions to mask (i.e., positions that don't need consideration, where the two terms at that position have no dependency) and which positions to retain (i.e., positions that need consideration, where the two terms at that position have a dependency). A dependency means that one term is semantically or syntactically associated with another term; this relationship may be contextual dependence, syntactic structure, or other semantic connections.

[0098] The numerical values ​​in the attention mask are typically represented by a first value or a second value. The first value indicates a dependency between two words at that position, while the second value indicates no dependency between them. Both the first and second values ​​can be represented by 0 or 1. For example, when the first value is 0, the second value is 1; when the first value is 1, the second value is 0. For ease of description, this application uses a first value of 0 and a second value of 1 as an example to illustrate the method provided. This application does not specifically limit the exact numerical values ​​of the first and second values.

[0099] By performing multiple clustering operations on the attention mask corresponding to the input sequence according to the number of categories, multiple clustering results can be obtained. Each clustering result records the information of the words in the input sequence that are classified into the same category. The information of the words can be the word identifier (id), or other features or attributes that can identify the words; this application does not impose any specific limitations.

[0100] For example, clustering result 1 can be {Category 1: lexical 1, lexical 2; Category 2: lexical 4, lexical 6, lexical 8, lexical 9 and lexical 10; Category 3: lexical 3, lexical 5, lexical 7}.

[0101] It should be understood that the above clustering results are merely a specific example. In practical applications, they can be presented in other forms, and this application does not impose any specific limitations.

[0102] When clustering the attention mask corresponding to the input sequence according to the number of categories, different clustering methods can be selected, such as K-means clustering, hierarchical clustering, density clustering, etc. This application does not specifically limit the clustering method.

[0103] In one possible implementation, the attention mask (matrix size S*S, where S is the number of words in the input sequence) corresponding to the input sequence is clustered according to the number of categories to obtain multiple clustering results as follows:

[0104] (1) Extract features from the attention mask corresponding to the input sequence to obtain a dataset A (number of samples S, feature length S) that highlights the clustering relationship.

[0105] (2) Reduce the dimensionality of dataset A to obtain a low-dimensional dataset B with S samples and K feature dimensions.

[0106] After centering the sample set of dataset A, the variance matrix is ​​calculated. Considering each row of the S*S dataset A as a sample and each column as a feature dimension, the principal component analysis (PCA) algorithm is used to reduce the dimensionality of the variance matrix, resulting in a low-dimensional dataset of S*K, where K is the feature dimension and K is less than S.

[0107] It should be noted that, in addition to PCA, other algorithms can be used to reduce the dimensionality of the attention mask corresponding to the input sequence, such as linear discriminant analysis (LDA), etc. This application does not make specific limitations on this.

[0108] (3) Perform multiple clustering operations on the low-dimensional dataset B to obtain K-1 clustering results.

[0109] Using an unsupervised learning algorithm, exemplified by K-Means, we cluster a low-dimensional dataset B of size S*K. During the clustering process, a pre-specified number of clusters, c, is required. Then, the algorithm iterates through clusters c from 2 to K, performing the clustering process on the low-dimensional dataset B according to the specified c, resulting in K-1 clustering results.

[0110] It should be understood that the above method of clustering the attention mask corresponding to the input sequence multiple times according to the number of categories to obtain multiple clustering results is only a specific example. In practical applications, it can also be implemented in other ways, and this application does not make any specific limitations.

[0111] S102: Rearrange multiple tokens in the input sequence according to the information indicated in each clustering result to obtain multiple rearranged sequences.

[0112] Although each clustering result records information about the words in the input sequence that are grouped into the same category, the resulting rearranged sequences will differ when multiple words in the input sequence are rearranged according to the order between different categories and / or the order of words within different categories. The following example, using clustering result 1 as {Category 1: word 1, word 2; Category 2: word 4, word 6, word 8, word 9, and word 10; Category 3: word 3, word 5, word 7}, illustrates several possible scenarios and their generated rearranged sequences:

[0113] Case 1: Following the rule of preserving the order of categories in the clustering result and the order of words within the same category, rearrange multiple words in the input sequence. The order of words in the rearranged sequence corresponding to clustering result 1 is: word 1, word 2, word 4, word 6, word 8, word 9, word 10, word 3, word 5, and word 7.

[0114] Case 2: Rearrange multiple words in the input sequence according to the rule of maintaining the order of the clusters in the clustering results but not the order of words within the same cluster. The possible order of words in the rearranged sequence corresponding to clustering result 1 is: word 2, word 1, word 4, word 6, word 8, word 9, word 10, word 3, word 5, and word 7; the possible order of words in the rearranged sequence corresponding to clustering result 1 is: word 1, word 2, word 6, word 4, word 9, word 8, word 10, word 3, word 5, and word 7.

[0115] Case 3: Rearrange multiple words in the input sequence according to the rule of not maintaining the order of the clusters in the clustering results but maintaining the order of words within the same cluster. The possible order of words in the rearranged sequence corresponding to clustering result 1 is: word 4, word 6, word 8, word 9, word 10, word 1, word 2, word 3, word 5, and word 7; the possible order of words in the rearranged sequence corresponding to clustering result 1 is: word 3, word 5, word 7, word 4, word 6, word 8, word 9, word 10, word 1, and word 2.

[0116] Case 4: Following the rule of not maintaining the order of categories in the clustering results and not maintaining the order of words within the same category, rearrange multiple words in the input sequence. The possible order of words in the rearranged sequence corresponding to clustering result 1 is: word 1, word 4, word 2, word 6, word 8, word 9, word 10, word 3, word 5, and word 7. Alternatively, the possible order of words in the rearranged sequence corresponding to clustering result 1 is: word 4, word 8, word 6, word 2, word 1, word 9, word 10, word 3, word 5, and word 7.

[0117] It should be understood that the above only introduces a few common situations. In practical applications, there may be many more situations, which will not be listed here.

[0118] The following describes the process provided in this application for rearranging multiple tokens in an input sequence according to information indicated in a clustering result to obtain a rearranged sequence:

[0119] (1) First, sort the clusters in the clustering results in descending order according to the number of words contained in each cluster;

[0120] (2) Then, according to the sorted category order, obtain the ID of each word element in each category.

[0121] (3) Finally, based on the obtained lexical IDs, multiple lexical units in the input sequence are rearranged. The relative order of multiple lexical units within the same category is the same as the relative order of the corresponding lexical units in the rearranged sequence corresponding to the clustering result.

[0122] For example, taking clustering result 1 as {Category 1: word element 1, word element 2; Category 2: word element 4, word element 6, word element 8, word element 9, and word element 10; Category 3: word element 3, word element 5, and word element 7}, the multiple words in the input sequence are rearranged according to the above process. First, the categories in the clustering result are sorted in descending order according to the number of words contained in each category. The order of the categories in clustering result 1 is: category 2, category 3, and category 1. Then, according to the sorted category order, the word IDs of each category are obtained sequentially, i.e., word element 4, word element 6, word element 8, word element 9, word element 10, word element 3, word element 5, word element 7, word element 1, and word element 2. Finally, based on the obtained lexical IDs, multiple lexical elements in the input sequence are rearranged to obtain a new sequence, namely the rearranged sequence corresponding to cluster result 1. The order of lexical elements in the rearranged sequence corresponding to cluster result 1 is: lexical element 4, lexical element 6, lexical element 8, lexical element 9, lexical element 10, lexical element 3, lexical element 5, lexical element 7, lexical element 1, and lexical element 2.

[0123] It should be understood that when sorting the clusters in the clustering results in descending order according to the number of terms contained in each cluster, there may be cases where two clusters contain the same number of terms. In such cases, this application does not restrict the order of these two clusters during sorting.

[0124] To make it easier to understand, let's illustrate this with an example.

[0125] For example, taking cluster result 2 as {Category 1: word element 1, word element 2, word element 4; Category 2: word element 6, word element 8, word element 9, and word element 10; Category 3: word element 3, word element 5, word element 7}, both Category 1 and Category 2 contain 3 words. When sorting the categories in cluster result 2 in descending order, the order of the categories in cluster result 2 can be: Category 2, Category 3, and Category 1. In this case, the corresponding rearranged sequence of cluster result 2 is: word element 6, word element 8, word element 9, word element 10, word element 1, word element 2, word element 4, word element 3, word element 5, word element 7. The order of the categories in cluster result 2 can also be: Category 2, Category 1, and Category 3. In this case, the corresponding rearranged sequence of cluster result 2 is: word element 6, word element 8, word element 9, word element 10, word element 3, word element 5, word element 7, word element 1, word element 2, word element 4.

[0126] It should be noted that although the above process only describes the process of rearranging the tokens in the input sequence according to the information indicated in a clustering result to obtain a rearranged sequence, the same process can be used to obtain the rearranged sequence corresponding to other clustering results.

[0127] It should be understood that the process provided in this application for rearranging multiple tokens in an input sequence according to the information indicated in a clustering result to obtain a rearranged sequence is merely a specific example. In practical applications, it can also be implemented in other ways, and this application does not make any specific limitations.

[0128] S103: Generate attention masks corresponding to multiple rearranged sequences.

[0129] After obtaining multiple rearranged sequences, an attention mask needs to be generated for each of the rearranged sequences. The following two rules must be followed when generating the attention mask for each rearranged sequence.

[0130] (1) The size of the attention mask matrix corresponding to the rearranged sequence is the same as the size of the attention mask matrix corresponding to the input sequence.

[0131] (2) The dependencies between each word in the attention mask corresponding to the rearranged sequence are the same as those between each word in the attention mask corresponding to the input sequence.

[0132] To make it easier to understand, we will use an example below.

[0133] For example, the input sequence contains 10 words, and the order of the words in the input sequence is: word 1, word 2, word 3, word 4, word 5, word 6, word 7, word 8, word 9, and word 10. The attention mask corresponding to this input sequence is a 10*10 matrix M. If there is a dependency between word 1 and word 2 in the input sequence, then M[0,1] is the first value. If there is no dependency between word 1 and word 3 in the input sequence, then M[0,2] is the second value. Taking the rearranged sequence corresponding to the input sequence with the word order of: word 6, word 8, word 9, word 10, word 3, word 5, word 7, word 1, word 2, and word 4 as an example, the attention mask corresponding to this rearranged sequence is a matrix N, and the size of matrix N is also 10*10. When generating matrix N, N[1,0] is used to indicate whether there is a dependency relationship between word 1 and word 2, and N[1,2] is used to indicate whether there is a dependency relationship between word 1 and word 3. Since the dependencies between words in the attention mask corresponding to the rearranged sequence are the same as the dependencies between words in the attention mask corresponding to the input sequence, N[1,0] is the first value and N[1,2] is the second value. The values ​​at other positions in matrix N are also generated based on the rule that the dependencies between words in the attention mask corresponding to the rearranged sequence are the same as the dependencies between words in the attention mask corresponding to the input sequence. For the sake of simplicity in the patent, this will not be elaborated here.

[0134] Although the above example only records the process of generating an attention mask from a rearranged sequence, the same method can be used to generate corresponding attention masks for other rearranged sequences in multiple rearranged sequences.

[0135] S104: Divide the rows and columns of each attention mask in the attention masks corresponding to the multiple rearranged sequences into a specified number to obtain the lattice mask corresponding to the multiple rearranged sequences.

[0136] Each rearranged sequence has a corresponding attention mask and a lattice mask. The attention mask includes multiple sub-attention masks, and the value at any position in the lattice mask is used to indicate whether there is a dependency relationship between the terms in the corresponding sub-attention mask.

[0137] Specifically, taking one of multiple rearranged sequences (hereinafter referred to as "rearranged sequence A") as an example, the lattice mask corresponding to rearranged sequence A can be determined as follows: First, the rows and columns of the attention mask corresponding to rearranged sequence A are divided into a specified number to obtain multiple sub-attention masks; then, the lattice mask corresponding to rearranged sequence A is determined based on the values ​​in the sub-attention masks. Specifically, when a first value exists in a sub-attention mask, the value at the corresponding position in the lattice mask is the first value; when a first value does not exist in a sub-attention mask, the value at the corresponding position in the lattice mask is the second value.

[0138] It should be noted that the specified quantity can be the number of computing nodes or other values; this application does not impose any specific limitations.

[0139] The following example illustrates how to obtain the corresponding lattice mask based on the attention mask corresponding to the rearranged sequence.

[0140] See Figure 3 , Figure 3 This is a schematic diagram illustrating the process of obtaining the corresponding lattice mask based on the attention mask, as provided in this application. Figure 3 In the diagram, (a) shows an attention mask corresponding to a rearranged sequence. This attention mask is a 100*100 matrix. In (a), white areas represent dependencies between words within the region, and the value in this region is the first value (0). Black areas represent no dependencies between words within the region, and the value in this region is the second value (1). Dividing the rows and columns of this attention mask into 10 parts (the number of computation nodes) yields (b). Here, (b) shows a partitioned attention mask, containing 100 equally sized grids. Each grid represents a sub-attention mask, and each sub-attention mask is a 10*10 matrix. (c) is the lattice mask corresponding to (a), a 10*10 matrix. As can be seen, all values ​​in sub-attention mask 1 in (b) are the first value (0), indicating that there is a dependency relationship between each word in the sub-attention mask. Therefore, the value of the corresponding position of the attention mask 1 in (c) is the first value (0). The values ​​in sub-attention mask 2 in (b) are both the first value (0) and the second value (1), indicating that there is a dependency relationship between each word in the sub-attention mask 2. Therefore, the value of the corresponding position of the sub-attention mask 2 in (c) is the first value (0). The values ​​in sub-attention mask 3 in (b) are all the second value (1), indicating that there is no dependency relationship between each word in the sub-attention mask. Therefore, the value of the corresponding position of the sub-attention mask 3 in (c) is the second value (1).

[0141] Although the above example only records the process of generating the corresponding lattice mask for the attention mask corresponding to a rearranged sequence, the same method can be used to generate the corresponding lattice mask for other rearranged sequences in the attention mask corresponding to multiple rearranged sequences.

[0142] S105: Determine the first lattice mask from the lattice masks corresponding to multiple rearranged sequences, and then determine the first rearranged sequence and the first attention mask.

[0143] Based on the evaluation results, the first lattice mask is determined from the lattice masks corresponding to multiple rearranged sequences. The first rearranged sequence is the rearranged sequence corresponding to the first lattice mask, and the first attention mask is the attention mask corresponding to the first lattice mask. The number of values ​​with the first value in the first lattice mask is less than or equal to the number of values ​​with the first value in the lattice mask corresponding to the input sequence.

[0144] In one possible implementation, the process of determining the first lattice mask from the lattice masks corresponding to multiple rearranged sequences is as follows:

[0145] (1) First, calculate the first result for each of the multiple lattice masks, where the first result is the number of times the value is the first value in each lattice mask.

[0146] (2) Then, calculate the second result for each of the multiple lattice masks. The second result is the variance of the number of computational tasks that each computation node can select in each lattice mask.

[0147] In one possible implementation, the meaning of each element at each position in the lattice mask is as follows: If the value at the i-th row and j-th column position of the lattice mask is the first value, then there exists a computation task at that position, and the query matrix in this computation task comes from computation node i, the key matrix K, and the value matrix V come from computation node j. Here, the values ​​of i and j range from [0, the number of computation nodes - 1]. If the value at the i-th row and j-th column position of the lattice mask is the second value, then there is no computation task at that position.

[0148] In another possible implementation, the meaning of each element at each position in the lattice mask is as follows: If the value at the i-th row and j-th column position of the lattice mask is the first value, then there exists a computation task at that position, and in this computation task, the query matrix comes from computation node j, and the key matrix K and value matrix V come from computation node i. Here, the values ​​of i and j range from [0, the number of computation nodes - 1]. If the value at the i-th row and j-th column position of the lattice mask is the second value, then there is no computation task at that position.

[0149] It should be noted that this application does not specifically limit the meaning of the elements at each position in the lattice mask.

[0150] Each computing node can choose a computation task that belongs to at least one of the query matrix and key-value matrix (including the key matrix and value matrix). Within each rearrangement sequence, the computation task that each computing node can choose is determined by the lattice mask corresponding to the rearrangement sequence.

[0151] To make it easier to understand, the following example will illustrate how to obtain the task list for each computing node.

[0152] To facilitate understanding, the following example illustrates how to determine the selectable computational tasks for a computation node based on a lattice mask.

[0153] See Figure 4 , Figure 4 This is a schematic diagram illustrating how the computational tasks selectable by a computation node are determined based on a lattice mask, as provided in this application. Figure 4 The image shows a lattice mask corresponding to a rearranged sequence. Taking compute node 0 as an example, compute node 0 needs to find the position with the first value (0) in the first row and first column of the lattice mask. Figure 4 As can be seen, the computation task selectable by computation node 0 is region 1 in the lattice mask; taking computation node 5 as an example, computation node 5 needs to find the position with the first value (0) in the sixth row and sixth column of the lattice mask. Figure 4 As can be seen, the area in the lattice mask where the computation task that can be selected by computation node 5 is region 2.

[0154] It should be understood that the above example only demonstrates how to determine the computational tasks that a computation node can choose from based on a lattice mask in a rearranged sequence, but the same method can be used to obtain the lattice mask that each computation node can choose from for other rearranged sequences in multiple rearranged sequences.

[0155] (3) Next, the evaluation result of each grid mask in the multiple grid masks is calculated, wherein the evaluation result is the weighted average of the first result and the second result, the weight of the first result is greater than the weight of the second result, and the specific values ​​of the weight of the first result and the weight of the second result are not specifically limited in this application.

[0156] (4) Finally, the grid mask corresponding to the smallest value among the multiple evaluation results is taken as the first grid mask.

[0157] It should be understood that the above method of determining the first lattice mask from the lattice masks corresponding to multiple rearranged sequences is only a specific example. In practical applications, it can also be implemented in other ways, and this application does not make any specific limitations.

[0158] See Figure 5 , Figure 5 A schematic diagram showing the effect comparison between the input sequence provided in this application and the first rearranged sequence. Figure 5 In the diagram, (a1) is the attention mask corresponding to the input sequence, (a2) is the attention mask corresponding to the first sequence, (b1) is the lattice mask corresponding to the input sequence, and (b2) is the lattice mask corresponding to the first rearranged sequence. The values ​​in the white areas of the attention masks represent the first value, and the values ​​in the black areas represent the second value. From... Figure 5 As can be seen, the difference between the input sequence and the first rearranged sequence is as follows:

[0159] (1) Figure 5 Comparing (a1) and (a2), the white area in the attention mask corresponding to the first rearranged sequence is more concentrated than the white area in the attention mask corresponding to the input sequence.

[0160] (2) Figure 5 Comparing (b1) and (b2), the number of first values ​​(0) in the first rearranged sequence (i.e., the total number of computational tasks) is less than the number of first values ​​(0) in the input sequence (i.e., the total number of computational tasks). Specifically, Figure 5 The number of values ​​in the lattice mask corresponding to the input sequence that are the first value is 52, while the number of values ​​in the lattice mask corresponding to the first rearranged sequence (the first lattice mask) that are the first value is 36. It should be understood that when the total computational task decreases, the number of computation rounds in subsequent task scheduling will also decrease accordingly.

[0161] S106: Generate task scheduling information based on the position of the first value in the first grid mask.

[0162] The task scheduling information records the computational tasks of each computing node in each computation round. The positions in the first grid mask where the value is the first value indicate the presence of a computational task at that position. Therefore, task scheduling information can be generated based on the positions in the first grid mask where the value is the first value.

[0163] In each computation round, each computing node must satisfy both a first condition and a second condition when selecting a computation task. The first condition is that each computing node can select a computation task that belongs to at least one of the query matrix and key-value matrix. The key-value matrix includes both a key matrix and a value matrix. The second condition is that the communication volume of each computing node in each computation round is less than or equal to a communication unit threshold. This communication unit threshold can be dynamically set according to actual conditions.

[0164] The communication volume of each compute node in each compute round includes the amount of data sent and received. Specifically, a compute node sends or receives 1 unit of data for the query matrix, and 2 units of data for the key and value matrices. If a compute node sends its query matrix to other compute nodes in a compute round, it will receive a compute result in a subsequent compute round. This compute result consists of 1 unit of data and represents the result of task computation performed by other devices using the query matrix on this compute node. Similarly, if a compute node receives a query matrix from another compute node in a compute round, it will need to send a compute result in a subsequent compute round. This compute result consists of 1 unit of data and represents the compute result for the next compute round.

[0165] In one possible implementation, the steps for generating task scheduling information based on the position of the first value in the first grid mask are as follows:

[0166] Step 1: In the j-th computation round, each computation node i first checks whether the value of the first lattice mask at position (i, (i+j)%N) is 0. If the value at that position is 1, proceed to step 2; otherwise, proceed to step 3. Here, N is the number of computation nodes, i ranges from [0, N-1], and j ranges from [0, N-1], but j is usually less than N-1.

[0167] Step 2: Record the computation task of computing node i in the j-th computation round as QiK(i+j)%NV(i+j)%N, and then update the value at position (i,(i+j)%N) on the first grid mask to 0.

[0168] Step 3: Computation node i searches in a specific order. First, in the i-th row of the first grid mask, computation node i checks the value of each position within the column range [0, (i+j)%N-1] to see if it is 0. If a position with a value of 0 is found within this range, computation node i records the computation task at that position and updates the value at that position to 1. If no position with a value of 0 is found within the above range, computation node i continues to check the value of each position within the column range [(i+j)%N+1, N-1] in the i-th row of the first grid mask. If a value of 0 is found within this range, the computation task is recorded and the value at that position is updated to 1. If neither of the above two searches is successful, computation node i searches upwards, checking the value of each position in the (i+j)%N-th column of the first grid mask within the row range [0, i-1]. If a value of 0 is found within this range, the computation task is recorded and the value at that position is updated to 1. If the computation task for computation node i in the j-th computation round is still not determined, computation node i will search downwards, checking if the value of each position in the first lattice mask with row range [i+1, N-1] and column (i+j)%N is 0. If a value of 0 is found in this range, the computation task is recorded and the value at that position is updated to 0.

[0169] After searching through steps 1 to 3, the computation task of computing node i in computation round j can be determined. Determining the computation tasks of other computing nodes and computation rounds is similar to the above method, and will not be explained in detail here.

[0170] It should be noted that after computing node i completes the search for the computation task in the j-th computation round, computing node i will also determine whether the data volume of each computing node in the (j-1)-th computation round exceeds the communication unit threshold. If it does, the computation task of that computing node in the (j-1)-th computation round is deleted, and the value of the corresponding position of that computation task in the first grid mask is set to 1. Otherwise, no action is taken. Here, j is greater than or equal to 1.

[0171] It should be understood that the above steps for generating task scheduling information are merely a specific example. In practical applications, they can also be implemented in other ways, and this application does not impose any specific limitations.

[0172] S107: Perform self-attention calculation on the first rearranged sequence based on the task scheduling information.

[0173] When performing self-attention calculation on the first rearranged sequence, it is first necessary to parse the task scheduling information to determine the computation tasks and communication tasks of each computing node in each computation round. The computation tasks can be directly obtained through the task scheduling information. The communication tasks include sending and receiving tasks. The sending task specifies which computing nodes need to send the query matrix or key matrix and value matrix from this computing node in the current computation round, and / or the computation results (the computation results from the previous computation round will only be sent in the current computation round if the query matrix from other computing nodes was received in the previous computation round). The receiving task specifies the query matrix or key matrix and value matrix from other computing nodes that need to be received in the current computation round for computation in this round, and / or the computation results (the computation results from other computing nodes will only be received if the query matrix from this computing node was sent to other computing nodes in the previous computation round). The query matrix or key-value matrix (including key and value matrices) sent by other computing nodes is the query matrix or key-value matrix (including key and value matrices) required for the computation task in the next computation round. Then, each computing node executes each computation round according to the computation and communication tasks. During the actual computation task execution, each computing node sends the prepared query matrix, key matrix, and value matrix, along with the relevant sub-attention masks, to the self-attention operator for computation. Finally, the computing nodes integrate the computation results to obtain the self-attention computation result of the first rearranged sequence.

[0174] In summary, this application provides a self-attention calculation method applied to distributed computing self-attention. First, multiple words in the input sequence are rearranged to obtain a first rearranged sequence, an attention mask corresponding to the first rearranged sequence, and a lattice mask. The attention mask includes multiple sub-attention masks. The value at any position in the lattice mask indicates whether there is a dependency relationship between words in the corresponding sub-attention mask. The number of values ​​in the lattice mask corresponding to the first rearranged sequence that are the first value is less than or equal to the number of values ​​in the lattice mask corresponding to the input sequence that are the first value. The first value in the lattice mask indicates that a computational task exists at the position corresponding to that value. Then, task scheduling information is generated based on the positions of the first values ​​in the lattice mask. Finally, self-attention calculation is performed on the first rearranged sequence based on the task scheduling information. Rearranging the input sequence reduces the total computational task. By changing the scheduling method, the computational tasks of each computing node across multiple computation rounds can be more evenly distributed, solving the problem of uneven load distribution among computing nodes.

[0175] See Figure 6 , Figure 6This application provides a flowchart illustrating another self-attention computation method, which is applied to non-distributed computational self-attention. The method includes:

[0176] S201: Rearrange multiple tokens in the input sequence to obtain the second rearranged sequence and the attention mask corresponding to the second rearranged sequence.

[0177] The process of rearranging multiple words in the input sequence to obtain a second rearranged sequence includes: First, clustering the attention mask corresponding to the input sequence according to the number of categories to obtain multiple clustering results and silhouette coefficients corresponding to the multiple clustering results. The process of obtaining multiple clustering results is described in S101, and will not be repeated here for the sake of simplicity. Then, the clustering result corresponding to the silhouette coefficient with the largest value among the multiple clustering results is taken as the first clustering result. Finally, the multiple words in the input sequence are rearranged according to the information indicated in the first clustering result to obtain a second rearranged sequence. The relative order of multiple words within the same category in the first clustering result is the same as the relative order of corresponding words in the second rearranged sequence. The process of obtaining the second rearranged sequence is described in S101, and will not be repeated here for the sake of simplicity.

[0178] Besides using the cluster with the largest silhouette coefficient among multiple clustering results as the first clustering result, other parameters can also be used as the standard for evaluating the first clustering result. For example, the cluster with the smallest standard deviation of the cluster size dev among multiple clustering results can be used as the first clustering result. This application does not specifically limit the method for determining the first clustering result.

[0179] The process of generating the attention mask corresponding to the second rearranged sequence based on the second rearranged sequence is described in S103, and will not be repeated here for the sake of the patent's simplicity.

[0180] See Figure 7 , Figure 7 A comparative diagram showing the attention mask corresponding to the input sequence provided in this application and the attention mask corresponding to the second rearranged sequence. Figure 7 In the diagram, (a) is the attention mask corresponding to the input sequence, and (b) is the attention mask corresponding to the second rearranged sequence. Figure 7 The values ​​in the white area are the first values, and the values ​​in the black area are the second values. From... Figure 7 As can be seen, the white area of ​​the attention mask corresponding to the second rearranged sequence is more concentrated than that of the attention mask corresponding to the input sequence, which indicates that the computation task is more focused after the input sequence is rearranged.

[0181] S202: Determine the position information of one or more regions in the attention mask corresponding to the second rearrangement sequence.

[0182] The process of determining the position information of one or more regions in the attention mask corresponding to the second rearrangement sequence is as follows: First, the attention mask corresponding to the second rearrangement sequence is mapped onto a coordinate system; then, regions in the attention mask corresponding to the second rearrangement sequence that satisfy the user's pre-defined values ​​are identified in the coordinate system, where all values ​​in these regions are the second value. Users can pre-define the shape and size of the regions according to their needs. For example, the shape of the region can be circular, and the minimum radius of the region can be the first radius; another example is that the shape of the region can be rectangular, with a minimum length of the first length and a minimum width of the first width. This application does not specifically limit the shape and size of the regions. A region satisfying the user's pre-defined values ​​refers to a region that satisfies the shape set by the user and whose size is greater than or equal to the user's preset size; finally, the position information of each identified region is recorded. For example, when the user pre-defines the shape of the region as a rectangle, with a minimum length of the first length and a minimum width of the first width, after finding one or more regions that satisfy the first length and the first width from the attention mask corresponding to the second rearrangement sequence, the coordinates of the four corners of this region are recorded. This region can be... Figure 7 The first and second regions.

[0183] S203: Input the location information of one or more regions and the attention mask corresponding to the second rearranged sequence into the self-attention operator to perform self-attention calculation on the second rearranged sequence.

[0184] By inputting the query matrix, key matrix, and value matrix corresponding to the second rearranged sequence, the attention mask corresponding to the second rearranged sequence, and the location information of one or more regions into the self-attention operator, self-attention computation of the second rearranged sequence can be achieved. The self-attention operator can be either block attention or flash attention; this application does not impose any specific limitations.

[0185] The location information of one or more regions is used to indicate that the corresponding position of the region in the second rearrangement sequence will not be subject to self-attention calculation. Compared with the self-attention operators in the prior art, the self-attention operator in this application adds the function of determining the range of self-attention calculation based on the location information of one or more input regions, reducing unnecessary calculations and saving computational resources.

[0186] In summary, this application provides another self-attention computation method applied to non-distributed self-attention computation. First, multiple tokens in the input sequence are rearranged to obtain a second rearranged sequence and an attention mask corresponding to it. Then, the position information of one or more regions in the attention mask corresponding to the second rearranged sequence is determined. Each region is defined as a region in the attention mask that satisfies a first length and a first width, and all values ​​within the region are second values. Finally, the position information of one or more regions and the attention mask corresponding to the second rearranged sequence are input into the self-attention operator to perform self-attention computation on the second rearranged sequence. The position information of one or more regions indicates that the corresponding position of that region in the second rearranged sequence should not be subject to self-attention computation. By implementing these steps, the self-attention operator can skip unnecessary computations, reducing the waste of computational resources.

[0187] This application provides a self-attention computing device, such as Figure 8 As shown, Figure 8 A schematic diagram of a self-attention computing device 300 provided in this application includes:

[0188] The acquisition module is used to rearrange multiple words in the input sequence to obtain a first rearranged sequence, an attention mask and a lattice mask corresponding to the first rearranged sequence. The attention mask includes multiple sub-attention masks. The value at any position in the lattice mask is used to indicate whether there is a dependency relationship between words in the corresponding sub-attention mask. The number of values ​​with the first value in the lattice mask corresponding to the first rearranged sequence is less than or equal to the number of values ​​with the first value in the lattice mask corresponding to the input sequence. The value with the first value in the lattice mask is used to indicate that there is a computation task at the position corresponding to that value.

[0189] The generation module is used to generate task scheduling information based on the position of the first value in the grid mask;

[0190] The calculation module is used to perform self-attention calculation on the first rearranged sequence based on task scheduling information.

[0191] In possible implementations, the acquisition module is used for:

[0192] Clustering is performed on the attention mask corresponding to the input sequence to obtain multiple clustering results. Each clustering result records the information of the words in the input sequence that are classified into the same category.

[0193] Multiple words in the input sequence are rearranged according to the information indicated in each clustering result to obtain multiple rearranged sequences. Each clustering result has a corresponding rearranged sequence. The relative order of multiple words in the same category in the clustering result is the same as the relative order of the corresponding words in the rearranged sequence of the clustering result.

[0194] Generate attention masks corresponding to multiple rearranged sequences, wherein the dependencies between each word in the attention mask corresponding to the rearranged sequence are the same as the dependencies between each word in the attention mask corresponding to the input sequence;

[0195] Divide the rows and columns of each attention mask in the attention mask corresponding to multiple rearranged sequences into a specified number to obtain the lattice mask corresponding to multiple rearranged sequences;

[0196] The first lattice mask is determined from the lattice masks corresponding to multiple rearranged sequences. The rearranged sequence corresponding to the first lattice mask is the first rearranged sequence. The attention mask corresponding to the first lattice mask is the attention mask corresponding to the first rearranged sequence. The first lattice mask is the lattice mask corresponding to the first rearranged sequence.

[0197] In a possible implementation, when there is a dependency between terms within a sub-attention mask, the value of the corresponding position of the sub-attention mask on the corresponding lattice mask is the first value.

[0198] In possible implementations, the acquisition module is used for:

[0199] Calculate the first result for each of the multiple lattice masks, where the first result is the number of times the value in each lattice mask is the first value;

[0200] A second result is calculated for each of the multiple lattice masks, the second result being the variance of the number of computational tasks that each computing device can select in each lattice mask;

[0201] Calculate the evaluation result for each lattice mask among multiple lattice masks. The evaluation result is the weighted average of the first result and the second result.

[0202] The grid mask corresponding to the smallest value among multiple evaluation results is used as the first grid mask.

[0203] In a possible implementation, each computing node can choose a computing task that belongs to the current computing node, either the query matrix or the key-value matrix, where the key-value matrix includes the key matrix and the value matrix.

[0204] In a possible implementation, when each computing node selects a computing task in each computing round in the task scheduling information, it needs to satisfy a first condition and a second condition. The first condition is that the computing task selected by each computing node is a computing task belonging to the current computing node in at least one of the query matrix and the key-value matrix, where the key-value matrix includes the key matrix and the value matrix. The second condition is that the communication volume of each computing node in each computing round is less than or equal to the communication unit threshold.

[0205] In a possible implementation, the acquisition module is used to rearrange multiple tokens in the input sequence when the self-attention computation is non-distributed computation, to obtain the second rearranged sequence and the attention mask corresponding to the second rearranged sequence.

[0206] The module also includes a determination module, used to determine the position information of one or more regions in the attention mask corresponding to the second rearrangement sequence. The region is a region in the attention mask corresponding to the second rearrangement sequence that satisfies a first length and a first width, and all values ​​in the region are second values.

[0207] The calculation module is used to input the location information of one or more regions and the attention mask corresponding to the second rearrangement sequence into the self-attention operator to perform self-attention calculation on the second rearrangement sequence. The location information of one or more regions is used to indicate that the corresponding position of the region in the second rearrangement sequence does not perform self-attention calculation.

[0208] In possible implementations, the acquisition module is used for:

[0209] Cluster the attention mask corresponding to the input sequence to obtain the first clustering result. The first clustering result records the information of the words in the input sequence that are classified into the same category.

[0210] The multiple words in the input sequence are rearranged according to the information indicated in the first clustering result to obtain the second rearranged sequence. The relative order of multiple words in the same category in the first clustering result is the same as the relative order of the corresponding words in the second rearranged sequence.

[0211] Generate an attention mask corresponding to the second rearranged sequence. The dependencies between the terms in the attention mask corresponding to the second rearranged sequence are the same as those between the terms in the attention mask corresponding to the input sequence.

[0212] In a possible implementation, the acquisition module is used to cluster the attention mask corresponding to the input sequence to obtain multiple clustering results and silhouette coefficients corresponding to the multiple clustering results; the clustering result corresponding to the silhouette coefficient with the largest value among the multiple clustering results is taken as the first clustering result.

[0213] The acquisition module, generation module, calculation module, and determination module can all be implemented in software or hardware. For example, the implementation of the acquisition module will be described below. Similarly, the implementation methods of the generation module, calculation module, and determination module can refer to the implementation method of the acquisition module.

[0214] As an example of a software functional unit, a module can include code running on a computing instance. A computing instance can include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned computing instance can be one or more. For example, a module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region can include multiple AZs.

[0215] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0216] As an example of a hardware functional unit, module A may include at least one computing device, such as a server. Alternatively, the acquisition module may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0217] The acquisition module includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the acquisition module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the acquisition module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0218] It should be noted that, in other embodiments, the acquisition module can be used to execute any step in the self-attention calculation method, the generation module can be used to execute any step in the self-attention calculation method, the calculation module can be used to execute any step in the self-attention calculation method, and the determination module can be used to execute any step in the self-attention calculation method. The steps implemented by the acquisition module, generation module, calculation module, and determination module can be specified as needed. By implementing different steps in the self-attention calculation method through the acquisition module, generation module, calculation module, and determination module, all functions of the self-attention calculation device can be realized.

[0219] This application also provides a computing device 400. For example... Figure 9As shown, the computing device 400 includes a processor 410, a memory 420, a communication interface 430, and a bus 440. The processor 410, memory 420, and communication interface 430 communicate with each other via the bus 440. The computing device 400 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 400.

[0220] The processor 410 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, TPU, DPU, SoC, offload card, accelerator card, etc.

[0221] Memory 420 may include volatile memory, such as random access memory (RAM). Processor 410 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 420 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0222] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 420, or two or more types of storage media can be configured to realize the function of memory 420. This application does not limit this.

[0223] The memory 420 stores executable program code, which the processor 410 executes to implement the functions of the aforementioned acquisition module, generation module, calculation module, and determination module, thereby realizing the self-attention calculation method. That is, the memory 420 stores instructions for executing the self-attention calculation method.

[0224] The communication interface 430 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 400 and other devices or communication networks.

[0225] Bus 440 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc. The Unified Bus is also known as the Lingqu Bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 5 The bus 440 is represented by only one line, but this does not mean that there is only one bus or one type of bus. The bus 440 may include a path for transmitting information between various components of the computing device 400 (e.g., processor 410, memory 420, communication interface 430). The unified bus may also be called the Lingqu bus.

[0226] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0227] like Figure 10 As shown, the computing device cluster includes at least one computing device 400. The memory 420 of one or more computing devices 400 in the computing device cluster may store the same instructions for performing the self-attention computing method.

[0228] In some possible implementations, the memory 420 of one or more computing devices 400 in the computing device cluster may also store partial instructions for executing the self-attention computing method. In other words, a combination of one or more computing devices 400 can jointly execute the instructions for executing the self-attention computing method.

[0229] It should be noted that the memories 420 in different computing devices 400 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the self-attention computing device. That is, the instructions stored in the memories 420 of different computing devices 400 can implement the functions of one or more modules among the acquisition module, generation module, calculation module, and determination module.

[0230] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 11 One possible implementation is shown. For example... Figure 11 As shown, two computing devices 400A and 400B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 420 in computing device 400A stores instructions for executing the functions of the computing module. Simultaneously, the memory 420 in computing device 400B stores instructions for executing the functions of the acquisition module, template generation module, and determination module.

[0231] Figure 11 The connection method between the computing device clusters shown can be considered in this application, which requires relatively complex preprocessing and modification of the scheduling method to balance the load of each computing node. Therefore, the functions implemented by the acquisition module, generation module and determination module are considered to be executed by the computing device 400B.

[0232] It should be understood that Figure 11 The functions of the computing device 400A shown can also be performed by multiple computing devices 400. Similarly, the functions of the computing device 400B can also be performed by multiple computing devices 400.

[0233] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 9 The connection method of the computing device cluster is different in that the memory 420 of one or more computing devices 400 in the computing device cluster can store the same instructions for executing the self-attention computing method.

[0234] In some possible implementations, the memory 420 of one or more computing devices 400 in the computing device cluster may also store partial instructions for executing the self-attention computing method. In other words, a combination of one or more computing devices 400 can jointly execute the instructions for executing the self-attention computing method.

[0235] It should be noted that the memory 420 in different computing devices 400 in the computing device cluster can store different instructions to implement the functions of the self-attention computing device.

[0236] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a self-attention computation method.

[0237] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a self-attention method.

[0238] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A self-attention calculation method, characterized in that, The method includes: Multiple words in the input sequence are rearranged to obtain a first rearranged sequence, an attention mask and a lattice mask corresponding to the first rearranged sequence. The attention mask includes multiple sub-attention masks. The value at any position in the lattice mask is used to indicate whether there is a dependency relationship between words in the sub-attention mask corresponding to that position. The number of values ​​with a first value in the lattice mask corresponding to the first rearranged sequence is less than or equal to the number of values ​​with a first value in the lattice mask corresponding to the input sequence. The value with a first value in the lattice mask is used to indicate that there is a computational task at the position corresponding to that value. Based on the position of the first value in the lattice mask, task scheduling information is generated; Based on the task scheduling information, self-attention calculation is performed on the first rearranged sequence.

2. The method according to claim 1, wherein rearranging multiple tokens in the input sequence to obtain a first rearranged sequence, an attention mask corresponding to the first rearranged sequence, and a lattice mask comprises: Clustering is performed on the attention mask corresponding to the input sequence to obtain multiple clustering results, wherein each clustering result records information about the words in the input sequence that are classified into the same category; Multiple words in the input sequence are rearranged according to the information indicated in each clustering result to obtain multiple rearranged sequences. Each clustering result has a corresponding rearranged sequence. The relative order of multiple words in the same category in the clustering result is the same as the relative order of the corresponding words in the rearranged sequence corresponding to the clustering result. Generate attention masks corresponding to the plurality of rearranged sequences, wherein the dependencies between each word in the attention mask corresponding to the rearranged sequence are the same as the dependencies between each word in the attention mask corresponding to the input sequence; The rows and columns of each attention mask in the attention masks corresponding to the plurality of rearranged sequences are divided into a specified number to obtain the lattice mask corresponding to the plurality of rearranged sequences; A first lattice mask is determined from the lattice masks corresponding to the plurality of rearranged sequences, wherein the rearranged sequence corresponding to the first lattice mask is the first rearranged sequence, the attention mask corresponding to the first lattice mask is the attention mask corresponding to the first rearranged sequence, and the first lattice mask is the lattice mask corresponding to the first rearranged sequence.

3. The method according to claim 1 or 2, characterized in that, When there is a dependency between terms within a sub-attention mask, the value of the sub-attention mask at the corresponding position on the corresponding lattice mask is the first value.

4. The method according to claim 2 or 3, characterized in that, Determining the first lattice mask from the lattice masks corresponding to the plurality of rearranged sequences includes: Calculate a first result for each of the multiple lattice masks, where the first result is the number of values ​​in each lattice mask that are the first value; Calculate a second result for each of the plurality of lattice masks, the second result being the variance of the number of computational tasks that each computation node can select in each lattice mask; Calculate the evaluation result for each of the plurality of lattice masks, wherein the evaluation result is a weighted average of the first result and the second result; The grid mask corresponding to the smallest value among multiple evaluation results is used as the first grid mask.

5. The method according to claim 4, characterized in that, The computation task that each computing node can select is a computation task that belongs to the current computing node, which is either a query matrix or a key-value matrix, wherein the key-value matrix includes a key matrix and a value matrix.

6. The method according to claim 5, characterized in that, In the task scheduling information, each computing node needs to meet a first condition and a second condition when selecting a computing task in each computing round. The first condition is that the computing task selected by each computing node is a computing task belonging to the current computing node, where the key-value matrix includes a key matrix and a value matrix. The second condition is that the communication volume of each computing node in each computing round is less than or equal to the communication unit threshold.

7. The method according to claim 1, characterized in that, The method further includes: When the self-attention computation is non-distributed computation, multiple words in the input sequence are rearranged to obtain a second rearranged sequence and an attention mask corresponding to the second rearranged sequence; Determine the position information of one or more regions in the attention mask corresponding to the second rearrangement sequence, wherein the region is a region in the attention mask corresponding to the second rearrangement sequence that satisfies a first length and a first width, and all values ​​in the region are second values; The location information of one or more regions and the attention mask corresponding to the second rearrangement sequence are input into the self-attention operator to perform self-attention calculation on the second rearrangement sequence. The location information of one or more regions is used to indicate that the corresponding position of the region in the second rearrangement sequence is not subject to self-attention calculation.

8. The method according to claim 7, characterized in that, The step of rearranging multiple tokens in the input sequence to obtain a second rearranged sequence and an attention mask corresponding to the second rearranged sequence includes: Clustering is performed on the attention mask corresponding to the input sequence to obtain a first clustering result, which records the information of the words in the input sequence that are classified into the same category; The multiple words in the input sequence are rearranged according to the information indicated in the first clustering result to obtain a second rearranged sequence. The relative order of multiple words in the same category in the first clustering result is the same as the relative order of the corresponding words in the second rearranged sequence. Generate an attention mask corresponding to the second rearranged sequence. The dependencies between each word in the attention mask corresponding to the second rearranged sequence are the same as the dependencies between each word in the attention mask corresponding to the input sequence.

9. The method according to claim 8, characterized in that, The step of clustering the attention mask corresponding to the input sequence to obtain the first clustering result includes: Cluster the attention mask corresponding to the input sequence to obtain multiple clustering results and the contour coefficients corresponding to the multiple clustering results; The clustering result with the largest silhouette coefficient among the multiple clustering results is taken as the first clustering result.

10. A self-attention computing device, characterized in that, The device includes: The acquisition module is used to rearrange multiple tokens in the input sequence to obtain a first rearranged sequence, an attention mask and a lattice mask corresponding to the first rearranged sequence. The attention mask includes multiple sub-attention masks. The value at any position in the lattice mask is used to indicate whether there is a dependency relationship between the tokens in the sub-attention mask corresponding to the position. The number of values ​​with a first value in the lattice mask corresponding to the first rearranged sequence is less than or equal to the number of values ​​with a first value in the lattice mask corresponding to the input sequence. The value with a first value in the lattice mask is used to indicate that there is a computational task at the position corresponding to the value. The generation module is used to generate task scheduling information based on the position of the first value in the lattice mask; The calculation module is used to perform self-attention calculation on the first rearranged sequence based on the task scheduling information.

11. The apparatus according to claim 10, characterized in that, The acquisition module is used for: Clustering is performed on the attention mask corresponding to the input sequence to obtain multiple clustering results, wherein each clustering result records information about the words in the input sequence that are classified into the same category; Multiple words in the input sequence are rearranged according to the information indicated in each clustering result to obtain multiple rearranged sequences. Each clustering result has a corresponding rearranged sequence. The relative order of multiple words in the same category in the clustering result is the same as the relative order of the corresponding words in the rearranged sequence corresponding to the clustering result. Generate attention masks corresponding to the plurality of rearranged sequences, wherein the dependencies between each word in the attention mask corresponding to the rearranged sequence are the same as the dependencies between each word in the attention mask corresponding to the input sequence; The rows and columns of each attention mask in the attention masks corresponding to the plurality of rearranged sequences are divided into a specified number to obtain the lattice mask corresponding to the plurality of rearranged sequences; A first lattice mask is determined from the lattice masks corresponding to the plurality of rearranged sequences, wherein the rearranged sequence corresponding to the first lattice mask is the first rearranged sequence, the attention mask corresponding to the first lattice mask is the attention mask corresponding to the first rearranged sequence, and the first lattice mask is the lattice mask corresponding to the first rearranged sequence.

12. The apparatus according to claim 10 or 11, characterized in that, When there is a dependency between terms within a sub-attention mask, the value of the sub-attention mask at the corresponding position on the corresponding lattice mask is the first value.

13. The apparatus according to claim 11 or 12, characterized in that, The acquisition module is used for: Calculate a first result for each of the multiple lattice masks, where the first result is the number of values ​​in each lattice mask that are the first value; Calculate a second result for each of the plurality of lattice masks, the second result being the variance of the number of computational tasks that each computation node can select in each lattice mask; Calculate the evaluation result for each of the plurality of lattice masks, wherein the evaluation result is a weighted average of the first result and the second result; The grid mask corresponding to the smallest value among multiple evaluation results is used as the first grid mask.

14. The apparatus according to claim 13, characterized in that, The computation task that each computing node can select is a computation task that belongs to the current computing node, which is either a query matrix or a key-value matrix, wherein the key-value matrix includes a key matrix and a value matrix.

15. The apparatus according to claim 13 or 14, characterized in that, In the task scheduling information, each computing node needs to meet a first condition and a second condition when selecting a computing task in each computing round. The first condition is that the computing task selected by each computing node is a computing task belonging to the current computing node, where the key-value matrix includes a key matrix and a value matrix. The second condition is that the communication volume of each computing node in each computing round is less than or equal to the communication unit threshold.

16. The apparatus according to claim 10, characterized in that, The acquisition module is used to rearrange multiple words in the input sequence when the self-attention calculation is non-distributed calculation, to obtain a second rearranged sequence and an attention mask corresponding to the second rearranged sequence; The module further includes a determining module, which is used to determine the position information of one or more regions in the attention mask corresponding to the second rearrangement sequence. The region is a region in the attention mask corresponding to the second rearrangement sequence that satisfies a first length and a first width, and all values ​​in the region are second values. The calculation module is used to input the location information of the one or more regions and the attention mask corresponding to the second rearrangement sequence into the self-attention operator to perform self-attention calculation on the second rearrangement sequence, wherein the location information of the one or more regions is used to indicate that the corresponding position of the region in the second rearrangement sequence does not perform self-attention calculation.

17. The apparatus according to claim 16, characterized in that, The acquisition module is used for: Clustering is performed on the attention mask corresponding to the input sequence to obtain a first clustering result, which records the information of the words in the input sequence that are classified into the same category; The multiple words in the input sequence are rearranged according to the information indicated in the first clustering result to obtain a second rearranged sequence. The relative order of multiple words in the same category in the first clustering result is the same as the relative order of the corresponding words in the second rearranged sequence. Generate an attention mask corresponding to the second rearranged sequence. The dependencies between each word in the attention mask corresponding to the second rearranged sequence are the same as the dependencies between each word in the attention mask corresponding to the input sequence.

18. The apparatus according to claim 17, characterized in that, The acquisition module is used to cluster the attention mask corresponding to the input sequence to obtain multiple clustering results and contour coefficients corresponding to the multiple clustering results; and to take the clustering result corresponding to the contour coefficient with the largest value among the multiple clustering results as the first clustering result.

19. A computing device, characterized in that, The computing device includes a memory and a processor; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1 to 9.

20. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 9.

21. A computer program product containing instructions, characterized in that, When the instructions are executed on a cluster of computing devices, the cluster of computing devices performs the method as described in any one of claims 1 to 9.

22. A computer-readable storage medium, characterized in that, Includes computer program instructions that, when executed on a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1 to 9.

Images