Distributed attention processing method, apparatus, device, system and storage medium

Abstract

The disclosure provides a distributed attention processing method, device, equipment, system and storage medium, relates to the technical field of data processing, and particularly relates to the technical field of artificial intelligence, big data and the like. The specific implementation scheme is as follows: determining an initial matrix to be subjected to attention processing and a plurality of computing nodes required for performing attention processing; performing block processing on the initial matrix to obtain a plurality of computing regions; determining the number of effective computing blocks in each computing region subjected to mask processing; based on the number of effective computing blocks in each computing region, sequentially allocating the required computing nodes to each computing region to obtain target nodes corresponding to each computing region; wherein the target node is one of the plurality of computing nodes and is used for performing computing processing on the effective computing blocks in the computing region.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to the fields of artificial intelligence, big data and other technologies. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep learning models have been widely used in many fields such as natural language processing and computer vision, with the demand for processing long sequence data increasing day by day. In long sequence processing scenarios, the attention mechanism, as a core component, can effectively capture long-distance dependencies in sequence data, significantly improving the model's representation ability and task performance. Therefore, it has been widely integrated into various deep learning models (such as the Transformer model).

[0003] However, the computational and storage complexity of the Attention mechanism increases quadratically with the sequence length, leading to a sharp increase in computational load when processing extremely long sequence data. At the same time, it places extremely high demands on the GPU's memory space, which can easily cause GPU memory overflow (OOM) problems, severely limiting the feasibility and efficiency of long sequence tasks (such as training or inference tasks). Summary of the Invention

[0004] This disclosure provides a distributed attention processing method, apparatus, device, system, and storage medium.

[0005] According to one aspect of this disclosure, a distributed attention processing method is provided, comprising: Determine the initial matrix to be subjected to attention processing, and the multiple computing nodes required to perform the attention processing; wherein, the initial matrix is ​​obtained after feature processing of the initial text; The initial matrix is ​​divided into blocks to obtain multiple computational regions; Determine the number of valid computation blocks in each computation region after masking; Based on the number of valid computing blocks in each computing region, the required computing nodes are allocated to each computing region in sequence to obtain the target node corresponding to each computing region; wherein, the target node is one of the plurality of computing nodes, used to perform computing processing on the valid computing blocks in the computing region.

[0006] According to another aspect of this disclosure, a distributed attention processing apparatus is provided, comprising: A preprocessing unit is used to determine an initial matrix to be subjected to attention processing, and multiple computing nodes required to perform the attention processing; wherein, the initial matrix is ​​obtained after feature processing of the initial text; the initial matrix is ​​divided into blocks to obtain multiple computing regions; and the number of effective computing blocks in each computing region after masking processing is determined. The resource allocation unit is used to allocate the required computing nodes to each computing region in sequence based on the number of valid computing blocks in each computing region, so as to obtain the target node corresponding to each computing region; wherein, the target node is one of the plurality of computing nodes, and is used to perform computing processing on the valid computing blocks in the computing region.

[0007] According to another aspect of this disclosure, a distributed attention processing system is provided, comprising: Multiple computing nodes; A scheduling node is used to determine the initial matrix to be processed for attention, and the multiple computing nodes required to perform the attention processing; wherein the initial matrix is ​​obtained after feature processing of the initial text; the initial matrix is ​​divided into blocks to obtain multiple computing regions; the number of valid computing blocks in each computing region after masking is determined; based on the number of valid computing blocks in each computing region, the required computing nodes are allocated to each computing region in sequence to obtain the target node corresponding to each computing region; wherein the target node is one of the multiple computing nodes, used to perform computation processing on the valid computing blocks in the computing region.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0011] In this way, the proposed solution first divides the initial matrix into blocks to obtain multiple computation regions, then counts the number of effective computation blocks in each computation region after masking, and finally, sorts the computation regions based on the number of effective computation blocks and allocates the required target nodes. Thus, on the one hand, the proposed solution achieves accurate assessment of computational load through quantitative statistics of effective computation blocks, and on the other hand, achieves dynamic load balancing of computing nodes in a distributed environment through an allocation strategy based on the quantitative statistics results. This effectively avoids the problem of memory overflow of computing nodes, and also avoids the problem of some computing nodes being overloaded while others are idle and wasting resources, thereby improving the overall processing efficiency. Especially for long sequences, it can significantly improve the efficiency of attention computation.

[0012] Furthermore, the proposed solution is applicable to any mask processing strategy. In other words, the proposed solution supports diverse and flexible mask processing schemes, thus improving overall processing efficiency while also possessing versatility.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is an illustrative flowchart of a distributed attention processing method according to an embodiment of this application. Figure 1 ; Figure 2 This is an illustrative schematic diagram of a calculation region according to an embodiment of this application; Figure 3(a) is an illustrative schematic diagram of an index-number of valid blocks mapping table according to an embodiment of this application; Figure 3(b) is an illustrative schematic diagram of the calculation results of the region allocation according to an embodiment of this application; Figure 4 This is an illustrative diagram illustrating a masked computational region and the computational node to which the computational region belongs, according to an embodiment of this application. Figure 5 This is an illustrative flowchart of a distributed attention processing method according to an embodiment of this application. Figure 2 ; Figure 6 This is a schematic flowchart of a distributed attention processing method according to an embodiment of this application; Figure 7(a) is a schematic diagram of the implementation flow of a distributed attention processing method according to an embodiment of the present application in a specific example; Figure 7(b) is a schematic diagram of the allocation process of a distributed attention processing method according to an embodiment of the present application in a specific example; Figure 7(c) is a schematic diagram of the execution scheduling of overlapping computation and communication in a specific example of a distributed attention processing method according to an embodiment of the present application; Figure 8 This is a schematic diagram of a distributed attention processing device according to an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a distributed attention processing system according to an embodiment of this application; Figure 10 This is a block diagram of an electronic device used to implement the distributed attention processing method of the embodiments of this disclosure. Detailed Implementation

[0015] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0016] In this document, the term "and / or" merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The term "at least one" in this document indicates any combination of at least two of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this document refer to and distinguish between multiple similar technical terms, not to restrict the order or to limit there to only two. For example, "first feature" and "second feature" refer to two categories / two features; the first feature can be one or more, and the second feature can also be one or more.

[0017] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can still be practiced even without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0018] The following describes the related technologies of the embodiments of this disclosure. The following related technologies are optional solutions and can be combined with the technical solutions of the embodiments of this disclosure in any way, and they all fall within the protection scope of the embodiments of this disclosure.

[0019] In large-scale model training or inference based on self-attention models, or what may be called Transformer architectures, the attention mechanism is a core computational module. It can effectively capture long-distance dependencies in long-sequence data and directly affect the model's representation ability and task performance.

[0020] For example, in one implementation, the standard calculation formula for the Attention mechanism can be expressed as: ; Where Q represents the query matrix, K represents the key matrix, V represents the value matrix, and Mask represents the mask matrix; Y represents the output result after attention processing.

[0021] Furthermore, in one example, the dimension of Y can be represented as [B, S, H]. The dimension of Mask can be specifically [B, a, S, S]; where B represents the batch size, S is the sequence length of the input sequence, H is the hidden size, and a is the number of attention heads.

[0022] At this point, in long sequence processing scenarios (such as training scenarios), the computational and storage complexity of the Attention mechanism increases quadratically with the sequence length S (i.e., O(S²)). This characteristic leads to a sharp increase in computation when processing ultra-long sequence data, placing extremely high demands on the GPU's memory space and easily causing GPU memory overflow (OOM) problems.

[0023] Based on this, in order to solve the aforementioned memory overflow problem, the relevant technology adopts a distributed training scheme of context parallelism. This scheme divides the input Q, K, V along the dimension of the sequence length S and distributes the computational tasks after division to multiple GPUs for parallel execution, thereby alleviating the memory pressure of a single GPU.

[0024] However, the masks used in actual training are usually highly sparsity, which makes QK TIn this matrix, a large number of elements are masked and do not need to participate in the actual computation. However, the context parallel strategy simply segments the input along the S dimension without considering the sparsity of the mask. This results in a severe imbalance in the number of non-masked regions within the slice areas handled by different GPUs. Consequently, during the distributed attention computation, some GPUs are overloaded while others remain idle for extended periods. This not only fails to fully leverage the parallel advantages of distributed computing but also further reduces the overall training efficiency due to synchronization delays between devices, making it difficult to meet the actual needs of training large models and long sequences.

[0025] Based on this, the present application aims to address the problems of GPU OOM caused by uneven GPU load, limited mask type compatibility, high communication overhead, and low operating efficiency in existing distributed attention processing methods. Specifically, this disclosure proposes a novel dynamic load balancing distributed attention mechanism computation method. This method not only accurately quantifies the load status of each computing node through a refined calculation of the number of effective computing blocks, but also optimizes the allocation strategy of computing regions based on computing resource characteristics and a preset communication penalty factor, thereby achieving a balanced global load distribution and effective control of communication overhead.

[0026] Specifically, Figure 1 This is an illustrative flowchart of a distributed attention processing method according to an embodiment of this application. Figure 1 This method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0027] Furthermore, the method includes at least a portion of the following: For example... Figure 1 As shown, it includes: Step S101: Determine the initial matrix to be processed for attention, and the multiple computing nodes required to perform the attention processing.

[0028] Here, the initial matrix is ​​obtained after feature processing of the initial text (e.g., long sequence text); the feature processing can refer to a series of preprocessing operations that convert the initial text in natural language form into a numerical feature matrix that can be recognized and calculated by the deep learning model. Its core purpose is to preserve the semantic information and sequence order information of the initial text and adapt to the input format requirements of the attention mechanism.

[0029] Furthermore, in one example, the initial text could be a long sequence of text from a large model's long sequence training task.

[0030] Furthermore, in one example, the initial matrix obtained by performing feature processing on the initial text may specifically include the query matrix Q, the key matrix K, and the value matrix V required for attention processing. For instance, in one example, this initial matrix is ​​the matrix obtained by performing matrix multiplication (e.g., dot product) on the query matrix Q and the key matrix K, and can represent the similarity between the query matrix Q and the key matrix K, such as QK in attention processing. T .

[0031] It should be noted that the computing node is a hardware unit with data computing and storage capabilities. For example, in one example, the computing node may specifically be a Graphics Processing Unit (GPU), or it may be other neural network processors that can be used for neural network processing. This disclosure does not impose specific limitations on the computing node.

[0032] It should be noted that this disclosed solution can be applied to model training scenarios or model inference scenarios, and this disclosed solution does not impose specific restrictions on specific application scenarios.

[0033] Step S102: Divide the initial matrix into blocks to obtain multiple computation regions.

[0034] For example, in one example, QK is the initial matrix used in the attention calculation process. T In this case, matrix QK can be used. T The computation is divided into blocks, creating multiple computation regions of predetermined sizes. Each computation region is an independent computation block, denoted as matrix QK. T A subset (also called a local submatrix) of matrix QK has a size smaller than matrix QK. T This provides data support for subsequent distributed computing.

[0035] It should be noted that in practical applications, the number of computing regions can be related to the number of computing nodes. In other words, in practical applications, the number of computing regions can be determined based on the number of computing nodes that can be called.

[0036] For example, in one example, assuming there are 'a' callable computing nodes, the number of computing regions can be determined to be an integer multiple of 'a', such as k (an integer greater than or equal to 1), based on the size of the initial matrix and / or the length of the initial text sequence, or the processing power of the computing nodes, resulting in a total of 'ka' computing regions. Further, the M x N matrix QK can be divided column-wise. T The system is divided into ka computational regions, each with a size of M rows (N / (ka)) columns.

[0037] For example, there are 4 callable computation nodes, matrix QK.T The size is 16×16. At this time, as shown... Figure 2 As shown, a 16×16 matrix can be divided into 16 computational regions by columns, with each computational region being 16×1.

[0038] Step S103: Determine the number of valid computation blocks in each computation region after masking.

[0039] Here, in one example, a preset mask matrix can be used to mask some elements of the initial matrix. For example, the initial matrix after masking can be represented as follows: At this point, based on the aforementioned matrix, the computational regions after masking can be determined.

[0040] It should be noted that this disclosure does not restrict the specific processing logic of masking. In other words, this disclosure is applicable to any masking scheme, thus further enhancing the practicality of this disclosure.

[0041] Furthermore, it should be noted that in one example, after determining the initial matrix, it can be masked first, and then the masked initial matrix can be divided into blocks to obtain the masked computational regions. Alternatively, in another example, the initial matrix can be divided into blocks to obtain the computational regions, and then the initial matrix can be masked to obtain the masked computational regions. In other words, this disclosed solution does not impose specific restrictions on the execution order of masking and block division.

[0042] Furthermore, in one example, a valid computation block can specifically refer to a data block that is not masked after masking and needs to participate in the actual computation. For example, in one example, a valid computation block is a data block in the matrix whose element values ​​are not specified (-∞ or 0) after masking.

[0043] For example, in one example, a Boolean matrix can be used to count the number of valid computation blocks. For instance, if the Boolean matrix of a data block is 1, then the data block is considered a valid computation block and needs to participate in subsequent actual computation; otherwise, the data block is considered an invalid computation block and does not need to participate in subsequent actual computation.

[0044] It should be noted that this public scheme does not impose specific restrictions on the statistical method of valid computing blocks.

[0045] Step S104: Based on the number of valid computing blocks in each computing region, allocate the required computing nodes to each computing region in sequence to obtain the target nodes corresponding to each computing region.

[0046] Here, the target node is one of the plurality of computing nodes, used to perform computational processing on the valid computational blocks (i.e., valid data blocks) in the computational region. Thus, by utilizing each computing node to perform computational processing on the valid computational blocks of each computational region, the target attention processing result for the initial text can be obtained.

[0047] In this way, the proposed solution first divides the initial matrix into multiple computational regions, then counts the number of effective computational blocks in each computational region after masking, and finally allocates the required target nodes to the computational regions based on the number of effective computational blocks. Thus, on the one hand, the proposed solution achieves accurate assessment of computational load through the quantitative statistics of effective computational blocks; on the other hand, it achieves dynamic load balancing of computational nodes in a distributed environment through an allocation strategy based on the quantitative statistics results. This effectively avoids the problem of memory overflow of computational nodes, and also avoids the problem of resource waste where some computational nodes are overloaded and others are idle, thereby improving the overall processing efficiency. In particular, for long sequences, it can significantly improve the efficiency of attention computation.

[0048] Furthermore, this disclosed solution is applicable to any mask processing strategy. In other words, this disclosed solution supports diverse and flexible mask processing schemes, thus improving processing efficiency while also possessing versatility.

[0049] The following are two specific implementation methods for determining the target node: The first implementation method is to prioritize the sorting of computing regions based on the number of valid computing blocks and use a round-robin allocation method to allocate the required computing nodes to each computing region.

[0050] Specifically, in one particular example, the method further includes: Based on the number of valid computing blocks, the multiple computing regions are sorted to obtain a target sequence that represents the priority order of the multiple computing regions.

[0051] Accordingly, the above-described allocation of required computing nodes to each computing region based on the number of effective computing blocks in each computing region, to obtain the target node corresponding to each computing region (e.g., step S104), can specifically include: Based on the priority order of the computing regions in the target sequence, and in accordance with the round-robin allocation method, the required computing nodes are allocated to each computing region in turn to obtain the target nodes corresponding to each computing region.

[0052] For example, in one example, based on the number of valid computing blocks obtained in the aforementioned steps, multiple computing regions are sorted in descending order to obtain a target sequence representing the priority order of the multiple computing regions. For instance, in one example, an index-number of valid blocks mapping table as shown in Figure 3(a) is obtained. The index in this mapping table is used to uniquely identify the computing region, and the number of valid blocks in the mapping table is used to identify the number of valid computing blocks in that computing region. Further, after obtaining this mapping table, based on the priority order of the computing regions in the target sequence, and using a round-robin allocation method, each computing region can be allocated to the corresponding computing node in a round-robin fashion to obtain the target node corresponding to each computing region.

[0053] For example, in one example, attention computation uses a 4-way compute node parallel architecture, that is, it can call 4 GPUs (which can be denoted as GPU0, GPU1, GPU2 and GPU3 respectively). In this case, based on the index-number of valid blocks mapping table shown in Figure 3(a), when allocating each compute region using a round-robin allocation method, the order of the compute node round-robin allocation is as follows: GPU0→GPU1→GPU2→GPU3.

[0054] Furthermore, in one example, to further achieve load balancing among compute nodes, a "snake-like" allocation method can be specifically used when performing round-robin allocation. For example, the "snake-like" round-robin order of compute nodes could be: GPU0→GPU1→GPU2→GPU3→GPU3→GPU2→GPU1→GPU0→GPU1……..

[0055] For example, as shown in Figure 3(b), according to the priority order of the computing regions in the target sequence, the first computing region (e.g., computing region 0, also known as Region0) is assigned to GPU0, that is, the target node of the first computing region is determined to be GPU0; the second computing region (e.g., computing region 1, also known as Region1) is assigned to GPU1, that is, the target node of the second computing region is determined to be GPU1; the third computing region (e.g., computing region 2, also known as Region2) is assigned to GPU2, that is, the target node of the third computing region is determined to be GPU2; the fourth computing region (e.g., computing region 3, also known as Region3) is assigned to GPU3, that is, the target node of the fourth computing region is determined to be GPU3. At this time, each of the four GPUs is assigned a computing region. Further, a second round of polling is performed, allocating the 5th computing region (e.g., computing region 4, denoted as Region 4) to GPU 3, thus determining the target node for the 5th computing region as GPU 3; allocating the 6th computing region (e.g., computing region 5, denoted as Region 5) to GPU 2, thus determining the target node for the 6th computing region as GPU 2; allocating the 7th computing region (e.g., computing region 6, denoted as Region 6) to GPU 1, thus determining the target node for the 7th computing region as GPU 1; allocating the 8th computing region (e.g., computing region 7, denoted as Region 7) to GPU 0, thus determining the target node for the 8th computing region as GPU 0. This process continues, resulting in the allocation outcome shown in Figure 3(b).

[0056] In this way, the proposed solution can orderly allocate computing regions with different computing power requirements to each computing node by prioritizing the computing regions in the target sequence and combining it with a round-robin allocation method. This ensures that high-load computing regions are allocated processing resources first, and the mechanism of region sorting + round-robin allocation achieves a uniform distribution of computing tasks. Thus, on the one hand, accurate assessment of computing load is achieved through the quantitative statistics of effective computing blocks; on the other hand, dynamic load balancing of computing nodes in a distributed environment is achieved through the allocation strategy based on quantitative statistics and round-robin allocation. This effectively avoids the problem of memory overflow of computing nodes, and also avoids the problem of some computing nodes being overloaded while others are idle and wasting resources, thereby improving the overall processing efficiency. Especially for long sequences, it can significantly improve the efficiency of attention computing.

[0057] Furthermore, in a specific example, to further avoid communication overhead, the host node to which the computing region belongs can be further considered during the polling period; specifically, the above-described allocation of the required computing nodes to each computing region based on the priority order of the computing regions in the target sequence and according to the polling allocation method, to obtain the target node corresponding to each computing region, includes: Determine the home node to which each computing region belongs; wherein, the home node is one of the plurality of computing nodes; Based on the priority order of the computing regions in the target sequence and the host node to which each computing region belongs, and in combination with the principle of priority of host node, a round-robin allocation method is used to allocate the required computing nodes to each computing region in order to obtain the target node corresponding to each computing region.

[0058] Here, the home node priority principle means that within the polling cycle of the polling allocation, the computing area that needs to be processed is preferentially allocated to the home node to which the computing area belongs.

[0059] For example, in one instance, the process of allocating the required computing nodes to each computing region based on the priority order of the computing regions in the target sequence and the node to which each computing region belongs, combined with the priority principle of the node to which the node belongs, using a round-robin allocation method to obtain the target node corresponding to each computing region, may specifically include: Determine the j-th computation region that needs node allocation; j is a natural number greater than 1. Determine the home node to which the j-th computing region belongs; wherein the home node is one of the plurality of computing nodes; Identify at least one candidate node that is in an allocatable state during the current polling cycle; the candidate node is one of multiple computing nodes. Based on the home node to which the j-th computing region belongs, and at least one candidate node that is in an allocatable state during the current polling period, the target node corresponding to the j-th computing region is determined.

[0060] For example, in one instance, if at least one candidate node in the allocatable state during the current polling cycle contains the allocatable node to which the j-th computing region belongs, then the allocatable node to which the j-th computing region belongs is taken as the target node corresponding to the j-th computing region.

[0061] Alternatively, if none of the candidate nodes in the allocatable state during the current polling cycle belong to the node to which the j-th computing region belongs, the target node corresponding to the j-th computing region is determined from the at least one candidate node in the allocatable state according to the current polling order.

[0062] For example, continuing with the four GPUs mentioned above (which can be labeled GPU0, GPU1, GPU2, and GPU3 respectively), and continuing with... Figure 2 The example shown illustrates this, and in order to obtain... Figure 2 Following the 16 computation regions shown (which can be denoted as computation region 0, computation region 2, ..., computation region 15 respectively), as follows... Figure 4 As shown, the initial allocation results of the computing regions can be obtained, namely: computing regions 0-3 are assigned to GPU0, that is, GPU0 is the node to which computing regions 0-3 belong; computing regions 4-7 are assigned to GPU1, that is, GPU1 is the node to which computing regions 4-7 belong; computing regions 8-11 are assigned to GPU2, that is, GPU2 is the node to which computing regions 8-11 belong; and computing regions 12-15 are assigned to GPU3, that is, GPU3 is the node to which computing regions 12-15 belong.

[0063] Furthermore, after determining the home node to which each computing region belongs, the polling allocation method can also follow the home node priority principle. Specifically, if the home node of a computing region is currently in an allocable state during the polling period, in other words, if the computing nodes currently in an allocable state include the home node of the computing region that needs to be allocated, then the computing region that needs to be allocated will be preferentially allocated to its home node, and the polling logic will continue to be executed.

[0064] For example, when allocating the j-th compute region, if the node that owns the j-th compute region is in an allocable state within the current polling period, then the j-th compute region is preferentially allocated to its own node. For instance, for the first compute region to be allocated, if GPU0 to GPU3 are all in an allocable state within the current polling period, then the first compute region can be directly allocated to its own node, GPU0. Furthermore, for the second compute region to be allocated, if GPU1 to GPU3 are all in an allocable state within the current polling period, and since there is no owning node among the GPUs in an allocable state, the second compute region is allocated to GPU1 according to the polling order. Similarly, for the third compute region to be allocated, if GPU2 and GPU3 are both in an allocable state within the current polling period, and since there is no owning node among the GPUs in an allocable state, the third compute region is allocated to GPU2 according to the polling order. This process continues until all compute regions have been allocated.

[0065] Thus, this disclosed solution, based on the priority order of computing regions in the target sequence and the home node to which each computing region belongs, uses a round-robin allocation method that combines the home node priority principle. This ensures that task allocation is completed according to the priority order of the number of effective computing blocks, and prioritizes the allocation of computing regions to their home nodes, reducing the communication overhead caused by cross-node data migration. At the same time, the sorting and round-robin allocation logic ensures the load balance of each computing node, avoiding the problem of excessive load on a single node due to concentrated high-load tasks, and further optimizing the execution efficiency and stability of distributed attention processing.

[0066] The second implementation method is to allocate the required computing nodes to each computing region based on the computing resource characteristics of the computing nodes and the number of effective computing blocks in the computing region.

[0067] Figure 5 This is an illustrative flowchart of a distributed attention processing method according to an embodiment of this application. Figure 2 This method can be optionally applied to electronic devices, such as personal computers, servers, and server clusters. It is understood that the above... Figures 1 to 4 The relevant content of the method shown in any of the attached figures can also be applied to this example, and the relevant content will not be described again in this example.

[0068] Furthermore, the method includes at least a portion of the following: For example... Figure 5 As shown, it includes: Step S501: Determine the initial matrix to be processed for attention, and the multiple computing nodes required to perform the attention processing.

[0069] Here, the initial matrix is ​​obtained after feature processing of the initial text.

[0070] Step S502: Divide the initial matrix into blocks to obtain multiple computation regions.

[0071] Step S503: Determine the number of valid computation blocks in each computation region after masking.

[0072] Step S504: Determine the i-th computing region that needs to be assigned nodes.

[0073] Here, i is a natural number greater than 1. Furthermore, the value of i is less than or equal to the number of computational regions.

[0074] Step S505: Determine the node to which the i-th computing region belongs.

[0075] For example, in one example, after performing block processing to obtain each computing region, the required home node can be pre-assigned to each computing region. Furthermore, after determining the home node of each computing region, the data required for subsequent computing processing corresponding to the computing region can be pre-cached in the cache area of ​​the home node, thus providing support for subsequent fast attention processing.

[0076] For example, attention computation uses a 4-way compute node parallel architecture, meaning it can utilize 4 GPUs (which can be denoted as GPU0, GPU1, GPU2, and GPU3 respectively); here, continuing with... Figure 2 The example shown illustrates this, and in order to obtain... Figure 2 Following the 16 computation regions shown (which can be denoted as computation region 0, computation region 2, ..., computation region 15 respectively), as follows... Figure 4 As shown, the initial allocation results of the computing regions can be obtained, namely: computing regions 0-3 are assigned to GPU0, that is, GPU0 is the node to which computing regions 0-3 belong; computing regions 4-7 are assigned to GPU1, that is, GPU1 is the node to which computing regions 4-7 belong; computing regions 8-11 are assigned to GPU2, that is, GPU2 is the node to which computing regions 8-11 belong; and computing regions 12-15 are assigned to GPU3, that is, GPU3 is the node to which computing regions 12-15 belong.

[0077] Furthermore, the initial matrix is ​​masked to obtain the masked computational regions, such as... Figure 4 As shown, to visually illustrate the masking effect, the regions that are not masked and need to participate in the actual attention calculation are highlighted with color, while the masked regions that do not need to participate in the actual attention calculation are marked with white. For example, green is used to mark the effective calculation blocks in calculation regions 0-3; yellow is used to mark the effective calculation blocks in calculation regions 4-7; pink is used to mark the effective calculation blocks in calculation regions 8-11; and purple is used to mark the effective calculation blocks in calculation regions 12-15. The remaining masked invalid calculation parts are marked with white, thus obtaining the following result: Figure 4 The calculation regions shown are after masking.

[0078] Step S506: Based on the current computing resource characteristics of each computing node, determine at least one candidate node for the i-th computing region from the plurality of computing nodes.

[0079] It should be noted that the current computing resource characteristics of the computing node can be specifically understood as the remaining computing resources of the computing node, or the computing resources that the computing node has already occupied, etc. This disclosure does not impose specific restrictions on this, as long as the current computing resource characteristics can directly or indirectly indicate the load status of the computing node.

[0080] Furthermore, in one example, based on the current characteristics of computing resources, nodes with more remaining computing resources or fewer occupied computing resources can be selected from multiple computing nodes as candidate nodes, thus providing support for subsequent balanced allocation of resources.

[0081] Here, the execution order of steps S505 and S506 can be interchanged, and this disclosure does not impose specific restrictions on this.

[0082] Step S507: Based on the home node to which the i-th computing region belongs, and at least one candidate node for the i-th computing region, determine the target node corresponding to the i-th computing region.

[0083] Following the above method, the target nodes corresponding to each computational region can be obtained.

[0084] Thus, this disclosed solution provides a refined allocation scheme for determining the target nodes required for allocating computing regions. In the process of orderly allocation, this refined allocation scheme not only considers the node to which the computing region belongs, but also selects candidate nodes that can be allocated based on the current characteristics of computing resources. In this way, allocating based on the above two dimensions ensures the load balance of each computing node on the one hand, and reduces unnecessary cross-node data communication and communication overhead on the other hand, thereby improving the overall processing efficiency and resource utilization. Especially for long sequences, it can significantly improve the efficiency of attention computation.

[0085] Furthermore, in a specific example, the present disclosure can also determine the current computing resource characteristics of a computing node in the following way, specifically, the method further includes: Step S506-1: Determine the current computing cost of the computing node and the current computing list of the computing node.

[0086] Here, the current computation cost is determined based on the number of valid computation blocks in each computation region allocated to the computation node; the current computation list records the computation regions that need to be processed, in other words, the current computation list records each computation region allocated to the computation node.

[0087] Step S506-2: Determine the current computing resource characteristics of the computing node based on its current computing cost and current computing list.

[0088] In other words, in this example, determining the current computing resource characteristics requires considering not only the number of valid computing blocks in each computing region allocated to the computing node, but also the number of computing regions allocated to the computing node. This multi-dimensional approach to representing the current computing resource characteristics provides quantitative metrics for subsequent load balancing.

[0089] In a specific example, candidate nodes can be determined as follows: From the current computing resource characteristics of each computing node, select nodes that meet one of the following conditions as candidate nodes: Condition 1: Based on the current computing cost, sort the computing nodes, for example, in ascending order, to obtain the cost ranking result; from the cost ranking result, select the computing nodes that meet the cost requirements, for example, select the k computing nodes with the lowest current computing cost, and use the selected computing nodes as candidate nodes.

[0090] Condition 2: Based on the current calculation list, sort the calculation nodes, for example, in ascending order, to obtain the list sorting result; from the list sorting result, select the calculation nodes that meet the list requirements, for example, select the m calculation nodes that contain the fewest calculation areas in the current calculation list, and use the selected calculation nodes as candidate nodes.

[0091] Condition 3: Based on the current computing cost, sort the computing nodes, for example, in ascending order, to obtain a cost sorting result; select computing nodes that meet the cost requirements from the cost sorting result; and based on the current computing list, sort the computing nodes, for example, in ascending order, to obtain a list sorting result; select computing nodes that meet the list requirements from the list sorting result; and from the computing nodes selected based on the above two filtering methods, select p computing nodes with the lowest current computing cost and the fewest computing regions contained in the current computing list as candidate nodes.

[0092] It should be noted that k, m and p are all natural numbers. In practical applications, their values ​​can be set according to actual needs. This application does not impose specific restrictions on this.

[0093] Thus, this disclosed solution provides a refined approach to determine the current computing resource characteristics of computing nodes. By using multiple dimensions, it achieves a comprehensive assessment of the resource status of computing nodes, providing a reliable basis for the selection of candidate nodes. This ensures the rationality of computing area allocation and the stability of load balancing, effectively avoids imbalance in computing task allocation, and further improves the overall processing efficiency and resource utilization of distributed attention computing.

[0094] Figure 6This is a schematic flowchart (Figure 3) illustrating a distributed attention processing method according to an embodiment of this application. This method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It is understood that the above... Figures 1 to 5 The relevant content of the method shown in any of the attached figures can also be applied to this example, and the relevant content will not be described again in this example.

[0095] Furthermore, the method includes at least a portion of the following: For example... Figure 6 As shown, it includes: Step S601: Determine the initial matrix to be processed for attention, and the multiple computing nodes required to perform the attention processing.

[0096] Here, the initial matrix is ​​obtained after feature processing of the initial text.

[0097] Step S602: Divide the initial matrix into blocks to obtain multiple computation regions.

[0098] Step S603: Determine the number of valid computation blocks in each computation region after masking.

[0099] Step S604: Determine the i-th computing region that needs to be assigned nodes.

[0100] Step S605: Determine the node to which the i-th computing region belongs.

[0101] Here, the home node is one of the plurality of computing nodes.

[0102] Step S606: Based on the current computing resource characteristics of each computing node, determine at least one candidate node for the i-th computing region from the plurality of computing nodes.

[0103] Step S607: Determine whether the at least one candidate node contains the node to which the i-th computing region belongs. If yes, proceed to step S608; otherwise, if the at least one candidate node does not contain the node to which the i-th computing region belongs, proceed to step S610.

[0104] Step S608: Assign the node to which the i-th computing region belongs as the target node corresponding to the i-th computing region. Proceed to step S609.

[0105] Step S609: Update the computing resource characteristics of the target node corresponding to the i-th computing region based on the number of valid computing blocks in the i-th computing region.

[0106] For example, in one example, the computing resource characteristics are updated as follows: the number of valid computing blocks in the i-th computing region is directly added to the current computing cost of the target node, and the computing region is added to the current computing list of the target node. In this way, the computing resource characteristics of the complete target node are updated.

[0107] It should be noted that in this allocation method, since the target node corresponding to the i-th computing region is the home node of that computing region, the computing is performed on its home node, and there is no cross-node communication data migration. In other words, unnecessary communication overhead is effectively avoided, laying the foundation for further improving the overall processing efficiency.

[0108] For example, in one example with four compute nodes (e.g., GPUs, specifically GPU0, GPU1, GPU2, and GPU3), when allocating the first compute region (e.g., compute region 0, denoted as Region0), as follows: Figure 4 As shown, the first computing region belongs to GPU0, and the candidate nodes for this first computing region determined based on the current computing resource characteristics include GPU0, GPU1, GPU2, and GPU3. That is, the belonging node of the first computing region is within the range of candidate nodes. Therefore, GPU0 can be used as the target node for the first computing region. Further, if the number of valid data blocks (i.e., valid computing blocks) in the first computing region is 16, the computing resource characteristics of GPU0 can be updated as follows: the current computing cost of GPU0 is updated from 0 to 0 + 16 = 16, and the index of the first computing region is added to the current computing list of GPU0.

[0109] Step S610: Based on the current computing resource characteristics of each candidate node, determine the target node corresponding to the i-th computing region from the at least one candidate node. Proceed to step S611.

[0110] In other words, in this example, if the at least one candidate node does not include the node to which the i-th computing region belongs, that is, during the attention processing, the resource utilization rate of the node to which the i-th computing region belongs is higher than that of other computing nodes, then a computing node can be selected from the candidate nodes as the target node for the i-th computing region.

[0111] For example, in one instance, if the identified candidate node is unique, then that candidate node is selected as the target node corresponding to the i-th computing region. Further, if the candidate nodes are not unique, then based on the current computing resource characteristics of each candidate node, the target node corresponding to the i-th computing region is determined from all candidate nodes. For instance, the candidate node with the lowest current resource utilization rate is selected as the target node. For example, the candidate node with the lowest current computing cost is selected as the target node for the i-th computing region; or, the candidate node with the lowest current computing cost and the fewest current computation lists is selected as the target node for the i-th computing region.

[0112] For example, in one example with four compute nodes (e.g., GPUs, specifically GPU0, GPU1, GPU2, and GPU3), when allocating the second compute region (e.g., compute region 1, denoted as Region1), as follows... Figure 4 As shown, the second computing region belongs to GPU0, but the candidate nodes for this second computing region identified based on the current computing resource characteristics include GPU1, GPU2, and GPU3. This means the node to which the second computing region belongs is not among the candidate nodes. In this case, priority can be assigned based on the current computing resource characteristics of each candidate node. For example, they can be sorted from low to high computing cost, resulting in GPU1 > GPU2 > GPU3. Therefore, GPU1 is determined as the target node for the second computing region. It should be noted that this node selection method is only an example; in actual selection, other factors such as the current computing list can also be considered. This disclosure does not impose specific limitations on this.

[0113] Step S611: Update the computing resource characteristics of the target node corresponding to the i-th computing region based on the number of valid computing blocks in the i-th computing region and the preset communication penalty factor.

[0114] Here, the preset communication penalty factor can be specifically a weighted coefficient. For example, it can specifically refer to a weighted coefficient used to quantify the additional processing cost caused by the communication overhead of cross-node data migration. In this way, the communication overhead can be converted into virtual computing cost. In other words, the scheme disclosed in this disclosure also fully considers the virtual computing cost caused by communication overhead during the allocation process, thus further laying the foundation for achieving load balancing and improving overall processing efficiency.

[0115] In one example, the preset communication penalty factor is usually a preset value greater than 1, such as 1.1, 1.2, etc. In practical applications, it can be set based on the actual communication situation or can be dynamically adjusted. This disclosure does not impose any specific restrictions on this.

[0116] For example, in another example, continuing with the second computing region, if the number of valid computing blocks in the second computing region is 16, and the target node corresponding to the second computing region is GPU1, then the computing resource characteristics of GPU1 can be updated in the following way: multiply the number of valid computing blocks in the second computing region (i.e., 16) by a preset communication penalty factor (e.g., 1.1), and update the computing resource characteristics of GPU1 based on the product. For example, update the current computing cost of GPU1 from the original 0 to 0 + (16 × 1.1) = 17.6, and add the index of the second computing region to the current computing list.

[0117] Thus, this disclosed solution provides a refined allocation scheme that fully considers the home node of the computing region and the current load of all computing nodes. For example, it first selects candidate nodes available for allocation based on the current load of all computing nodes, and then determines whether the candidate nodes include the home node to which the computing region to be allocated belongs. For instance, if the candidate nodes include the home node to which the computing region belongs, the home node is determined as the target node for that computing region, and the computing resource characteristics of the target node are updated according to the number of valid computing blocks in the computing region to be allocated. Alternatively, if the candidate nodes do not include the home node to which the computing region belongs, the target node for the computing region to be allocated is determined from the candidate nodes based on the current computing resource characteristics of each candidate node. By combining the effective computing blocks of the computing region to be allocated with the preset communication penalty factor, the computing resource characteristics of the target node are updated. In this way, on the one hand, different target nodes are used to determine the logic and update the computing resource characteristics to adapt to different computing scenarios. This fully considers the home node without limiting it to the home node. For example, when the home node has sufficient computing resources, the home node is selected first to avoid unnecessary cross-node communication overhead. When the home node is under high load, the target node is selected reasonably based on the load situation. In this way, load balancing is achieved while minimizing communication overhead. On the other hand, the preset communication penalty factor can be used to quantify the cross-node migration cost, realizing dynamic optimization of load balancing and communication efficiency, and further improving the overall stability and execution efficiency of distributed attention computing.

[0118] Furthermore, in a specific example, in both of the above-described implementations for determining the target node, the computational region where the target node performs computational processing can include at least one of the following: The region to which the target node belongs is one of the plurality of computing regions; And non-attributed regions that do not belong to the target node, wherein the non-attributed region is one of the plurality of computing regions.

[0119] In other words, the target node, such as any computing node, includes the region that needs to be processed, including the region belonging to itself and / or the region not belonging to itself.

[0120] For example, in one scenario, the computational region that the target node needs to process includes the computational region belonging to that target node, i.e., the home region. In this case, since the computational region that the target node needs to process is the computational region belonging to that target node—in other words, it is a pre-allocated computational region—and the relevant data of this home region is pre-cached, there is no cross-node data migration and no communication overhead during the processing of this home region.

[0121] For example, in another example, the computational region that the target node needs to process includes computational regions that do not belong to the target node, i.e., non-belonging regions. In this case, since the computational region that the target node needs to process is a computational region that does not belong to the target node, in other words, it is necessary to migrate the relevant data of the non-belonging region from other computational nodes (such as the belonging node to which the non-belonging region belongs) to the target node. Therefore, there is cross-node data migration, and corresponding communication overhead must be incurred.

[0122] For example, in another example, the computational region that the target node needs to perform computational processing includes the region belonging to the target node and the non-governed region not belonging to the target node. It should be noted that the computational region that the target node needs to perform computational processing includes both the region belonging to the target node and the non-governed region not belonging to the target node. Furthermore, through the dynamic allocation strategy of this disclosed solution, an effective balance between local computing load and cross-node communication costs can be achieved, thereby laying the foundation for achieving overall load balancing and rational and effective resource utilization.

[0123] Thus, the proposed solution clearly defines that the processing objects of the target node can include two types of computing regions: computing regions belonging to the target node (also known as the home region) and computing regions not belonging to the target node (also known as the non-home region). This enables flexible allocation of computing resources, thereby effectively improving the load balancing and resource utilization of the attention computing mechanism.

[0124] Furthermore, in a specific example, the method further includes: When the computational region for processing at the target node includes both the home region and the non-home region, the first and second processes are executed in parallel using the target node.

[0125] Here, the first processing can be specifically used to perform attention processing based on valid computation blocks in the home region to obtain the attention processing result corresponding to the home region; the second processing is used to obtain valid data of valid computation blocks in the non-home region from the home node to which the non-home region belongs, and store the obtained valid data in the communication buffer. In other words, the target node processes in parallel: attention processing for the home region and data writing operations for the non-home region.

[0126] Here, the communication buffer can be a pre-allocated buffer area for the target node, and its size can be adaptively configured according to the expected amount of data in the effective computing blocks of the non-home region. This disclosure does not impose specific restrictions on the size of the communication buffer or the adaptive configuration method.

[0127] In other words, in this example, the communication buffer is specifically a pre-allocated buffer area used to temporarily cache valid data of valid computation blocks in the non-home region obtained from the home node to which the non-home region belongs. Furthermore, in one example, the communication buffer can also achieve data isolation between the valid data of valid computation blocks in the non-home region (i.e., external data) and the valid data of valid computation blocks corresponding to the home region (i.e., local data), thus effectively avoiding performance loss caused by read / write conflicts.

[0128] For example, taking GPU3 out of the four GPUs mentioned above as an example, after allocation, the computational regions that GPU3 needs to process include computational region 14 (the assigned region), computational region 3 (a non-assigned region, belonging to GPU0), computational regions 5 and 7 (both non-assigned regions, belonging to GPU1), and computational region 10 (a non-assigned region, belonging to GPU2). At this point, for GPU3, after starting computation, the following processes can be performed in parallel: The first process involves using GPU3 to perform Attention calculations on the local computing region 14 to obtain the attention processing results corresponding to computing region 14. The second process involves using GPU3 to obtain valid data of valid computing blocks in computing region 3 from GPU0, valid data of valid computing blocks in computing region 5 and computing region 7 from GPU1, and valid data of valid computing blocks in computing region 10 from GPU2. The obtained valid data is then stored in the communication buffer. In other words, GPU3 performs data write operations on the following data: valid data of valid computing blocks in computing region 3, valid data of valid computing blocks in computing region 5, valid data of valid computing blocks in computing region 7, and valid data of valid computing blocks in computing region 10.

[0129] It should be noted that in practical applications, the disclosed solution can also fix the upper limit of the number of computing regions that a computing node performs Attention calculations each time. This is to prevent the problem of video memory OOM caused by a single computing node, such as a GPU, being allocated too many computing regions under extreme load conditions.

[0130] For example, in one instance, if the number of computation regions to be processed in the first step exceeds the maximum number of computation regions that a computation node can process for attention each time, then all computation regions requiring attention calculation can be processed in batches during the first step. In the batch processing flow within the first step, if the data writing operations for non-home regions have been completed in the second step, then any of the following methods can be used for processing: Method 1: Add the non-home regions that have completed the write operation to the batch processing flow so that the computing nodes can complete all the required Attention calculations in batches.

[0131] Method 2: After determining that the attention calculation for all home regions has been completed, the attention calculation for non-home regions is performed. If the number of non-home regions has exceeded the upper limit of the number of regions to be calculated for each attention calculation, then batch processing continues until all the attention calculations that need to be completed are completed.

[0132] Thus, this disclosed solution utilizes the target node to execute the first and second processing in parallel when the computational region requiring computational processing includes both the home region and the non-home region. This achieves efficient utilization of GPU resources and minimizes additional communication latency while achieving load balancing. Furthermore, by storing the valid data of valid computational blocks in the non-home region obtained from the home node to which the non-home region belongs in a storage buffer, it also achieves data isolation between the valid data of valid computational blocks in the non-home region and the attention processing results corresponding to the home region, effectively avoiding performance loss caused by read-write conflicts, thereby laying the foundation for further improving overall processing efficiency.

[0133] Furthermore, in another specific example, the method further includes: If the target node's communication buffer contains all the valid data required for attention processing of the non-home region, and the attention processing of the home region is completed, then the target node is used to perform attention processing based on the valid computation blocks in the non-home region to obtain the attention processing result corresponding to the non-home region. Using the target node, the attention processing results corresponding to the non-home region are sent to the home node to which the non-home region belongs.

[0134] For example, in one example, continuing with GPU3 as described above, it is confirmed that the communication buffer contains all the valid data required for attention processing of the non-home regions. For instance, the buffer contains valid data of valid computation blocks belonging to computation region 3 of GPU0, valid data of valid computation blocks belonging to computation regions 5 and 7 of GPU1, and valid data of valid computation blocks belonging to computation region 10 of GPU2. Simultaneously, after confirming that attention processing of the home region (i.e., computation region 14) is complete, attention is performed based on the valid data of valid computation blocks in the non-home regions (i.e., computation regions 3, 5, 7, and 10). The processing involves using GPU3 to perform attention calculations on computation regions 3, 5, 7, and 10 to obtain the attention processing results for each non-home region. Further, after the calculation is complete, the attention processing results for each non-home region are sent to the home node to which the non-home region belongs. For example, using GPU3, the attention processing results for computation region 3 are sent to its home node GPU0, the attention processing results for computation region 5 and 7 are sent to their home nodes GPU1, and the attention processing results for computation region 10 are sent to its home node GPU2.

[0135] Thus, this disclosed solution provides a refined scheme for computing nodes to perform attention processing (or attention computation). In this scheme, computing nodes prioritize performing attention processing on their home regions. After completing attention processing on all home regions, they then perform attention processing on non-home regions. That is, after confirming that the communication buffer contains all the valid data required for attention processing on non-home regions and that attention processing on home regions is complete, attention processing is performed on the valid computing blocks in non-home regions to obtain the attention processing result for the non-home regions. Then, the attention processing result for the non-home regions is sent to the home node to which the non-home regions belong through the target node. In this way, through the dual-condition confirmation mechanism, not only can computing resources, such as GPU resources, be used efficiently, but also load balancing can be achieved while minimizing additional communication latency. This further improves the execution reliability and efficiency of the distributed attention mechanism, thus laying the foundation for further improving overall processing efficiency.

[0136] The following detailed description of the present invention is provided with specific examples. These examples offer a novel load-balanced distributed attention mechanism computation method that considers Mask sparsity. This enables dynamic load balancing among GPUs, significantly mitigating the problems of uneven GPU load and excessively long waiting times during distributed attention mechanism computation. Specifically, Figure 7(a) is a schematic flowchart of the implementation of a distributed attention processing method according to an embodiment of the present application in a specific example; Figure 7(b) is a schematic flowchart of the allocation process of a distributed attention processing method according to an embodiment of the present application in a specific example; Figure 7(c) is a schematic diagram of the execution scheduling of overlapping computation and communication in a distributed attention processing method according to an embodiment of the present application in a specific example. As shown in Figures 7(a), 7(b), and 7(c), the distributed attention processing method in this example can be implemented through the following steps, specifically including: Phase 1, Calculation of Region Division and Valid Block Statistics, specifically includes the following steps: Step S701: Divide the matrix (i.e. the initial matrix) that needs to be calculated for Attention into blocks to obtain multiple calculation regions.

[0137] For example, the matrix QK that needs to be calculated for attention... T The computation is divided into multiple computational regions of predetermined size, each a smaller computational unit than a slice of the original Q, K, V matrix. For example, in one example, there are four callable computational nodes, and the matrix QK... TThe size is 16×16. As shown in Figure 7(b), the 16×16 matrix can be divided into 16 computational regions by columns, and the size of each computational region is 16×1.

[0138] It should be noted that, as shown in Figure 7(b), each small square can further represent a matrix. In other words, the present invention does not impose specific restrictions on the size of the initial matrix, the preset size, the number of calculation areas, etc. In practical applications, these can be set based on actual needs.

[0139] Step S702: Mask the matrix (i.e., the initial matrix) to be used for Attention calculation, for example, to obtain the matrix (QK). T +Mask), where Mask represents the mask matrix. Further, the computation regions after masking are obtained, and the number of effective calculation blocks in each region after masking is counted.

[0140] For example, counting data blocks in the computational region (also represented by a matrix) whose element values ​​are not specified (-∞ or 0) after statistical masking represents the number of valid computational blocks. These valid computational blocks are the data blocks that actually need to participate in the computation. For instance, in one example, a Boolean matrix can be used to count the number of valid computational blocks. If the Boolean matrix of a data block is 1, then the data block is considered a valid computational block and needs to participate in the subsequent actual computation; otherwise, the data block is considered an invalid computational block and does not need to participate in the subsequent actual computation. Thus, the index-number of valid blocks mapping table shown in Figure 7(b) is obtained. Here, the index uniquely indicates a computational region, and the number of valid blocks refers to the number of valid computational blocks in that computational region.

[0141] Phase Two: Dynamic Task Allocation Based on Computing Resources and Preset Communication Penalty Factors. For example, in this case, a greedy strategy and communication penalties are combined to perform dynamic task scheduling to achieve load balancing among GPUs. This includes the following steps: Step S703: Initialize the computing resources of each computing node that can be called, such as the GPU. For example, initialize the computing cost and the computing region index list (i.e., the computing list). For example, initialize the computing cost of each GPU to 0 and the computing region index list to empty.

[0142] Step S704: Traverse each calculation region and determine whether all calculation regions have been processed; if yes, end the processing and proceed to step S712; otherwise, proceed to step S705.

[0143] Step S705: Determine the current computing region to be processed, for example, denoted as the i-th computing region; and determine the location (or source GPU) of the i-th computing region, that is, determine which GPU the data of the i-th computing region is stored in, and denot the location (i.e., the location node mentioned above) of the i-th computing region as the GPU. orig-i .

[0144] Step S706: Identify one or a group of GPUs among all GPUs that currently meet the computing resource requirements. For example, identify the GPU or group of GPUs with the lowest current computing cost and denote it as the set of GPUs. min_cost-i .

[0145] Step S707: Determine the source GPU, that is, the GPU to which the i-th computing region belongs. orig-i If the source GPU to which the i-th computing region belongs is not in the current set of GPUs with the lowest computing cost, proceed to step S708; otherwise, proceed to step S710.

[0146] Step S708: Prioritize allocating the i-th computing region to its home GPU. orig-i Proceed to step S709.

[0147] Step S709: Update the GPU to which the i-th computing region belongs based on the number of valid computing blocks in the i-th computing region. orig-i The computing resources are depleted. The process then returns to step S704.

[0148] In other words, assuming the source GPU is among the GPUs with the lowest current computational cost, i.e., GPUs... orig-i ∈GPU min_cost-i If so, the i-th computing region will be preferentially allocated to its home GPU. orig-i This avoids cross-node communication. Furthermore, the number of valid compute blocks in the i-th compute region can be used to determine the location of the GPU. orig-i Update the computing resources, for example, update to: Cost (GPU) orig-i )←[Cost(GPU orig [) + the number of effective computation blocks in the i-th computation region, meaning the number of effective computation blocks in the i-th computation region can be directly added to the GPU. orig-i This is used to update the computational cost within the computational cost. Simultaneously, the index of the i-th computational region is added to the GPU. orig-i The list of computed region indexes.

[0149] For example, in one example, as shown in Figure 7(b), when allocating the first computing region (e.g., computing region 0, denoted as Region0), the source GPU of this first computing region is GPU0, the number of effective computing blocks is 16, and in the current state, the set of GPUs with the lowest computing cost includes GPU0, GPU1, GPU2, and GPU3; that is, for the first computing region, its source GPU is in the set of GPUs with the lowest current computing cost. At this time, GPU0 can be directly used as the target node corresponding to the first computing region. In other words, the first computing region is allocated to GPU0 to avoid communication overhead. Furthermore, the computing resources of GPU0 can be updated as follows: the current computing cost of GPU0 is updated from 0 to 0 + 16 = 16. At the same time, the index of the first computing region is added to the computing region index list of GPU0.

[0150] Step S710: Assign the i-th computing region to the GPU in the GPU set that has the lowest current computing cost and the shortest list of computing region indices, for example, let it be GPU _i ... dest-i Proceed to step S711.

[0151] Step S711: Based on the number of valid computing blocks in the i-th computing region and the preset communication penalty factor, update the non-home GPU allocated to the i-th computing region. dest-i The computing resources are used up. Then return to step S704.

[0152] In other words, if the source GPU is not among the GPUs with the lowest current computational cost, that is, if the GPU... orig-i Not a GPU min_cost-i Then, the i-th computing region is assigned to the GPU in the GPU set that has the lowest current computing cost and the shortest current computing region index list (for example, it can be denoted as GPU). dest-i This achieves load balancing within the GPU. dest-i It is a non-owned node of the i-th computational region.

[0153] Furthermore, the GPU allocated to the i-th computing region can be determined based on the number of valid computing blocks in the i-th computing region and a preset communication penalty factor (e.g., denoted as α). dest-i Update the computing resources, for example, update to: Cost (GPU) dest-i )←[Cost(GPU dest-i The expression is: ) + the number of effective computation blocks in the i-th computation region × α. In other words, the number of effective computation blocks in the i-th computation region can be multiplied by a preset communication penalty factor, and the product result can be accumulated in the GPU. orig-iThis is used to update the computational cost within the computational cost. Simultaneously, the index of the i-th computational region is added to the GPU. orig-i The list of computed region indexes.

[0154] For example, in one example, as shown in Figure 7(b), when allocating the second computing region (e.g., computing region 1, denoted as Region1), the source GPU of this second computing region is GPU0, the number of effective computing blocks is 16, and in the current state, the set of GPUs with the lowest computing cost includes GPU1, GPU2, and GPU3; that is, for the second computing region, its source GPU is not in the set of GPUs with the lowest current computing cost. In this case, the GPU with the lowest current computing cost and the shortest computing region index list can be selected from the set of GPUs with the lowest computing cost as the target GPU corresponding to the second computing region. For example, GPU1 can be selected as the target GPU corresponding to the second computing region; in other words, the second computing region is allocated to GPU1. Furthermore, the computing resources of GPU1 can be updated in the following way: Multiply the number of valid computing blocks in the second computing region (i.e., 16) by a preset communication penalty factor (e.g., α is 1.1), and update the computing cost of GPU1 based on the product. For example, update the current computing cost of GPU1 from 0 to 0 + (16 × 1.1) = 17.6. At the same time, add the index of the second computing region to the computing region index list of GPU1.

[0155] It should be noted that this preset communication penalty factor can be used to convert communication overhead into virtual computing cost, thereby guiding the allocator to make a trade-off between load balancing and minimizing communication.

[0156] Step S712: Output the allocation result.

[0157] For example, after traversing all computing regions, the allocation result shown in Figure 7(b) is obtained. At this point, computing regions 0, 9, and 13 are allocated to GPU0, whose current computing cost is 27; computing regions 1, 11, and 15 are allocated to GPU1, whose current computing cost is 27.5; computing regions 2, 4, 6, 8, and 12 are allocated to GPU2, whose current computing cost is 25.9; and computing regions 3, 5, 7, 10, and 14 are allocated to GPU3, whose current computing cost is 26. It can be seen that load balancing is effectively achieved.

[0158] Phase 3, as shown in Figure 7(c), is the execution scheduling of overlapping Attention computation and communication.

[0159] It should be noted that in the scheme of this application, the upper limit of the number of computing regions for each GPU to perform Attention calculation each time can be fixed. In this case, when a certain GPU is assigned to a large number of computing regions, the scheme of this application can adopt the computing communication overlap method shown in Figure 7(c) to handle the problem, thereby avoiding the GPU memory OOM caused by a certain GPU being assigned to a large number of computing regions in extreme cases. At the same time, the communication overhead is masked by the computing communication overlap method.

[0160] In other words, this disclosed solution employs an execution scheduling strategy that overlaps attention computation and communication, thereby minimizing the communication overhead between GPUs caused by task reallocation. Specifically, this may include the following processing methods: Method 1: Communication Buffer Allocation; A fixed-size communication buffer (also called a communication buffer) is pre-allocated on each GPU to temporarily store valid data (such as Q, K, and V data of the computation region) and computation results transferred across GPUs. In other words, it is used to store valid data from other computation regions not belonging to that GPU, as well as the computation results of Attention calculations.

[0161] Method 2: Local computation priority; At the start of computation, each GPU prioritizes the execution of Attention calculations in computation regions whose computation region index list belongs to local storage, that is, it prioritizes the execution of Attention calculations in computation regions whose location is the GPU itself.

[0162] For example, as shown in Figure 7(c), taking GPU3 out of the four GPUs mentioned above as an example, after allocation, the computational regions for which GPU3 needs to perform Attention calculations include computational region 14 (the locally stored region, i.e., the region it belongs to as mentioned above), computational region 3 (the non-locally stored region, i.e., the non-belonging region mentioned above, whose location is GPU0), computational regions 5 and 7 (the non-locally stored regions, i.e., the non-belonging regions mentioned above, whose location is GPU1), and computational region 10 (the non-locally stored region, i.e., the non-belonging region mentioned above, whose location is GPU2). At this time, for GPU3, after starting computation, GPU3 prioritizes performing Attention calculations on computational region 14, which belongs to the locally stored region.

[0163] Method 3: Remote Data Transfer; For each GPU, the parallel (overlapping) processing of Attention calculations with the local storage computing area involves: retrieving valid data (Q, K, V data of the computing area) allocated to its own computing area from other GPUs and writing it into its own communication buffer. In other words, the parallel (overlapping) processing of Attention calculations with the local storage computing area involves: the data writing operation of valid data from other computing areas that are not local but allocated to the local computing area.

[0164] For example, as shown in Figure 7(c), continuing with GPU3 as an example, GPU3 performs Attention calculations on local computing regions. For instance, while performing Attention calculations on computing region 14, it performs the following data write operations: writing valid data for valid computing blocks in computing region 3, writing valid data for valid computing blocks in computing region 5, writing valid data for valid computing blocks in computing region 7, and writing valid data for valid computing blocks in computing region 10. In other words, while performing Attention calculations on computing region 14, GPU3 can also obtain valid data for valid computing blocks in computing region 3 from GPU0, valid data for valid computing blocks in computing region 5 and valid data for valid computing blocks in computing region 7 from GPU1, and valid data for valid computing blocks in computing region 10 from GPU2, and store the obtained valid data in the communication buffer.

[0165] Method 4: Remote Computation and Result Feedback; When the local computing region's Attention calculation is complete, and the communication buffer is full or has acquired valid data from all valid computing blocks in the remote computing regions (or remote computing regions) that require its own computation, the GPU switches to execute Attention calculations for the acquired remote computing regions. After the calculation is complete, the computation results of the remote computing regions are fed back to the communication buffer of the respective remote computing region's home location.

[0166] For example, as shown in Figure 7(c), continuing with GPU3 as an example, when the Attention calculation of computing region 14 belonging to GPU3 is completed, and the communication buffer is full or has obtained valid data of valid computing blocks in all the off-site computing regions that need to be calculated (i.e., computing regions 3, 5, 7, and 10), GPU3 switches to execute Attention calculation for the obtained off-site computing regions to obtain the attention processing results corresponding to computing regions 3, 5, 7, and 10. Further, the attention processing results of the off-site computing regions are sent back to the communication buffer of the region to which each computing region belongs. For example, GPU3 sends the attention processing results of computing region 3 belonging to GPU0 back to GPU0, the attention processing results of computing regions 5 and 7 belonging to GPU1 back to GPU1, and the attention processing results of computing region 10 belonging to GPU2 back to GPU2.

[0167] Method 5: Batch processing; when the communication buffer cannot hold all the valid data acquired from the remote computing area at once, the computing and communication process can be divided into multiple iterations.

[0168] It should be noted that each GPU can execute the same overlapping computation and communication process. Once all GPUs have completed their respective tasks, the entire distributed attention computation is complete.

[0169] In summary, this disclosed solution has the following advantages: First, enhanced versatility and flexibility: Compared to existing technologies, such as Striped Attention and DistFlashAttention, which only support Causal Mask, this disclosed solution can combine FlashMask technology and an effective block statistics allocation strategy to support more flexible and diverse mask representation methods, such as Full Attention, Causal Mask, Block-Local Attention, and Prefix-Tuning Mask, thus offering greater versatility and flexibility.

[0170] Second, efficient load balancing and performance improvement: This disclosed solution dynamically allocates tasks based on computational cost and preset communication penalty factors, effectively solving the problem of uneven load among GPUs in distributed attention computation, significantly reducing GPU idle waiting time, and significantly improving the end-to-end performance of distributed training.

[0171] Third, superior communication efficiency: Addressing the issue of fragmented communication caused by the fine-grained segmentation of attention calculation in Magi Attention technology, this disclosed solution employs an adjustable-granularity computation region partitioning. This allows for flexible adjustment of communication granularity, minimizing fragmented communication and significantly improving communication efficiency. Furthermore, by introducing a preset communication penalty factor α, an optimal trade-off is achieved between load balancing and communication overhead.

[0172] Fourth, resource security and stability: By limiting the upper limit of the computing area for each GPU to perform attention calculations each time, the GPU memory overflow (OOM) problem caused by a GPU being assigned too many tasks under extreme load conditions is effectively avoided, thus enhancing the stability and robustness of the system.

[0173] Fifth, latency masking: By adopting a scheduling mechanism that overlaps computation and communication, the latency of cross-GPU data communication caused by task reallocation is effectively masked, further improving the actual running efficiency.

[0174] This disclosure also provides a distributed attention processing device, such as Figure 8 As shown, the device includes: The preprocessing unit 801 is used to determine an initial matrix to be subjected to attention processing, and multiple computing nodes required to perform the attention processing; wherein, the initial matrix is ​​obtained after feature processing of the initial text; the initial matrix is ​​divided into blocks to obtain multiple computing regions; and the number of effective computing blocks in each computing region after masking processing is determined. The resource allocation unit 802 is used to allocate the required computing nodes to each computing region in sequence based on the number of valid computing blocks in each computing region, so as to obtain the target node corresponding to each computing region; wherein, the target node is one of the plurality of computing nodes, used to perform computing processing on the valid computing blocks in the computing region.

[0175] In a specific example of the disclosed solution, the resource allocation unit is specifically used for: Determine the i-th computational region that needs node allocation; i is a natural number greater than 1. Determine the home node to which the i-th computing region belongs; wherein the home node is one of the plurality of computing nodes; Based on the current computing resource characteristics of each computing node, at least one candidate node for the i-th computing region is determined from the plurality of computing nodes; Based on the home node to which the i-th computing region belongs, and at least one candidate node for the i-th computing region, the target node corresponding to the i-th computing region is determined.

[0176] In a specific example of the scheme disclosed herein, the preprocessing unit is further configured to: Determine the current computing cost of the computing node and the current computing list of the computing node; wherein, the current computing cost is determined based on the number of valid computing blocks in each computing region allocated to the computing node; the current computing list records the computing regions that need to be processed. Based on the current computing cost and current computing list of the computing node, determine the current computing resource characteristics of the computing node.

[0177] In a specific example of the disclosed solution, the resource allocation unit is specifically used for: If at least one candidate node includes the node to which the i-th computing region belongs, then the node to which the i-th computing region belongs shall be taken as the target node corresponding to the i-th computing region. Update the computing resource characteristics of the target node corresponding to the i-th computing region based on the number of valid computing blocks in the i-th computing region.

[0178] In a specific example of the disclosed solution, the resource allocation unit is specifically used for: If none of the candidate nodes belong to the i-th computing region, the target node corresponding to the i-th computing region is determined from the at least one candidate node based on the current computing resource characteristics of each candidate node. The computing resource characteristics of the target node corresponding to the i-th computing region are updated based on the number of valid computing blocks in the i-th computing region and the preset communication penalty factor.

[0179] In a specific example of the disclosed scheme, wherein, The preprocessing unit is further configured to sort the plurality of computing regions based on the number of effective computing blocks to obtain a target sequence characterizing the priority order of the plurality of computing regions; The resource allocation unit is specifically used to allocate the required computing nodes to each computing region in turn according to the priority order of the computing regions in the target sequence and in a round-robin allocation method, so as to obtain the target nodes corresponding to each computing region.

[0180] In a specific example of the disclosed solution, the resource allocation unit is specifically used for: Determine the home node to which each computing region belongs; wherein, the home node is one of the plurality of computing nodes; Based on the priority order of the computing regions in the target sequence and the home node to which each computing region belongs, and in conjunction with the home node priority principle, a round-robin allocation method is used to allocate the required computing nodes to each computing region in order to obtain the target node corresponding to each computing region; wherein, the home node priority principle means that within the round-robin allocation cycle, the computing region is preferentially allocated to the home node to which the computing region belongs.

[0181] In a specific example of the scheme disclosed herein, the computational region where the target node performs computational processing includes at least one of the following: The region to which the target node belongs is one of the plurality of computing regions; The non-attributed region that does not belong to the target node is one of the plurality of computing regions.

[0182] In a specific example of the present disclosure, the device further includes: an attention processing unit; wherein, The attention processing unit is used to perform first processing and second processing using the target node when the computational region for computational processing at the target node includes both the home region and the non-home region. The first process is used to perform attention processing based on the valid computation blocks in the home region to obtain the attention processing result corresponding to the home region; the second process is used to obtain the valid data of the valid computation blocks in the non-home region from the home node to which the non-home region belongs, and store the obtained valid data in the communication buffer.

[0183] In a specific example of the scheme disclosed herein, the attention processing unit is further configured to: If it is confirmed that the target node's communication buffer contains all the valid data required for attention processing of the non-home region, and that attention processing of the home region is completed, attention processing is performed on the target node based on the valid computation blocks in the non-home region to obtain the attention processing result corresponding to the non-home region. Using the target node, the attention processing results corresponding to the non-home region are sent to the home node to which the non-home region belongs.

[0184] For a description of the specific functions and examples of each unit of the apparatus in this disclosure embodiment, please refer to the relevant descriptions of the corresponding steps in the above method embodiments, which will not be repeated here.

[0185] This disclosure also provides a distributed attention processing system, such as Figure 9 As shown, the system includes: Multiple compute nodes 901; Scheduling node 902 is used to determine the initial matrix to be processed for attention, and the multiple computing nodes required to perform the attention processing; wherein, the initial matrix is ​​obtained after feature processing of the initial text; the initial matrix is ​​divided into blocks to obtain multiple computing regions; the number of valid computing blocks in each computing region after masking is determined; based on the number of valid computing blocks in each computing region, the required computing nodes are allocated to each computing region in sequence to obtain the target node corresponding to each computing region; wherein, the target node is one of the multiple computing nodes, used to perform computation processing on the valid computing blocks in the computing region.

[0186] It should be noted that the scheduling node can specifically be the distributed attention processing device described above. In other words, its specific functions and examples can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0187] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0188] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0189] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0190] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded into random access memory (RAM) 1003 from storage unit 1008. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0191] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0192] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the distributed attention processing method. For example, in some embodiments, the distributed attention processing method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the distributed attention processing method described above can be performed. Alternatively, in other embodiments, computing unit 1001 may be configured to perform a distributed attention processing method by any other suitable means (e.g., by means of firmware).

[0193] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0194] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0195] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0196] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0197] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0198] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0199] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0200] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A distributed attention processing method, comprising: Determine the initial matrix to be subjected to attention processing, and the multiple computing nodes required to perform the attention processing; wherein, the initial matrix is ​​obtained after feature processing of the initial text; The initial matrix is ​​divided into blocks to obtain multiple computational regions; Determine the number of valid computation blocks in each computation region after masking; Based on the number of valid computing blocks in each computing region, the required computing nodes are allocated to each computing region in sequence to obtain the target node corresponding to each computing region; wherein, the target node is one of the plurality of computing nodes, used to perform computing processing on the valid computing blocks in the computing region.

2. The method according to claim 1, wherein, The process of allocating the required computing nodes to each computing region based on the number of valid computing blocks in each computing region, to obtain the target nodes corresponding to each computing region, includes: Determine the i-th computational region that needs node allocation; i is a natural number greater than 1. Determine the home node to which the i-th computing region belongs; wherein the home node is one of the plurality of computing nodes; Based on the current computing resource characteristics of each computing node, at least one candidate node for the i-th computing region is determined from the plurality of computing nodes; Based on the home node to which the i-th computing region belongs, and at least one candidate node for the i-th computing region, the target node corresponding to the i-th computing region is determined.

3. The method according to claim 2, further comprising: Determine the current computing cost of the computing node and the current computing list of the computing node; wherein, the current computing cost is determined based on the number of valid computing blocks in each computing region allocated to the computing node; the current computing list records the computing regions that need to be processed. Based on the current computing cost and current computing list of the computing node, determine the current computing resource characteristics of the computing node.

4. The method according to claim 2, wherein, The step of determining the target node corresponding to the i-th computing region based on the belonging node to which the i-th computing region belongs and at least one candidate node for the i-th computing region includes: If at least one candidate node includes the home node to which the i-th computing region belongs, then the home node to which the i-th computing region belongs shall be taken as the target node corresponding to the i-th computing region. Update the computing resource characteristics of the target node corresponding to the i-th computing region based on the number of valid computing blocks in the i-th computing region.

5. The method according to claim 2, wherein, The step of determining the target node corresponding to the i-th computing region based on the belonging node to which the i-th computing region belongs and at least one candidate node for the i-th computing region includes: If none of the candidate nodes belong to the i-th computing region, the target node corresponding to the i-th computing region is determined from the at least one candidate node based on the current computing resource characteristics of each candidate node. The computing resource characteristics of the target node corresponding to the i-th computing region are updated based on the number of valid computing blocks in the i-th computing region and the preset communication penalty factor.

6. The method according to claim 1, further comprising: Based on the number of effective computing blocks, the multiple computing regions are sorted to obtain a target sequence that represents the priority order of the multiple computing regions. The process of allocating required computing nodes to each computing region based on the number of valid computing blocks in each region, to obtain the target node corresponding to each computing region, includes: Based on the priority order of the computing regions in the target sequence, and in accordance with the round-robin allocation method, the required computing nodes are allocated to each computing region in turn to obtain the target nodes corresponding to each computing region.

7. The method according to claim 6, wherein, The process of allocating the required computing nodes to each computing region in a round-robin fashion, based on the priority order of the computing regions in the target sequence, to obtain the target nodes corresponding to each computing region, includes: Determine the home node to which each computing region belongs; wherein, the home node is one of the plurality of computing nodes; Based on the priority order of the computing regions in the target sequence and the home node to which each computing region belongs, and in conjunction with the home node priority principle, a round-robin allocation method is used to allocate the required computing nodes to each computing region in order to obtain the target node corresponding to each computing region; wherein, the home node priority principle means that within the round-robin allocation cycle, the computing region that needs to be processed is preferentially allocated to the home node to which the computing region belongs.

8. The method according to claim 4, 5, or 7, wherein, The computational region where the target node performs computational processing includes at least one of the following: The region to which the target node belongs is one of the plurality of computing regions; The non-attributed region that does not belong to the target node is one of the plurality of computing regions.

9. The method according to claim 8, further comprising: When the computational region for processing at the target node includes both the home region and the non-home region, the first and second processing are performed using the target node. The first process is used to perform attention processing based on the valid computation blocks in the home region to obtain the attention processing result corresponding to the home region; the second process is used to obtain the valid data of the valid computation blocks in the non-home region from the home node to which the non-home region belongs, and store the obtained valid data in the communication buffer.

10. The method of claim 9, further comprising: If the target node's communication buffer contains all the valid data required for attention processing of the non-home region, and the attention processing of the home region is completed, then the target node is used to perform attention processing based on the valid computation blocks in the non-home region to obtain the attention processing result corresponding to the non-home region. Using the target node, the attention processing results corresponding to the non-home region are sent to the home node to which the non-home region belongs.

11. A distributed attention processing device, comprising: A preprocessing unit is used to determine an initial matrix to be subjected to attention processing, and multiple computing nodes required to perform the attention processing; wherein, the initial matrix is ​​obtained after feature processing of the initial text; the initial matrix is ​​divided into blocks to obtain multiple computing regions; and the number of effective computing blocks in each computing region after masking processing is determined. The resource allocation unit is used to allocate the required computing nodes to each computing region in sequence based on the number of valid computing blocks in each computing region, so as to obtain the target node corresponding to each computing region; wherein, the target node is one of the plurality of computing nodes, and is used to perform computing processing on the valid computing blocks in the computing region.

12. A distributed attention processing system, comprising: Multiple computing nodes; A scheduling node is used to determine the initial matrix to be processed for attention, and the multiple computing nodes required to perform the attention processing; wherein the initial matrix is ​​obtained after feature processing of the initial text; the initial matrix is ​​divided into blocks to obtain multiple computing regions; the number of valid computing blocks in each computing region after masking is determined; based on the number of valid computing blocks in each computing region, the required computing nodes are allocated to each computing region in sequence to obtain the target node corresponding to each computing region; wherein the target node is one of the multiple computing nodes, used to perform computation processing on the valid computing blocks in the computing region.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images