A load balancing assignment system for attention computation

Abstract

The application relates to the technical field of GPU task scheduling, in particular to a load balancing and distribution system for attention calculation, which determines a target key value sequence length threshold for splitting from multiple dimensions according to an average key value sequence length, a key value sequence length variance value and a memory pressure state value corresponding to a calculation task set, avoids excessive splitting or insufficient splitting caused by a fixed splitting threshold, improves the reliability of calculation task splitting, and avoids memory overflow in combination with the memory pressure state value, can adapt to task differences of different users, meets the multi-user concurrent scheduling demand, and further improves the load balancing and distribution effect of the calculation task, and further improves the processing efficiency of the overall calculation task.

Description

Technical Field

[0001] This invention relates to the field of GPU task scheduling technology, and in particular to a load balancing distribution system for attention computing. Background Technology

[0002] Currently, in the application scenarios of large language models, there is often a need for multiple users to perform attention calculation tasks in parallel, which in turn requires scheduling multiple calculation tasks to different basic computing units of the GPU chip for parallel processing.

[0003] To avoid excessive computation in a single computing task, existing technologies typically divide computing tasks into multiple subtasks for scheduling. However, when dividing computing tasks, existing technologies usually use a fixed key-value sequence length. This fixed key-value sequence length often fails to meet the division requirements of a set of computing tasks. For example, when the key-value sequence length is set too small, over-segmentation may occur, leading to increased integration costs. Conversely, when the key-value sequence length is set too large, under-segmentation may occur, resulting in memory overflow and other issues.

[0004] Therefore, how to improve the load balancing of computing tasks in attention-based computing scenarios, and thus improve the overall processing efficiency of computing tasks, has become an urgent problem to be solved. Summary of the Invention

[0005] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows: A load balancing distribution system for attention computing, the system comprising: M basic computing units, a processor, and a memory storing a computer program, wherein M is a positive integer, and when the computer program is executed by the processor, the following steps are implemented: S101, based on the acquired set of computing tasks, determine the average key-value sequence length, key-value sequence length variance, and memory pressure status value corresponding to the set of computing tasks, wherein the set of computing tasks includes computing tasks corresponding to N users, where N is a positive integer; S102, determine the target key-value sequence length threshold corresponding to the computing task set based on the average key-value sequence length, the key-value sequence length variance, and the memory pressure status value; S103, based on the target key value sequence length threshold, perform task segmentation on each computing task in the computing task set to obtain a target segmentation scheme, wherein the target segmentation scheme includes A target sub-tasks, where A is a positive integer; S104, determine the basic computing units corresponding to the A target sub-tasks respectively; S105 schedules each target subtask to its corresponding basic computing unit.

[0006] Compared with the prior art, the present invention has significant advantages. Through the above technical solution, the load balancing distribution system for attention computing provided by the present invention achieves considerable technological progress and practicality, and has broad industrial application value. It has at least the following advantages: This invention determines the target key-value sequence length threshold for segmentation from multiple dimensions based on the average key-value sequence length, key-value sequence length variance, and memory pressure status value corresponding to the computing task set. This avoids over-segmentation or under-segmentation caused by a fixed segmentation threshold, improving the reliability of computing task segmentation. Furthermore, by combining the memory pressure status value, it avoids memory overflow, adapts to the task differences of different users, meets the concurrent scheduling needs of multiple users, and thus improves the load balancing distribution effect of computing tasks, thereby improving the overall processing efficiency of computing tasks. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is a flowchart illustrating the execution of a computer program by a processor in a load balancing distribution system for attention computation, as provided in an embodiment of the present invention. Detailed Implementation

[0009] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0010] This embodiment provides a load balancing distribution system for attention computation. See [link to relevant documentation]. Figure 1 This is a flowchart illustrating the execution of a computer program by a processor in a load balancing distribution system for attention computation, as provided in an embodiment of the present invention. The system includes: M basic computing units, a processor, and a memory storing the computer program, where M is a positive integer. When the computer program is executed by the processor, the following steps are implemented: S101, based on the acquired set of computing tasks, determine the average key-value sequence length, key-value sequence length variance, and memory pressure status value corresponding to the set of computing tasks, wherein the set of computing tasks includes computing tasks corresponding to N users, where N is a positive integer; S102, determine the target key-value sequence length threshold corresponding to the computing task set based on the average key-value sequence length, the key-value sequence length variance, and the memory pressure status value; S103, based on the target key value sequence length threshold, perform task segmentation on each computing task in the computing task set to obtain a target segmentation scheme, wherein the target segmentation scheme includes A target sub-tasks, where A is a positive integer; S104, determine the basic computing units corresponding to the A target sub-tasks respectively; S105 schedules each target subtask to its corresponding basic computing unit.

[0011] The basic computing unit can refer to the hardware unit that performs the attention computing subtask. In this embodiment, it is the smallest hardware granularity for scheduling the computing subtask. The computing task set can include computing tasks corresponding to N users to be scheduled, and a single computing task can correspond to one attention computing.

[0012] The average key-value sequence length can be used to represent the overall scale of the key-value sequence of a computation task, the variance of the key-value sequence length can be used to represent the uniformity of the distribution of the key-value sequence length of each computation task within the set of computation tasks, and the memory pressure status value can be used to characterize the degree of stress on hardware memory resources.

[0013] The target key value sequence length threshold can be used to limit the length of the key value sequence of the subtasks after splitting to no more than the target key value sequence length threshold.

[0014] The target segmentation scheme can refer to the final determined segmentation scheme, which may include A target subtasks and the length of the key-value sequence of each target subtask.

[0015] In one specific implementation, determining the average key-value sequence length, key-value sequence length variance, and memory pressure status value corresponding to the acquired set of computing tasks includes: The average key value sequence length corresponding to the computing task set is obtained by calculating the mean based on the reference key value sequence lengths corresponding to each computing task in the computing task set. Variance is calculated based on the reference key value sequence length corresponding to each computing task in the computing task set to obtain the key value sequence length variance value corresponding to the computing task set. The summation result is obtained by summing the memory usage of each computing task in the computing task set. The memory pressure status value corresponding to the set of computing tasks is obtained by comparing the summation result with the obtained memory capacity.

[0016] The reference key-value sequence length can refer to the length of the original key-value sequence for the corresponding computation task.

[0017] Memory usage can refer to the amount of memory space required to execute a corresponding computation task. This memory space is typically used for storing query tensors, key tensors, and value tensors, as well as storing intermediate computation results.

[0018] Memory capacity can refer to the total memory capacity of all basic computing units.

[0019] In one specific implementation, determining the target key-value sequence length threshold corresponding to the computing task set based on the average key-value sequence length, the key-value sequence length variance, and the memory pressure status value includes: The base length threshold is determined based on the average key value sequence length and the preset initial length threshold; The memory pressure adjustment coefficient is determined based on the memory pressure status value and the preset pressure status threshold. The load balancing adjustment coefficient is determined based on the variance of the key value sequence length and the average key value sequence length. Based on the basic length threshold, each computing task in the computing task set is divided into intermediate division schemes, wherein the intermediate division scheme includes B intermediate subtasks, where B is a positive integer; The amount of intermediate tasks to be allocated is determined based on the number of basic computing units M and the number of intermediate subtasks B. Based on the intermediate task allocation amount and the preset allocation amount threshold, determine the integration overhead adjustment coefficient; The target key-value sequence length threshold is determined based on the base length threshold, the memory pressure adjustment coefficient, the load balancing adjustment coefficient, and the consolidation overhead adjustment coefficient.

[0020] The initial length threshold can refer to a preset basic segmentation threshold. In this embodiment, the average key value sequence length is multiplied by the first preset coefficient to obtain the first multiplication result. The larger value between the first multiplication result and the initial length threshold is used as the basic length threshold. The first preset coefficient can be set to 1.5.

[0021] The memory pressure adjustment coefficient can be used to characterize the degree of influence of memory pressure on the target key value sequence length threshold. The pressure state threshold may include a first pressure state sub-threshold and a second pressure state sub-threshold. When the memory pressure state value is greater than the first pressure state sub-threshold, the memory pressure adjustment coefficient is determined to be a first preset value. When the memory pressure state value is less than the second pressure state sub-threshold, the memory pressure adjustment coefficient is determined to be a second preset value. Otherwise, the memory pressure adjustment coefficient is determined to be a third preset value. In this embodiment, the first pressure state sub-threshold can be set to 0.8, the second pressure state sub-threshold can be set to 0.4, the first preset value can be 1.3, the second preset value can be 0.7, and the third preset value can be 1.

[0022] The load balancing adjustment coefficient can be used to characterize the impact of the volatility of the key value sequence on the target key value sequence length threshold. When the variance of the key value sequence length is greater than the average key value sequence length, the load balancing adjustment coefficient is determined to be the fourth preset value; otherwise, the load balancing adjustment coefficient is determined to be the third preset value. The fourth preset value can be 0.8.

[0023] The intermediate segmentation scheme can refer to the segmentation result obtained by segmenting according to the basic length threshold. The intermediate task allocation amount can refer to the number of intermediate subtasks allocated to a single basic computing unit. The allocation amount threshold can refer to the upper limit of the number of subtasks allocated to a single basic computing unit. When the intermediate task allocation amount is greater than the allocation amount threshold, it indicates that the integration overhead will increase beyond the limit and excessive segmentation needs to be avoided. The integration overhead adjustment coefficient is determined to be the fifth preset value. Otherwise, the integration overhead adjustment coefficient is determined to be the third preset value. In this embodiment, the allocation amount threshold can be set to 4 and the fifth preset value can be set to 1.2.

[0024] Multiply the base length threshold, memory pressure adjustment factor, load balancing adjustment factor, and consolidation overhead adjustment factor together, and use the result of the multiplication as the target key value sequence length threshold.

[0025] In one specific implementation, the step of dividing the computational tasks in the computational task set according to the target key value sequence length threshold to obtain a target segmentation scheme includes: For any computation task, the query sequence of the computation task is divided according to the preset query sequence division length to obtain C query subsequences, where C is a positive integer; The key-value sequence is traversed within the preset range of key-value sequence segmentation numbers, and the key-value sequence of the computation task is segmented according to each temporary segmentation number to obtain the temporary key-value sequence segmentation result corresponding to each temporary segmentation number. The temporary key-value sequence segmentation result includes several temporary segmentation sub-sequences and the temporary sequence length corresponding to each temporary segmentation sub-sequence. For any temporary key-value sequence segmentation result, determine the temporary segmentation scheme corresponding to the temporary key-value sequence segmentation result based on the temporary key-value sequence segmentation result and each query subsequence; The temporary segmentation schemes to which the temporary key value sequence segmentation results belong are deleted if the length of the temporary sequence corresponding to the temporary cleavage sequence is greater than the target key value sequence length threshold, and several reference segmentation schemes are obtained. Determine the cost evaluation value corresponding to each reference segmentation scheme; The reference segmentation scheme with the lowest cost evaluation value is selected as the target segmentation scheme.

[0026] The query sequence segmentation length can refer to the granularity of query sequence segmentation. Typically, the query sequence length is a fixed value, so the query sequence segmentation length can be set to a fixed value. For example, if the query sequence length is usually 128, then the query sequence segmentation length can be set to 64.

[0027] A query subsequence can refer to the segmentation result of a query sequence, and the length of the query subsequence is the same as the segmentation length of the query sequence.

[0028] The range of key-value sequence segmentation can refer to the optional range of key-value sequence segmentation. In this embodiment, the range of key-value sequence segmentation is set to [1, 8].

[0029] The number of each temporary segment within the range of key-value sequence segmentation is the integer value within the range of key-value sequence segmentation.

[0030] The result of temporary key value sequence segmentation can refer to the result obtained by segmenting according to the corresponding number of temporary segments, including the temporary segmentation sequence and the length of the temporary sequence corresponding to each temporary segmentation sequence.

[0031] A reference segmentation scheme can refer to a temporary segmentation scheme in which the temporary subsequence meets the threshold limit of the target key value sequence length. The reference segmentation scheme serves as a candidate scheme for cost evaluation.

[0032] Cost evaluation values ​​can be used to quantify the total execution cost of a reference partitioning scheme. The lower the cost evaluation value, the more reliable the corresponding reference partitioning scheme is.

[0033] In one embodiment, when the implementer obtains each reference segmentation scheme, the cache hit status of each temporary segmentation sequence in each reference segmentation scheme can be determined. Specifically, for any reference segmentation scheme, the cache hit status of each temporary segmentation sequence in the reference segmentation scheme is judged sequentially according to the initial order of each temporary segmentation sequence. For the current temporary segmentation sequence, if the current temporary segmentation sequence is the first temporary segmentation sequence, the cache hit status of the current temporary segmentation sequence is a miss. If the current temporary segmentation sequence is not the first temporary segmentation sequence, if the length of the temporary sequence corresponding to the current temporary segmentation sequence is less than the preset cache prefetch size, the cache hit status is a hit. If the length of the temporary sequence corresponding to the current temporary segmentation sequence is not less than the preset cache prefetch size, the cache hit status is a miss. This embodiment combines the prior information that GPUs usually retrieve in advance to prepare for the next computational subtask when the cache is executing the computational subtask, provided that the size supports it, to determine the cache hit status of each temporary segmentation sequence.

[0034] In one specific implementation, determining the cost evaluation value corresponding to each reference segmentation scheme includes: For any reference segmentation scheme, the computational cost corresponding to the reference segmentation scheme is determined based on the query sequence segmentation length, the preset attention head feature dimension, the preset first weight, and the temporary sequence lengths corresponding to each temporary segmentation subsequence contained in the temporary key value sequence segmentation result of the reference segmentation scheme. Based on the query sequence segmentation length, the attention head feature dimension, the preset second weight, the preset number of bytes, and the temporary sequence lengths corresponding to each temporary segmentation subsequence included in the temporary key-value sequence segmentation result in the reference segmentation scheme, the storage cost corresponding to the reference segmentation scheme is determined. The integration cost corresponding to the reference segmentation scheme is determined based on the number of temporary segmentation subsequences included in the temporary key-value sequence segmentation result in the reference segmentation scheme, the segmentation length of the query sequence, the dimension of the attention head feature, the preset third weight, and the preset number of bytes. Based on the computational cost, storage cost, and integration cost corresponding to the reference partitioning scheme, determine the cost evaluation value corresponding to the reference partitioning scheme.

[0035] Among them, computational cost can refer to the floating-point operation volume corresponding to the reference partitioning scheme, which is used to represent computational overhead; storage cost can refer to the memory data transfer volume corresponding to the reference partitioning scheme, which is used to represent storage overhead; and integration cost can refer to the memory data transfer volume required to merge the results of the partitioned subtasks, which is used to represent integration overhead.

[0036] The feature dimension of an attention head can refer to the feature dimension of a single attention head.

[0037] The preset byte count can refer to the number of bytes stored for a single data element.

[0038] Specifically, the lengths of the temporary sequences corresponding to each temporary segmentation subsequence included in the temporary key-value sequence segmentation result in the reference segmentation scheme are added together to obtain the first addition result. The first addition result, the query sequence segmentation length, the preset attention head feature dimension, the preset first weight, and the calculation count statistics are multiplied together to obtain the computation cost corresponding to the reference segmentation scheme. Since attention calculation involves multiplication and addition operations, the calculation count statistics can be set to 2.

[0039] Add the first sum to the query sequence segmentation length to obtain the second sum. Multiply the second sum, attention head feature dimension, preset second weight, and preset number of bytes to obtain the storage cost corresponding to the reference segmentation scheme.

[0040] When the number of temporary segmentation subsequences contained in the temporary key-value sequence segmentation result of the reference segmentation scheme is 1, the integration cost corresponding to the reference segmentation scheme is 0. When the number of temporary segmentation subsequences contained in the temporary key-value sequence segmentation result of the reference segmentation scheme is greater than 1, the integration cost corresponding to the reference segmentation scheme is obtained by multiplying the number of temporary segmentation subsequences contained in the temporary key-value sequence segmentation result of the reference segmentation scheme, the query sequence segmentation length, the attention head feature dimension, the preset third weight, and the preset number of bytes.

[0041] In one embodiment, when the cache hit status of each temporary segmentation sequence in the reference segmentation scheme has been determined, the cache locality cost can be increased. Specifically, for any temporary segmentation sequence in the reference segmentation scheme, if the cache hit status of the temporary segmentation sequence is hit, the cache locality cost corresponding to the temporary segmentation sequence is 0. If the cache hit status of the temporary segmentation sequence is not hit, the cache locality cost corresponding to the temporary segmentation sequence is the product of the preset fourth weight, the temporary sequence length corresponding to the temporary segmentation sequence, and the attention head feature dimension. The cache locality cost of the reference segmentation scheme is the sum of the cache locality costs corresponding to each temporary segmentation sequence in the reference segmentation scheme.

[0042] In this embodiment, the preset first weight can be set to 1, the preset second weight can be set to 2, the preset third weight can be set to 3, and the preset fourth weight can be set to 1.5. The weight settings are determined according to the time consumption of different processes. Taking the computational processing as the benchmark, the preset first weight is set to 1. The memory access time is usually twice that of the computational processing, so the preset second weight is set to 2. The integration processing involves synchronous processing and additional memory access, and the time consumption can usually be regarded as three times that of the computational processing, so the preset second weight is set to 3. The cache miss has a penalty of about 1.5 times the computational time, so the preset fourth weight is set to 1.5. The implementer can fine-tune the weights according to the actual situation.

[0043] In one specific implementation, determining the cost evaluation value corresponding to the reference partitioning scheme based on the computational cost, storage cost, and integration cost corresponding to the reference partitioning scheme includes: Based on the computational cost and preset computational throughput of the reference partitioning scheme, determine the computational cost sub-evaluation value corresponding to the reference partitioning scheme; Based on the storage cost and preset memory bandwidth corresponding to the reference partitioning scheme, determine the storage cost sub-evaluation value corresponding to the reference partitioning scheme; Based on the integration cost corresponding to the reference partitioning scheme and the preset memory bandwidth, determine the integration cost sub-evaluation value corresponding to the reference partitioning scheme; The cost evaluation value corresponding to the reference segmentation scheme is obtained by adding the sub-evaluation values ​​of the computational cost, storage cost, and integration cost corresponding to the reference segmentation scheme.

[0044] Among them, the preset computing throughput can refer to the floating-point operation throughput of the basic computing unit, which can represent the computing power of the basic computing unit, and the preset memory bandwidth can refer to the memory transmission bandwidth of the basic computing unit, which can represent the memory data transmission speed.

[0045] The computation cost sub-evaluation value can refer to the overhead of converting computation cost into time dimension, the storage cost sub-evaluation value can refer to the overhead of converting storage cost into time dimension, and the integration cost sub-evaluation value can refer to the overhead of converting integration cost into time dimension, in order to achieve normalization processing.

[0046] Specifically, the ratio of the computation cost corresponding to the reference partitioning scheme to the preset computation throughput is used as the computation cost sub-evaluation value of the reference partitioning scheme, the ratio of the storage cost corresponding to the reference partitioning scheme to the preset memory bandwidth is used as the storage cost sub-evaluation value of the reference partitioning scheme, and the ratio of the integration cost corresponding to the reference partitioning scheme to the preset memory bandwidth is used as the integration cost sub-evaluation value of the reference partitioning scheme.

[0047] In one embodiment, when obtaining the cache locality cost corresponding to the reference partitioning scheme, the computation cost sub-evaluation value, storage cost sub-evaluation value, integration cost sub-evaluation value, and cache locality cost corresponding to the reference partitioning scheme can be added together to obtain the cost evaluation value corresponding to the reference partitioning scheme.

[0048] In one specific implementation, determining the basic computing units corresponding to the A target sub-tasks includes: The basic computing units corresponding to the A target sub-tasks are determined sequentially according to a preset order; When the corresponding basic computing unit is determined for any target subtask, the load evaluation value corresponding to each basic computing unit is obtained, and the basic computing unit with the smallest load evaluation value is taken as the basic computing unit corresponding to the target subtask.

[0049] The preset order can refer to the order in which the computational complexity of the A target sub-tasks is from largest to smallest.

[0050] Load ratings can be used to quantify the load level of basic computing units. The smaller the load rating, the lower the load of the corresponding basic computing unit, and the more suitable it is to be assigned target subtasks.

[0051] In one specific implementation, the step of obtaining the load evaluation value corresponding to each basic computing unit when the corresponding basic computing unit is determined for any target subtask includes: When the corresponding basic computing unit is determined for any target subtask, for any basic computing unit, the estimated computing volume corresponding to the target subtask is added to the already carried load corresponding to the basic computing unit to obtain the computing evaluation value corresponding to the basic computing unit. Based on whether there is a subtask with the same batch index as the target subtask among the subtasks already allocated to the basic computing unit, determine the cache locality evaluation value corresponding to the basic computing unit; Based on the computation evaluation value and cache locality evaluation value corresponding to the basic computing unit, the load evaluation value corresponding to the basic computing unit is determined.

[0052] The estimated computational load can refer to the estimated floating-point operation volume of the corresponding target subtask, and the already carried load can refer to the total computational load of all target subtasks currently allocated to the corresponding basic computing unit, in order to represent the current load of the corresponding basic computing unit.

[0053] When there is a subtask with the same batch index as the target subtask among the subtasks allocated to the basic computing unit, the cache locality evaluation value corresponding to the basic computing unit is determined to be the sixth preset value. In this embodiment, the sixth preset value is set to -0.2. When there is no subtask with the same batch index as the target subtask among the subtasks allocated to the basic computing unit, the cache locality evaluation value corresponding to the basic computing unit is determined to be the third preset value. That is, when there is a cache reusable situation, a reward is given and the load evaluation value is reduced.

[0054] In this embodiment, the target key-value sequence length threshold for segmentation is determined from multiple dimensions based on the average key-value sequence length, key-value sequence length variance, and memory pressure status value corresponding to the computing task set. This avoids over-segmentation or under-segmentation caused by fixing the segmentation threshold, thus improving the reliability of computing task segmentation. Furthermore, by combining the memory pressure status value, memory overflow is avoided, which can adapt to the task differences of different users, meet the concurrent scheduling requirements of multiple users, and improve the load balancing distribution effect of computing tasks, thereby improving the overall processing efficiency of computing tasks.

[0055] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of this invention is defined by the appended claims.

Claims

1. A load balancing distribution system for attention computing, characterized in that, The system includes: M basic computing units, a processor, and a memory storing computer programs, where M is a positive integer. When the computer program is executed by the processor, the following steps are implemented: S101, based on the acquired set of computing tasks, determine the average key-value sequence length, key-value sequence length variance, and memory pressure status value corresponding to the set of computing tasks, wherein the set of computing tasks includes computing tasks corresponding to N users, where N is a positive integer; S102, determine the target key-value sequence length threshold corresponding to the computing task set based on the average key-value sequence length, the key-value sequence length variance, and the memory pressure status value; S103, based on the target key value sequence length threshold, perform task segmentation on each computing task in the computing task set to obtain a target segmentation scheme, wherein the target segmentation scheme includes A target sub-tasks, where A is a positive integer; S104, determine the basic computing units corresponding to the A target sub-tasks respectively; S105 schedules each target subtask to its corresponding basic computing unit.

2. The load balancing distribution system for attention computation according to claim 1, characterized in that, The step of determining the average key-value sequence length, key-value sequence length variance, and memory pressure status value corresponding to the acquired set of computing tasks includes: The average key value sequence length corresponding to the computing task set is obtained by calculating the mean based on the reference key value sequence lengths corresponding to each computing task in the computing task set. Variance is calculated based on the reference key value sequence length corresponding to each computing task in the computing task set to obtain the key value sequence length variance value corresponding to the computing task set. The summation result is obtained by summing the memory usage of each computing task in the computing task set. The memory pressure status value corresponding to the set of computing tasks is obtained by comparing the summation result with the obtained memory capacity.

3. The load balancing distribution system for attention computation according to claim 1, characterized in that, The step of determining the target key-value sequence length threshold corresponding to the computing task set based on the average key-value sequence length, the key-value sequence length variance, and the memory pressure status value includes: The base length threshold is determined based on the average key value sequence length and the preset initial length threshold; The memory pressure adjustment coefficient is determined based on the memory pressure status value and the preset pressure status threshold. The load balancing adjustment coefficient is determined based on the variance of the key value sequence length and the average key value sequence length. Based on the basic length threshold, each computing task in the computing task set is divided into intermediate division schemes, wherein the intermediate division scheme includes B intermediate subtasks, where B is a positive integer; The amount of intermediate tasks to be allocated is determined based on the number of basic computing units M and the number of intermediate subtasks B. Based on the intermediate task allocation amount and the preset allocation amount threshold, determine the integration overhead adjustment coefficient; The target key-value sequence length threshold is determined based on the base length threshold, the memory pressure adjustment coefficient, the load balancing adjustment coefficient, and the consolidation overhead adjustment coefficient.

4. The load balancing distribution system for attention computation according to claim 2, characterized in that, The step of dividing each computation task in the computation task set according to the target key value sequence length threshold to obtain the target division scheme includes: For any computation task, the query sequence of the computation task is divided according to the preset query sequence segmentation length to obtain C query subsequences, where C is a positive integer; The key-value sequence is traversed within the preset range of key-value sequence segmentation numbers, and the key-value sequence of the computation task is segmented according to each temporary segmentation number to obtain the temporary key-value sequence segmentation result corresponding to each temporary segmentation number. The temporary key-value sequence segmentation result includes several temporary segmentation sub-sequences and the temporary sequence length corresponding to each temporary segmentation sub-sequence. For any temporary key-value sequence segmentation result, determine the temporary segmentation scheme corresponding to the temporary key-value sequence segmentation result based on the temporary key-value sequence segmentation result and each query subsequence; The temporary segmentation schemes to which the temporary key value sequence segmentation results belong are deleted if the length of the temporary sequence corresponding to the temporary cleavage sequence is greater than the target key value sequence length threshold, and several reference segmentation schemes are obtained. Determine the cost evaluation value corresponding to each reference segmentation scheme; The reference segmentation scheme with the lowest cost evaluation value is selected as the target segmentation scheme.

5. The load balancing distribution system for attention computation according to claim 4, characterized in that, The determination of the cost evaluation value corresponding to each reference segmentation scheme includes: For any reference segmentation scheme, the computational cost corresponding to the reference segmentation scheme is determined based on the query sequence segmentation length, the preset attention head feature dimension, the preset first weight, and the temporary sequence lengths corresponding to each temporary segmentation subsequence contained in the temporary key value sequence segmentation result of the reference segmentation scheme. Based on the query sequence segmentation length, the attention head feature dimension, the preset second weight, the preset number of bytes, and the temporary sequence lengths corresponding to each temporary segmentation subsequence included in the temporary key-value sequence segmentation result in the reference segmentation scheme, the storage cost corresponding to the reference segmentation scheme is determined. The integration cost corresponding to the reference segmentation scheme is determined based on the number of temporary segmentation subsequences included in the temporary key-value sequence segmentation result in the reference segmentation scheme, the segmentation length of the query sequence, the dimension of the attention head feature, the preset third weight, and the preset number of bytes. Based on the computational cost, storage cost, and integration cost corresponding to the reference partitioning scheme, determine the cost evaluation value corresponding to the reference partitioning scheme.

6. The load balancing distribution system for attention computation according to claim 1, characterized in that, The step of determining the cost evaluation value corresponding to the reference partitioning scheme based on the computational cost, storage cost, and integration cost corresponding to the reference partitioning scheme includes: Based on the computational cost and preset computational throughput of the reference partitioning scheme, determine the computational cost sub-evaluation value corresponding to the reference partitioning scheme; Based on the storage cost and preset memory bandwidth corresponding to the reference partitioning scheme, determine the storage cost sub-evaluation value corresponding to the reference partitioning scheme; Based on the integration cost corresponding to the reference partitioning scheme and the preset memory bandwidth, determine the integration cost sub-evaluation value corresponding to the reference partitioning scheme; The cost evaluation value corresponding to the reference segmentation scheme is obtained by adding the sub-evaluation values ​​of the computational cost, storage cost, and integration cost corresponding to the reference segmentation scheme.

7. The load balancing distribution system for attention computation according to claim 1, characterized in that, The step of determining the basic computing units corresponding to the A target sub-tasks includes: The basic computing units corresponding to the A target sub-tasks are determined sequentially according to a preset order; When the corresponding basic computing unit is determined for any target subtask, the load evaluation value corresponding to each basic computing unit is obtained, and the basic computing unit with the smallest load evaluation value is taken as the basic computing unit corresponding to the target subtask.

8. The load balancing distribution system for attention computation according to claim 7, characterized in that, When determining the corresponding basic computing unit for any target subtask, the process of obtaining the load evaluation value corresponding to each basic computing unit includes: When the corresponding basic computing unit is determined for any target subtask, for any basic computing unit, the estimated computing volume corresponding to the target subtask is added to the already carried load corresponding to the basic computing unit to obtain the computing evaluation value corresponding to the basic computing unit. Based on whether there is a subtask with the same batch index as the target subtask among the subtasks already allocated to the basic computing unit, determine the cache locality evaluation value corresponding to the basic computing unit; Based on the computation evaluation value and cache locality evaluation value corresponding to the basic computing unit, the load evaluation value corresponding to the basic computing unit is determined.

Images