Sequence processing method, electronic device, storage medium, and program product

Abstract

This invention relates to the field of artificial intelligence technology, providing a sequence processing method, electronic device, storage medium, and program product. The method includes: obtaining a preset set of candidate block sizes, the set including multiple different candidate block sizes; obtaining the sequence length of a target sequence to be processed, and based on the sequence length, determining a target block size corresponding to the target sequence from the set of candidate block sizes, wherein the target block size is the candidate block size in the set that generates the least amount of invalid padding data when performing block processing on the target sequence; and performing attention calculation on the target sequence based on the target block size. This invention achieves adaptive optimization of computational load and sequence length by dynamically matching the optimal block size that minimizes invalid computation for sequences of different lengths, thereby significantly improving computational efficiency and throughput while reducing memory resource consumption.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, particularly to the field of deep learning sequence processing technology, and especially to a sequence processing method, electronic device, storage medium, and program product. Background Technology

[0002] In deep learning, especially in the application of large language models based on the Transformer architecture, in order to improve the parallel processing efficiency of hardware, multiple input sequences of different lengths are usually packaged into a batch for processing.

[0003] Currently, there are two main technical solutions for processing packaged variable-length sequences in deep learning models. One solution is to use cumulative sequence length for indexing. However, this solution is usually combined with fixed-size computational blocks, and when the batch contains short sequences, it will result in a significant waste of computational resources due to a large amount of invalid padding. The other solution is to use a mask matrix for masking, but for long sequences, this mask matrix will occupy a huge amount of storage space and bring high memory access bandwidth overhead, forming a significant performance bottleneck.

[0004] Therefore, how to efficiently process batches of sequences of varying lengths and reduce unnecessary computation and memory access overhead has become a pressing technical problem to be solved in the field of performance optimization of deep learning models (especially large language models). Summary of the Invention

[0005] This invention provides a sequence processing method, electronic device, storage medium, and program product to solve the defects in related technologies that result in wasted computing resources and high memory access overhead when processing sequences of different lengths due to the use of fixed block size or additional masks.

[0006] This invention provides a sequence processing method, comprising:

[0007] Obtain a preset set of candidate block sizes, wherein the set of candidate block sizes includes multiple different candidate block sizes;

[0008] Obtain the sequence length of the target sequence to be processed, and based on the sequence length, determine the target block size corresponding to the target sequence from the set of candidate block sizes, wherein the target block size is the candidate block size with the smallest amount of invalid padding data generated when the target sequence is segmented into blocks in the set of candidate block sizes;

[0009] Attention calculations are performed on the target sequence based on the target block size.

[0010] According to a sequence processing method provided by the present invention, obtaining the sequence length of the target sequence to be processed includes:

[0011] Obtain the cumulative sequence length array for the current batch, wherein the cumulative sequence length array records the start index position and end index position of each sequence in the current batch after concatenation;

[0012] The sequence length of the target sequence is determined based on the difference between the end index position and the start index position corresponding to the target sequence in the cumulative sequence length array.

[0013] According to a sequence processing method provided by the present invention, determining the target block size corresponding to the target sequence from the set of candidate block sizes based on the sequence length includes:

[0014] Determine whether there exists a first candidate block size in the set of candidate block sizes that is divisible by the sequence length;

[0015] If the first candidate block size exists, then the block size with the largest value among the first candidate block sizes is determined as the target block size.

[0016] According to a sequence processing method provided by the present invention, after the step of determining whether there is a first candidate block size in the set of candidate block sizes that is divisible by the sequence length, the method further includes:

[0017] If the first candidate block size does not exist, the computational waste generated when using each candidate block size in the set of candidate block sizes is calculated respectively. The computational waste is determined based on the difference between the candidate block size and the remainder after taking the sequence length modulo the candidate block size.

[0018] The candidate block size corresponding to the minimum calculated waste is determined as the target block size.

[0019] According to a sequence processing method provided by the present invention, determining the candidate block size corresponding to the minimum computational waste as the target block size includes:

[0020] Detect whether there are multiple second alternative block sizes that correspond to the one with the smallest calculated waste;

[0021] If it exists, the block size with the largest value among the multiple second candidate block sizes is determined as the target block size.

[0022] According to a sequence processing method provided by the present invention, obtaining a preset set of candidate block sizes includes:

[0023] Based on the thread bundle size of the hardware computing unit and the alignment requirements of the matrix operation dimension, the lower limit and multiple relationship of the candidate block size are determined.

[0024] Based on the hardware-based shared memory capacity limitations, determine the upper limit of the candidate block size;

[0025] The set of candidate block sizes is generated based on the lower limit of the value, the multiple relationship, and the upper limit of the value.

[0026] According to a sequence processing method provided by the present invention, before the step of determining the target block size corresponding to the target sequence from the set of candidate block sizes based on the sequence length, the method further includes:

[0027] Obtain a pre-built mapping table, which records the correspondence between a specified sequence length and the optimal block size, wherein the optimal block size is the block size with the best performance when processing the corresponding specified sequence length, determined based on benchmark tests.

[0028] Based on the sequence length of the target sequence, query the mapping table to see if there is an optimal block size corresponding to the sequence length;

[0029] If it exists, the optimal block size corresponding to the sequence length is determined as the target block size, and the step of filtering and determining the target block size from the set of candidate block sizes is skipped.

[0030] The present invention also provides a sequence processing apparatus, comprising:

[0031] A set acquisition unit is used to acquire a preset set of candidate block sizes, wherein the set of candidate block sizes includes multiple different candidate block sizes;

[0032] A size determination unit is used to obtain the sequence length of the target sequence to be processed, and based on the sequence length, determine the target block size corresponding to the target sequence from the set of candidate block sizes, wherein the target block size is the candidate block size with the smallest amount of invalid padding data generated when the target sequence is segmented into blocks in the set of candidate block sizes;

[0033] The computational execution unit is used to perform attention computation on the target sequence based on the target block size.

[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement any of the sequence processing methods described above.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sequence processing method as described above.

[0036] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the sequence processing methods described above.

[0037] The sequence processing method, electronic device, storage medium, and program product provided by this invention, before performing attention calculation, obtains a set of candidate block sizes and dynamically and adaptively selects an optimal target block size for each target sequence from the candidate block size set based on the sequence length of each target sequence to be processed. This minimizes the amount of invalid padding data generated by block processing for each target sequence, significantly reducing invalid computations introduced by data padding when processing sequences of varying lengths. Compared to schemes using fixed block sizes, this invention can flexibly adjust the computational granularity according to the actual length of the sequence. For shorter sequences, the system matches a smaller target block size, thereby reducing invalid computations caused by padding and significantly improving the utilization of hardware computing resources and computational efficiency. Simultaneously, compared to schemes using mask matrices, this invention directly adapts to the sequence length at the computational level, avoiding the memory consumption and high-bandwidth memory access bottlenecks caused by generating, storing, and loading large mask matrices, especially when processing long sequences. Therefore, this invention effectively solves the problem of computational waste without increasing additional storage burden. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of the structure of the general-purpose graphics processor provided by the present invention;

[0040] Figure 2 This is a flowchart illustrating the sequence processing method provided by the present invention;

[0041] Figure 3 This is a schematic diagram of the sequence processing device provided by the present invention;

[0042] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0044] In recent years, deep learning models based on the Transformer architecture, especially Large Language Models (LLMs), have achieved great success in the field of Natural Language Processing (NLP) and have been widely applied in many scenarios such as machine translation, text summarization, intelligent question answering, and content generation. One of the core computational units of these models is the attention mechanism, whose computational complexity is proportional to the square of the length of the input sequence.

[0045] In practical applications, whether in model training, fine-tuning, or inference, the lengths of the input text sequences (such as sentences and paragraphs) that need to be processed are often diverse and varied. To fully utilize the parallel computing capabilities of hardware accelerators such as Graphics Processing Units (GPUs) and General-purpose computing on Graphics Processing Units (GPGPUs) to improve computational efficiency and throughput, a common technique is to pack multiple input sequences into a batch for parallel processing. Sequence packing refers to concatenating or stacking multiple sequences of different lengths into one or more tensors for efficient batch processing computation.

[0046] Currently, there are two main technical approaches for handling bundled variable-length sequences in deep learning models. The first approach uses a cumulative sequence length (cu_seq_lens) for indexing. This approach uses a one-dimensional array to record the start and end positions of each sequence in the batch within the concatenated one-dimensional tensor. During attention computation, the model loads this cumulative sequence length array to determine the effective range of each sequence, thus ensuring that computation is performed only within its respective sequence and avoiding cross-computation between sequences.

[0047] The second approach uses a masking matrix. This method generates a two-dimensional masking matrix of the same size as the attention score matrix. The elements in the matrix identify valid and padding positions in the original sequence. After calculating the attention score, this masking matrix is ​​applied over the score matrix to mask the calculation results for all padding positions, ensuring that these invalid positions do not affect the final output.

[0048] However, in practical applications, it has been found that the above-mentioned technical solutions all have certain drawbacks, resulting in poor computational efficiency and resource utilization when processing sequences with large length differences. Specifically, for schemes that use cumulative sequence lengths, they are usually processed in conjunction with fixed-size computational chunks, which leads to a serious waste of computational resources. For hardware alignment and computational convenience, the computation process is often divided into fixed-size chunks. When a short sequence is assigned to a large computational chunk for processing, most of its length is occupied by invalid padding data. Although the computing units of hardware such as GPUs and GPGPUs still perform calculations on these padding positions, this part of the calculation is completely redundant and does not generate any effective value. When the sequence lengths within a batch differ greatly, for example, when a sequence of length 32 and a sequence of length 512 are processed by the same fixed-size chunk (e.g., 512), the short sequence will generate a large amount of invalid computation, resulting in a serious waste of computing resources and low overall computational efficiency.

[0049] For schemes employing mask matrices, significant storage and memory access bandwidth bottlenecks arise when processing long sequences. The size of the mask matrix increases quadratically with the sequence length. For the increasingly common long sequence or long document processing tasks, the mask matrix itself consumes a large amount of storage space. More seriously, this massive mask matrix needs to be frequently loaded from High Bandwidth Memory (HBM) during each attention calculation, putting immense pressure on memory bandwidth and creating a significant performance bottleneck. Especially in inference scenarios, this additional memory access overhead severely impacts the model's response speed and throughput.

[0050] To address this, the present invention provides a sequence processing method that dynamically and adaptively selects an optimal block size for each target sequence to be processed, minimizing the amount of invalid padding data generated during block processing. This significantly reduces invalid computations introduced by data padding when processing sequences of varying lengths, improves the computational efficiency and hardware resource utilization of deep learning models (especially attention-based models) during training and inference, and thus overcomes the aforementioned shortcomings.

[0051] It should be noted that the sequence processing method provided by this invention can be applied to scenarios involving efficient processing of variable-length sequences in deep learning models employing attention mechanisms. The execution entity of this method is typically an electronic device, such as a server, personal computer, mobile terminal, or embedded device. Specifically, this method can run on hardware equipped with a processor and memory. In deep learning applications, the processor can more specifically be a hardware accelerator capable of parallel computing, such as a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). The memory is used to store the program instructions required to execute this method and the data generated during processing.

[0052] The following section uses GPGPU as an example to briefly introduce the structure of the execution body of the method of this invention. Figure 1 This is a schematic diagram of the structure of the general-purpose graphics processor provided by the present invention, as shown below. Figure 1 As shown, the general-purpose graphics processor 100 includes at least multiple streaming processor clusters (SPCs) 101 and video memory 102. Each streaming processor cluster 101 includes multiple compute units (CUs) 103, and each compute unit 103 includes at least an on-chip cache 104 and a register 105. Here, the video memory 102 can be HBM or other types of memory.

[0053] On-chip cache 104 is a temporary memory; for example, it could be a GMB (GemmMain Buffer), with a smaller capacity than video memory 102 but a faster data exchange speed. In contrast, register 105 has a smaller capacity than on-chip cache 104 but a faster data exchange speed. Register 105 could specifically be a TLR (Thread Local Register).

[0054] Furthermore, the computation unit 103 can be either a vector computation unit or a tensor computation unit. The tensor computation unit is primarily used to perform tensor computations (such as matrix multiplication and convolution operations), and includes multiple tensor cores for executing specific computational tasks. The vector computation unit is primarily used to perform arithmetic and logical operations other than tensor computations such as matrix multiplication and convolution, such as accumulation, reduction, and regular addition, subtraction, multiplication, and division. The vector computation unit includes multiple vector cores for executing specific computational tasks.

[0055] In addition to the structures described above, the general-purpose graphics processor of this invention may also include other structures, and this invention does not specifically limit these. It should be understood that the target sequences, data blocks, etc., involved in the various embodiments of this invention can be stored in the video memory 102.

[0056] Figure 2 This is a flowchart illustrating the sequence processing method provided by the present invention, as shown below. Figure 2 As shown, the method includes:

[0057] Step S10: Obtain a preset set of candidate block sizes, which includes multiple different candidate block sizes.

[0058] Specifically, the candidate block size set refers to a predefined set containing a set of values, where each value represents a block size (denoted as chunk_size) that can be used to divide the sequence into blocks. The block size is the size of the basic processing unit for performing parallel computation (such as attention computation). To achieve adaptive processing of sequences of different lengths, embodiments of the present invention pre-define multiple different block sizes as candidates.

[0059] Understandably, the preset set of candidate block sizes is not arbitrarily set, but rather based on considerations of the hardware characteristics and memory limitations of the computation. Specifically, the selection of candidate block sizes can be based on factors such as the architectural characteristics of the hardware computing units and the hardware's memory limitations. For example, when performing computations on a GPGPU, its execution units are typically measured in warps, with warp sizes usually being 32. Therefore, to ensure hardware instruction alignment and efficient parallel computation, candidate block sizes are typically set to multiples of 32. Furthermore, matrix multiplication operations in Tensor Cores usually have specific alignment requirements for matrix dimensions (such as multiples of 8, 16, or 32), and candidate block sizes must also meet these requirements.

[0060] Furthermore, the block size cannot be infinitely large, as it is limited by the size of the cache or shared memory within the computing unit. If the block size is too large, the data required for processing a single computing block may not be fully loaded into the cache or shared memory, leading to performance degradation or even runtime errors. Therefore, considering the above factors, a set of alternative block sizes can be initialized, for example, this set of sizes could be [32, 64, 128, 256, 512, 1024].

[0061] Step S20: Obtain the sequence length of the target sequence to be processed, and based on the sequence length, determine the target block size corresponding to the target sequence from the set of candidate block sizes, wherein the target block size is the candidate block size with the smallest amount of invalid padding data generated when the target sequence is segmented into blocks in the set of candidate block sizes.

[0062] Specifically, after obtaining the set of candidate block sizes, for each target sequence to be processed, the most suitable processing size, i.e., the target block size, can be dynamically determined from the set of candidate block sizes. Here, the target sequence can be any input sequence received by the model during training, inference, etc., such as a text sequence input by the user.

[0063] Specifically, the first step is to obtain the length of the target sequence. The length of the sequence can be obtained directly, for example, by counting the number of elements in a single sequence when processing it, or by determining the length of each sequence by recording its boundary information when batch processing multiple sequences.

[0064] Next, based on the sequence length, a selection of candidate block sizes is made to determine the target block size. The selection criterion here is to minimize the amount of invalid padding data. Invalid padding data refers to the situation where, when a sequence is divided into blocks for processing, if its length is not an integer multiple of the selected block size, the last block needs to be padded with invalid data to reach a complete block size. The calculations performed on this invalid padding data are ineffective and wasteful. One of the core principles of this invention is to minimize this waste by dynamically selecting the block size.

[0065] For each target sequence to be processed, the determination of the corresponding target block size can be achieved through the following steps: Iterate through each candidate block size in the set of candidate block sizes, and calculate the amount of invalid data (i.e., invalid padding data) required to fill the last block when dividing the target sequence using that candidate block size. Then, select the candidate block size that minimizes the amount of invalid padding data as the final target block size used to process the target sequence.

[0066] Step S30: Perform attention calculation on the target sequence based on the target block size.

[0067] Specifically, once the optimal target block size is determined, this size can be used to perform core computational tasks, such as attention calculations, on the target sequence. This means that when calculating attention scores, the sequence is logically or physically divided into blocks of the target block size for processing. For example, the computational cores on a GPGPU can be configured to process one or more data blocks of this size.

[0068] For example, when performing attention computation on a GPGPU, the entire computation task is divided into multiple thread blocks, each of which is scheduled to be executed on a compute unit (CU). In this embodiment of the invention, the dynamically determined target block size directly determines the size of the data block allocated to a thread block for processing. Specifically, multiple threads within a thread block work collaboratively. They first load data blocks related to the current computation task from the GPGPU's video memory, specifically from the query (Q), key (K), and value (V) matrix corresponding to the target sequence, into the on-chip cache (e.g., shared memory) of the CU to which the thread block belongs. The size of the loaded data block is determined by the target block size. Since the target block size is dynamically optimized for the current target sequence, this ensures that the vast majority of the data loaded into the on-chip cache is valid data, thereby greatly reducing bandwidth consumption and cache space waste due to invalid padding data.

[0069] After the data block is loaded into the on-chip cache, each thread in the CU reads the required data from the cache into its private, faster-access registers. These threads then drive the computational cores within the CU (such as TensorCores) to perform matrix multiplication and accumulation operations to obtain the attention score. After the attention score is calculated, subsequent operations such as scaling, softmax, and multiplication with the value matrix can also be efficiently computed in blocks based on the selected target block size within this hardware architecture.

[0070] It is understandable that when using hardware units such as Tensor Cores, which are specifically designed for matrix multiplication and accumulation operations, the target block size preferred in this embodiment of the invention enables the dimensions of the data blocks involved in the operation to be better aligned with the granularity of the operation supported by the Tensor Core. This avoids the problem of inefficient execution of hardware instructions due to inappropriate data block size, thereby maximizing the computing performance of the hardware.

[0071] Furthermore, by employing a target block size dynamically optimized for the current target sequence, efficient utilization of computational resources (such as the CU of a GPGPU) can be ensured. Because the selected block size minimizes invalid padding, most of the computation performed by the hardware is directed towards valid data, thus avoiding the significant computational waste associated with traditional fixed-block-size methods when processing short sequences. Simultaneously, this method eliminates the need to load and process large mask matrices, reducing storage and bandwidth pressure.

[0072] The method provided in this invention, before performing attention calculation, obtains a set of candidate block sizes and dynamically and adaptively selects an optimal target block size for each target sequence from the candidate block size set based on the sequence length of each target sequence to be processed. This minimizes the amount of invalid padding data generated by block processing for each target sequence, significantly reducing invalid computations introduced by data padding when processing sequences of varying lengths. Compared to schemes using fixed block sizes, this invention can flexibly adjust the computational granularity according to the actual length of the sequence. For shorter sequences, the system matches a smaller target block size, thereby reducing invalid computations caused by padding and significantly improving the utilization of hardware computing resources and computational efficiency. Furthermore, compared to schemes using mask matrices, this invention directly adapts to the sequence length at the computational level, avoiding the memory consumption and high-bandwidth memory access bottlenecks caused by generating, storing, and loading large mask matrices, especially when processing long sequences. Therefore, this invention effectively solves the problem of computational waste without increasing additional storage burden.

[0073] Based on the above embodiments, step S10 specifically includes:

[0074] Step S11: Based on the thread bundle size of the hardware computing unit and the alignment requirements of the matrix operation dimension, determine the lower limit and multiple relationship of the candidate block size.

[0075] Specifically, the hardware computing unit can be a computing unit (CU) in a GPGPU, and a warp is the basic unit for scheduling and executing threads in a GPGPU, typically 32 in size. To ensure that all threads within a warp are effectively utilized and to avoid instruction fragmentation and resource idleness, the granularity of the computing task (i.e., the block size) should be an integer multiple of the warp size. Therefore, this typically determines the multiples of candidate block sizes (e.g., multiples of 32) and the lower bound (e.g., a minimum of 32).

[0076] The alignment requirements for matrix operation dimensions refer to the dedicated hardware units, such as Tensor Cores, designed in modern GPGPUs to accelerate matrix multiplication and addition operations. These units require that the dimensions of the matrices involved in the operation meet specific alignment conditions (e.g., multiples of 8, 16, or 32). In order to fully utilize these hardware acceleration units, the candidate block size also needs to meet these alignment requirements.

[0077] Taking into account the above factors, for example, the lower limit of the value can be determined to be 32, and the size of all candidate blocks is a multiple of 32.

[0078] Step S12: Determine the upper limit of the candidate block size based on the hardware's shared memory capacity limit.

[0079] Specifically, shared memory is a high-speed cache with limited capacity within the GPGPU computing unit. When performing complex operations such as attention calculations, the intermediate data required by a computing chunk (such as blocks of Q, K, and V matrices) is loaded into shared memory for fast access.

[0080] If the candidate block size is too large, the amount of data required by a single computation block will exceed the capacity limit of the shared memory. In this case, the computation core will have to frequently read and write data from slower global memory (such as video memory), resulting in increased computation latency, severe performance degradation, and even program errors. Therefore, a reasonable upper limit needs to be set based on the shared memory size provided by the specific hardware model, such as 1024 or 2048.

[0081] Step S13: Generate the set of candidate block sizes based on the lower limit of the value, the multiple relationship, and the upper limit of the value.

[0082] Specifically, after determining the range and constraints of the block size values, the final set can be generated. One possible generation method is to start from the lower limit of the value and increase it exponentially (e.g., by multiplying by 2) or by multiples until it does not exceed the upper limit of the value. For example, assuming the lower limit of the value is determined to be 32, the multiples are multiples of 32, and the upper limit of the value is 1024, then the set of candidate block sizes can be generated as [32, 64, 128, 256, 512, 1024].

[0083] The method for generating the candidate block size set provided in this embodiment of the invention ensures that the candidate block size set is hardware-aware. Each size in the set fully considers the parallel architecture characteristics and memory level limitations of the hardware. This ensures that no matter which block size is dynamically selected later, the computing task can be executed efficiently and in alignment on the hardware, avoiding hardware resource idleness or memory access bottlenecks caused by improper size selection.

[0084] Based on any of the above embodiments, in step S20, obtaining the sequence length of the target sequence to be processed includes:

[0085] Step S21: Obtain the cumulative sequence length array of the current batch, wherein the cumulative sequence length array records the start index position and end index position of each sequence spliced ​​in the current batch;

[0086] Step S22: Determine the sequence length of the target sequence based on the difference between the end index position and the start index position corresponding to the target sequence in the cumulative sequence length array.

[0087] It should be noted that in practical applications of deep learning, to improve hardware utilization, multiple sequences are typically packaged into a batch for parallel processing. This invention specifically describes how to efficiently obtain the length of each sequence in such a batch processing scenario.

[0088] Specifically, when processing multiple target sequences in a batch, the system first obtains a data structure called the cumulative sequence length array (i.e., the cu_seq_lens array). This cumulative sequence length array is a one-dimensional array whose purpose is to accurately record the start and end index positions of each original sequence within a long tensor when multiple sequences of different lengths are concatenated or packaged into a continuous one-dimensional tensor. It should be understood that for a batch containing N sequences, this array typically contains N+1 elements, where the 0th element is usually 0.

[0089] After obtaining the cumulative sequence length array, determining the sequence length of the i-th target sequence (indexed from 0) in the batch becomes very direct and efficient. According to the definition of the cumulative sequence length array, the starting index position of the i-th sequence is determined by the i-th element of the array (i.e., cu_seq_lens[i]), while its ending index position (excluding this position) is determined by the (i+1)-th element of the array (i.e., cu_seq_lens[i+1]).

[0090] Therefore, the sequence length of the target sequence is the difference between its end index position and its start index position, and the calculation formula is Li=cu_seq_lens[i+1]-cu_seq_lens[i], where Li represents the i-th target sequence.

[0091] For example, suppose the cumulative sequence length array cu_seq_lens for the current batch is [0, 10, 12, 16]. This means that the current batch contains 3 target sequences to be processed, which are concatenated into a continuous data stream with a total length of 16. For the first target sequence (index i=0), its starting index position is cu_seq_lens[0]=0 and its ending index position is cu_seq_lens[1]=10, which means that the first target sequence occupies the position of index [0, 10) (i.e., the positions from index 0 to index 9). Therefore, the sequence length L0 of the first target sequence is 10-0=10.

[0092] Similarly, for the second target sequence (index i=1), its starting index position is cu_seq_lens[1]=10 and its ending index position is cu_seq_lens[2]=12, which means that the second target sequence occupies the position of index [10,12), so its sequence length L1=12-10=2. For the third target sequence (index i=2), its starting index position is cu_seq_lens[2]=12 and its ending index position is cu_seq_lens[3]=16, so its sequence length L2=16-12=4.

[0093] This simple subtraction operation allows for the rapid and accurate determination of the sequence length for each target sequence in a batch. After determining the sequence length, subsequent steps can be performed: the optimal target block size is determined from the set of candidate block sizes based on this length, and finally, attention calculation is performed based on this target block size.

[0094] In this embodiment of the invention, by using an accumulated sequence length array to obtain the length of each target sequence in a batch processing scenario, the complex parsing or traversal of the packaged data is avoided. It can be completed with only one array lookup and one subtraction operation, which improves the execution efficiency of obtaining the sequence length.

[0095] Based on any of the above embodiments, in step S20, determining the target block size corresponding to the target sequence from the set of candidate block sizes based on the sequence length includes:

[0096] Step S23: Determine whether there exists a first candidate block size in the set of candidate block sizes that is divisible by the sequence length;

[0097] Step S24: If the first candidate block size exists, then the block size with the largest value among the first candidate block sizes is determined as the target block size.

[0098] Specifically, when selecting the target block size from the set of candidate block sizes, a zero-computation-waste judgment is first performed, prioritizing the selection of the largest block size divisible by the sequence length. In other words, when determining the target block size, the system first performs a highest-priority judgment: determining whether there exists a first candidate block size in the set of candidate block sizes that is divisible by the sequence length.

[0099] Here, the first candidate block size refers to those block sizes in the candidate block size set whose values ​​are completely divisible by the sequence length of the current target sequence. If such a block size exists, it means that the sequence can be divided into several complete blocks without any extra padding data, thus achieving zero computational waste.

[0100] If one or more such first candidate block sizes exist, the block size with the largest value is determined as the target block size. The reason for choosing the largest value is that, under the premise of the same amount of computation (both with zero waste), using a larger block size for computation can reduce the total number of blocks that the computing core needs to process, thereby reducing the additional system overhead caused by block switching, data loading and other operations, and further improving the overall performance.

[0101] For example, assume the target sequence length is 128, and the set of candidate block sizes is [32, 64, 128, 256, 512]. For candidate block size 32, since 128%32 = 0, it is divisible by the sequence length 128, so 32 is the first candidate block size. Similarly, since both 64 and 128 are divisible by the sequence length 128, 64 and 128 are also first candidate block sizes. At this point, there are multiple first candidate block sizes [32, 64, 128]. According to the rules of this step, the largest value, 128, can be selected as the final target block size. Once the target block size is determined in this step, the selection process ends, and no further computational waste comparison is needed.

[0102] Furthermore, if no zero-computation-waste solution exists, the block size with the minimum computational waste is selected. Therefore, after step S23, the method further includes:

[0103] Step S25: If the first candidate block size does not exist, calculate the computational waste amount generated when using each candidate block size in the candidate block size set. The computational waste amount is determined based on the difference between the candidate block size and the remainder after taking the sequence length modulo the candidate block size.

[0104] Step S26: Determine the candidate block size corresponding to the minimum calculated waste as the target block size.

[0105] Specifically, if it is determined that there is no block size in the set of candidate block sizes that is divisible by the sequence length, i.e. there is no first candidate block size, then the system will proceed to this step to find a suboptimal solution.

[0106] At this point, the system calculates the computational waste incurred when using each candidate block size from the set of candidate block sizes. This computational waste is numerically equivalent to the amount of invalid padding data; it quantifies the amount of invalid data padded to the last computational block for rounding purposes. It can be calculated based on the difference between the candidate block size and the remainder after dividing the sequence length by the candidate block size. The specific formula is as follows: Computational waste = Candidate block size - (Sequence length % Candidate block size), where the remainder after dividing the sequence length by the candidate block size represents the length of valid data in the last incomplete block. Subtracting this remainder from the candidate block size gives the length of invalid data that needs to be padded, i.e., the computational waste.

[0107] After calculating the computational waste for each candidate block size in the set, the system will select the candidate block size that minimizes the computational waste as the target block size.

[0108] Furthermore, when multiple options exist for minimizing computational waste, the maximum block size winning strategy is executed. Specifically, step S26 includes:

[0109] Detect whether there are multiple second alternative block sizes that correspond to the one with the smallest calculated waste;

[0110] If it exists, the block size with the largest value among the multiple second candidate block sizes is determined as the target block size.

[0111] Specifically, during the calculation of waste, multiple different candidate block sizes may produce the same minimum computational waste. In this case, the system first checks whether there are multiple second candidate block sizes with the minimum corresponding computational waste. Here, the second candidate block sizes refer to those candidate block sizes that share the minimum computational waste.

[0112] If multiple alternative block sizes exist, the system will not select one randomly. Instead, it will determine the block size with the largest value among these alternative block sizes as the final target block size. This is because, given the same level of computational waste, choosing a larger processing unit (i.e., the block size) can result in lower system overhead and higher execution efficiency.

[0113] For example, assume the target sequence length is 117 and the set of candidate block sizes is [32, 64, 128, 256]. First, a zero-waste check is performed, finding that no number in the set is divisible by 117. The process proceeds to the second step, calculating the computational waste for each candidate block size. For candidate block size 32, the corresponding computational waste is 32 - (117%32) = 11; for candidate block size 64, the computational waste is 64 - (117%64) = 11; for candidate block size 128, the computational waste is 128 - (117%128) = 11; for candidate block size 256, the computational waste is 256 - (117%256) = 139. The comparison revealed that the minimum computational waste was 11, and multiple second candidate block sizes corresponding to the minimum waste of 11 were detected, namely [32, 64, 128]. The one with the largest value, namely 128, was selected as the final target block size.

[0114] This invention provides a hierarchical, ordered method for determining block size. It prioritizes finding the largest block size divisible by the sequence length to achieve zero computational waste. If this is not possible, it prioritizes minimizing computational waste. Furthermore, when multiple options exist for minimizing computational waste, the method prioritizes the largest block size for the final decision. This method ensures that an optimal computational granularity is always determined regardless of the input sequence length, thereby maximizing the balance between computational efficiency and system overhead in various scenarios. This achieves refined utilization of hardware computing power and a significant improvement in overall processing performance.

[0115] Based on any of the above embodiments, before the step of determining the target block size corresponding to the target sequence from the candidate block size set based on the sequence length in step S20, the method further includes:

[0116] Obtain a pre-built mapping table, which records the correspondence between a specified sequence length and the optimal block size, wherein the optimal block size is the block size with the best performance when processing the corresponding specified sequence length, determined based on benchmark tests.

[0117] Based on the sequence length of the target sequence, query the mapping table to see if there is an optimal block size corresponding to the sequence length;

[0118] If it exists, the optimal block size corresponding to the sequence length is determined as the target block size, and the step of filtering and determining the target block size from the set of candidate block sizes is skipped.

[0119] It should be noted that the embodiments of the present invention introduce an optional fast path optimization mechanism based on table lookup. This mechanism aims to further reduce the computational overhead when dynamically selecting the target block size, and is especially suitable for scenarios with extremely high latency requirements, such as inference.

[0120] Specifically, the mapping table can be a hash table, dictionary, or similar key-value pair storage structure. The key is a specified sequence length, and the value is the optimal block size corresponding to that sequence length. This mapping table is not dynamically generated at runtime but is pre-built, specifically through the following steps: Before model deployment or during initialization, benchmark tests are run for some common or typical sequence lengths (e.g., 128, 256, 512, 1024, etc.). That is, for each specified sequence length, all candidate block sizes are traversed to perform actual attention calculations, and the execution latency or throughput is precisely measured for each case. Then, the block size with the best performance in the tests (e.g., lowest latency) is taken as the optimal block size for that specified sequence length, and its correspondence is stored in the mapping table. It should be understood that the optimal size obtained from benchmark tests is empirically optimal, taking into account various complex factors such as computational waste, cache efficiency, and instruction scheduling.

[0121] During model training or inference, for a given target sequence, its sequence length is first obtained, and then this length is used as the key to query the aforementioned mapping table. If an entry corresponding to the current sequence length is found in the mapping table, the optimal block size recorded in the table is directly used as the final target block size. Subsequently, all subsequent complex filtering steps, such as integer division checks, waste calculations, and size comparisons, are skipped. The process directly proceeds to step S30, which performs attention calculation based on the target block size obtained from the table.

[0122] If the value does not exist (i.e., no table match), it indicates that the current sequence length does not belong to the pre-tested typical length. At this point, the normal dynamic selection process begins, which involves continuing the filtering step of the candidate block size set and determining the target block size through calculation.

[0123] The table lookup optimization mechanism introduced in this embodiment of the invention is equivalent to adding a high-speed cache to the block size selection process. For frequently occurring or performance-critical sequence lengths, a single fast table lookup operation replaces the dynamic decision-making process that originally required multiple calculations and comparisons, reducing the performance overhead and latency of selecting the block size itself.

[0124] The sequence processing apparatus provided by the present invention is described below. The sequence processing apparatus described below and the sequence processing method described above can be referred to in correspondence.

[0125] Based on any of the above embodiments Figure 3 This is a schematic diagram of the sequence processing device provided by the present invention, as shown below. Figure 3 As shown, the device includes:

[0126] The set acquisition unit 310 is used to acquire a preset set of candidate block sizes, wherein the set of candidate block sizes includes multiple different candidate block sizes;

[0127] Size determination unit 320 is used to obtain the sequence length of the target sequence to be processed, and based on the sequence length, determine the target block size corresponding to the target sequence from the set of candidate block sizes, wherein the target block size is the candidate block size with the smallest amount of invalid padding data generated when the target sequence is segmented into blocks in the set of candidate block sizes;

[0128] The computation execution unit 330 is used to perform attention computation on the target sequence based on the target block size.

[0129] The apparatus provided in this invention, before performing attention calculation, acquires a set of candidate block sizes and dynamically and adaptively selects an optimal target block size for each target sequence from the candidate block size set based on the sequence length of each target sequence to be processed. This minimizes the amount of invalid padding data generated by block processing for each target sequence, significantly reducing invalid computations introduced by data padding when processing sequences of varying lengths. Compared to schemes using fixed block sizes, this invention can flexibly adjust the computational granularity according to the actual length of the sequence. For shorter sequences, the system matches a smaller target block size, thereby reducing invalid computations caused by padding and significantly improving the utilization rate and computational efficiency of hardware computing resources. Furthermore, compared to schemes using mask matrices, this invention directly adapts to the sequence length at the computational level, avoiding the memory consumption and high-bandwidth memory access bottlenecks caused by generating, storing, and loading large mask matrices, especially when processing long sequences. Therefore, this invention effectively solves the problem of computational waste without increasing additional storage burden.

[0130] Based on any of the above embodiments, the size determination unit includes a length acquisition subunit, which is specifically used for:

[0131] Obtain the cumulative sequence length array for the current batch, wherein the cumulative sequence length array records the start index position and end index position of each sequence in the current batch after concatenation;

[0132] The sequence length of the target sequence is determined based on the difference between the end index position and the start index position corresponding to the target sequence in the cumulative sequence length array.

[0133] Based on any of the above embodiments, the size determination unit is specifically used for:

[0134] Determine whether there exists a first candidate block size in the set of candidate block sizes that is divisible by the sequence length;

[0135] If the first candidate block size exists, then the block size with the largest value among the first candidate block sizes is determined as the target block size.

[0136] Based on any of the above embodiments, the size determination unit is further configured to:

[0137] If the first candidate block size does not exist, the computational waste generated when using each candidate block size in the set of candidate block sizes is calculated respectively. The computational waste is determined based on the difference between the candidate block size and the remainder after taking the sequence length modulo the candidate block size.

[0138] The candidate block size corresponding to the minimum calculated waste is determined as the target block size.

[0139] Based on any of the above embodiments, the size determination unit is further configured to:

[0140] Detect whether there are multiple second alternative block sizes that correspond to the one with the smallest calculated waste;

[0141] If it exists, the block size with the largest value among the multiple second candidate block sizes is determined as the target block size.

[0142] Based on any of the above embodiments, the set acquisition unit is specifically used for:

[0143] Based on the thread bundle size of the hardware computing unit and the alignment requirements of the matrix operation dimension, the lower limit and multiple relationship of the candidate block size are determined.

[0144] Based on the hardware-based shared memory capacity limitations, determine the upper limit of the candidate block size;

[0145] The set of candidate block sizes is generated based on the lower limit of the value, the multiple relationship, and the upper limit of the value.

[0146] Based on any of the above embodiments, the size determination unit is further configured to:

[0147] Obtain a pre-built mapping table, which records the correspondence between a specified sequence length and the optimal block size, wherein the optimal block size is the block size with the best performance when processing the corresponding specified sequence length, determined based on benchmark tests.

[0148] Based on the sequence length of the target sequence, query the mapping table to see if there is an optimal block size corresponding to the sequence length;

[0149] If it exists, the optimal block size corresponding to the sequence length is determined as the target block size, and the step of filtering and determining the target block size from the set of candidate block sizes is skipped.

[0150] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a sequence processing method, which includes: obtaining a preset set of candidate block sizes, the set of candidate block sizes including multiple different candidate block sizes; obtaining the sequence length of the target sequence to be processed, and determining, based on the sequence length, a target block size corresponding to the target sequence from the set of candidate block sizes, wherein the target block size is the candidate block size with the smallest amount of invalid padding data generated when the target sequence is segmented into blocks in the set of candidate block sizes; and performing attention calculation on the target sequence based on the target block size.

[0151] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the sequence processing method provided by the above methods. The method includes: obtaining a preset set of candidate block sizes, the set of candidate block sizes including multiple different candidate block sizes; obtaining the sequence length of the target sequence to be processed, and determining, based on the sequence length, a target block size corresponding to the target sequence from the set of candidate block sizes, wherein the target block size is the candidate block size with the smallest amount of invalid padding data generated when the target sequence is segmented in the set of candidate block sizes; and performing attention calculation on the target sequence based on the target block size.

[0153] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the sequence processing method provided by the above methods. The method includes: obtaining a preset set of candidate block sizes, the set of candidate block sizes including multiple different candidate block sizes; obtaining the sequence length of the target sequence to be processed, and determining, based on the sequence length, a target block size corresponding to the target sequence from the set of candidate block sizes, wherein the target block size is the candidate block size in the set of candidate block sizes that generates the smallest amount of invalid padding data when the target sequence is segmented; and performing attention calculation on the target sequence based on the target block size.

[0154] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data block processing method, characterized in that, include: Obtain a preset set of candidate block sizes, which includes multiple different candidate block sizes. The set of candidate block sizes is determined based on the thread bundle size and shared memory capacity limit of the hardware computing unit. Obtain the sequence length of the target sequence to be processed, and based on the sequence length, determine the target block size corresponding to the target sequence from the set of candidate block sizes, wherein the target block size is the candidate block size with the smallest amount of invalid padding data generated when the target sequence is segmented into blocks in the set of candidate block sizes; Attention calculations are performed on the target sequence based on the target block size.

2. The data block processing method according to claim 1, characterized in that, The step of obtaining the sequence length of the target sequence to be processed includes: Obtain the cumulative sequence length array for the current batch, wherein the cumulative sequence length array records the start index position and end index position of each sequence in the current batch after concatenation; The sequence length of the target sequence is determined based on the difference between the end index position and the start index position corresponding to the target sequence in the cumulative sequence length array.

3. The data block processing method according to claim 1, characterized in that, The step of determining the target block size corresponding to the target sequence from the set of candidate block sizes based on the sequence length includes: Determine whether there exists a first candidate block size in the set of candidate block sizes that is divisible by the sequence length; If the first candidate block size exists, then the block size with the largest value among the first candidate block sizes is determined as the target block size.

4. The data block processing method according to claim 3, characterized in that, After the step of determining whether there exists a first candidate block size in the set of candidate block sizes that is divisible by the sequence length, the method further includes: If the first candidate block size does not exist, the computational waste generated when using each candidate block size in the set of candidate block sizes is calculated respectively. The computational waste is determined based on the difference between the candidate block size and the remainder after taking the sequence length modulo the candidate block size. The candidate block size corresponding to the minimum calculated waste is determined as the target block size.

5. The data block processing method according to claim 4, characterized in that, The step of determining the candidate block size corresponding to the minimum calculated waste as the target block size includes: Detect whether there are multiple second alternative block sizes that correspond to the one with the smallest calculated waste; If it exists, the block size with the largest value among the multiple second candidate block sizes is determined as the target block size.

6. The data block processing method according to any one of claims 1 to 5, characterized in that, The process of obtaining a preset set of candidate block sizes includes: Based on the thread bundle size of the hardware computing unit and the alignment requirements of the matrix operation dimension, the lower limit and multiple relationship of the candidate block size are determined. Based on the hardware-based shared memory capacity limitations, determine the upper limit of the candidate block size; The set of candidate block sizes is generated based on the lower limit of the value, the multiple relationship, and the upper limit of the value.

7. The data block processing method according to any one of claims 1 to 5, characterized in that, Before the step of determining the target block size corresponding to the target sequence from the set of candidate block sizes based on the sequence length, the method further includes: Obtain a pre-built mapping table, which records the correspondence between a specified sequence length and the optimal block size, wherein the optimal block size is the block size with the best performance when processing the corresponding specified sequence length, determined based on benchmark tests. Based on the sequence length of the target sequence, query the mapping table to see if there is an optimal block size corresponding to the sequence length; If it exists, the optimal block size corresponding to the sequence length is determined as the target block size, and the step of filtering and determining the target block size from the set of candidate block sizes is skipped.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the data block processing method as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data block processing method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data block processing method as described in any one of claims 1 to 7.

Images