Operator optimization method, electronic device, storage medium and program product

Abstract

The present invention relates to the field of artificial intelligence chip technology, and provides an operator optimization method, electronic device, storage medium and program product, wherein the method includes: determining the block size of the operator data in each dimension based on the batch size and mask mode of the operator data, and the hardware parameters of the computing device, wherein the dimensions include the batch dimension and the sequence length dimension; based on the block size of the operator data in each dimension, the operator data is segmented to obtain multiple data blocks; the computing tasks corresponding to the multiple data blocks are assigned to multiple processing units on the computing device, and the computing tasks corresponding to the multiple data blocks are executed in parallel based on the multiple processing units. The present invention simultaneously segments the data in the batch dimension and the sequence length dimension, so that when the data batch is small, multiple processing units on the computing device can also participate in the calculation at the same time, thereby avoiding idle hardware resources, improving the utilization rate of hardware resources, and thus improving the overall computing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence chip technology, and in particular to an operator optimization method, electronic equipment, storage medium, and program product. Background Art

[0002] With the rapid development of AI (Artificial Intelligence) technology, various AI devices, leveraging their powerful parallel computing capabilities, have become the core driving force behind complex computing tasks such as deep learning. To fully leverage the parallel computing potential of these devices, existing operator execution strategies often employ data parallelism, assigning each batch of data to a processing unit within the device for processing, and assigning different batches of data to different processing units within the device for execution. In scenarios with large data batches, this strategy enables multiple processing units to process different batches of data simultaneously, effectively utilizing hardware resources and improving computing efficiency, thereby accelerating model training or inference.

[0003] However, as practical application scenarios become increasingly diverse, small data batches are increasingly common. Existing operator execution strategies, however, have exposed significant flaws. When data batches are small, a large number of processing units within the device remain idle, resulting in underutilized hardware resources, significantly reduced hardware utilization, low computing efficiency, and difficulty in fully realizing the performance advantages of AI devices. Summary of the Invention

[0004] The present invention provides an operator optimization method, electronic device, storage medium and program product to solve the defect in related technologies that hardware resources cannot be fully utilized in small batch data scenarios, resulting in low operator calculation efficiency.

[0005] The present invention provides an operator optimization method, comprising:

[0006] Determining, based on the batch size and masking pattern of the operator data and hardware parameters of the computing device, the chunking size of the operator data in each dimension, including the batch dimension and the sequence length dimension;

[0007] Based on the block size of the operator data in each dimension, the operator data is segmented to obtain multiple data blocks;

[0008] The computing tasks corresponding to the multiple data blocks are allocated to multiple processing units on the computing device, and the computing tasks corresponding to the multiple data blocks are executed in parallel based on the multiple processing units.

[0009] According to an operator optimization method provided by the present invention, determining the block size of the operator data in each dimension based on the batch size and mask mode of the operator data and the hardware parameters of the computing device includes:

[0010] Determining a block size of the operator data in the batch dimension based on the batch size of the operator data and the number of processing units in the hardware parameters;

[0011] Based on the mask mode of the operator data and the buffer size in the hardware parameters, a block size of the operator data in the sequence length dimension is determined.

[0012] According to an operator optimization method provided by the present invention, determining the block size of the operator data in the batch dimension based on the batch size of the operator data and the number of processing units in the hardware parameters includes:

[0013] Calculating the greatest common divisor of the batch size and the number of processing units;

[0014] Based on the greatest common divisor and the number of processing units, a block size of the operator data in the batch dimension is determined.

[0015] According to an operator optimization method provided by the present invention, the block size in the sequence length dimension includes the block size in the row direction and the block size in the column direction, and determining the block size of the operator data in the sequence length dimension based on the mask mode of the operator data and the buffer size in the hardware parameters includes:

[0016] determining a block size of the operator data in a column direction based on a mask pattern of the operator data;

[0017] The block size of the operator data in the row direction is determined based on the buffer size and the block size of the operator data in the column direction.

[0018] According to an operator optimization method provided by the present invention, allocating computing tasks corresponding to the multiple data blocks to multiple processing units on the computing device includes:

[0019] Based on a preset allocation strategy, computing tasks corresponding to the multiple data blocks are allocated to multiple processing units on the computing device. The preset allocation strategy is any one of an equal distribution strategy and a load balancing strategy.

[0020] According to an operator optimization method provided by the present invention, allocating computing tasks corresponding to the multiple data blocks to multiple processing units on the computing device includes:

[0021] When the mask mode of the operator data is a causal mask, the computing tasks corresponding to the data blocks are allocated to the processing units in an alternating manner of positive order allocation and reverse order allocation based on the task numbers corresponding to the data blocks.

[0022] According to an operator optimization method provided by the present invention, the parallel execution of computing tasks corresponding to the multiple data blocks based on the multiple processing units includes:

[0023] For each processing unit, a data block related to the computing task of the processing unit is preloaded from an off-chip memory to a target memory area, and the data block is loaded from the target memory area to execute the computing task corresponding to the data block. The target memory area is any one of a non-uniform memory access area and an on-chip memory.

[0024] The present invention also provides an operator optimization device, comprising:

[0025] a block determination module for determining block sizes of the operator data in various dimensions, including a batch dimension and a sequence length dimension, based on a batch size and a mask mode of the operator data and hardware parameters of a computing device;

[0026] A data segmentation module, configured to segment the operator data based on the block size of the operator data in each dimension to obtain multiple data blocks;

[0027] The task execution module is used to assign the computing tasks corresponding to the multiple data blocks to the multiple processing units on the computing device, and execute the computing tasks corresponding to the multiple data blocks in parallel based on the multiple processing units.

[0028] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the operator optimization method described above is implemented.

[0029] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the above-described operator optimization methods when executed by a processor.

[0030] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned operator optimization methods.

[0031] The operator optimization method, electronic device, storage medium and program product provided by the present invention can quickly determine the block size of the operator data in various dimensions such as the batch dimension and the sequence length dimension based on the batch size of the operator data, the mask mode and the hardware parameters of the computing device. This step fully takes into account the data characteristics and hardware capabilities, and lays the foundation for subsequent efficient processing. By dividing the operator data according to the determined block size of each dimension, the operator data can be reasonably disassembled into small units that are convenient for parallel processing, and finally the computing tasks corresponding to these divided data blocks are reasonably allocated to multiple processing units on the computing device for parallel execution. The present invention divides the data in the batch dimension and the sequence length dimension at the same time, so that even in the face of a scenario with a small data batch, multiple processing units on the computing device can participate in the calculation at the same time, thereby avoiding idle hardware resources, making full use of the parallel computing capabilities of the hardware, significantly improving the utilization rate of hardware resources, and thus greatly improving the overall computing efficiency, achieving efficient data processing and calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 Schematic diagram of the attention block calculation process provided by the present invention;

[0034] Figure 2 is a schematic structural diagram of a graphics processor provided by the present invention;

[0035] Figure 3 It is a flowchart of the operator optimization method provided by the present invention;

[0036] Figure 4 is a schematic diagram of the causal mask provided by the present invention;

[0037] Figure 5 It is a schematic diagram of computing task allocation provided by the present invention;

[0038] Figure 6 Schematic diagram of the structure of the operator optimization device provided by the present invention;

[0039] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0041] With the rapid development of artificial intelligence (AI) technology, various AI devices, such as GPUs (Graphics Processing Units), GPGPUs (General-purpose computing on Graphics Processing Units), and TPUs (Tensor Processing Units), have become key drivers of complex computing tasks like deep learning thanks to their powerful parallel computing capabilities. To fully tap the parallel computing potential of these devices, existing operator execution strategies typically employ data parallelism, distributing different batches of data to different processing units within the device for execution.

[0042] For example, a GPU contains multiple SPCs (Streaming Processor Clusters), each of which can be considered a processing unit. When executing an operator, each batch of data is assigned to an SPC for processing, and different batches of data are assigned to different SPCs for parallel processing. This strategy effectively utilizes the device's parallel computing resources in large batch data scenarios, allowing multiple SPCs to simultaneously process different batches of data, significantly improving computing efficiency and accelerating model training or inference.

[0043] However, with the diversification of real-world application scenarios, the use of small data batches is increasing. For example, in scenarios with high real-time requirements, such as online learning, small-sample learning, and edge computing, small data batches often need to be processed. Existing operator execution strategies, however, exhibit significant flaws. For example, assuming the GPU has 16 SPCs (SPC0 through SPC15) and the data batch size is only four, consisting of batches 0, 1, 2, and 3, the existing strategy allocates only four batches of data to the four SPCs for processing. For example, batch 0 is allocated to SPC0, batch 1 to SPC1, batch 2 to SPC2, and batch 3 to SPC3. The remaining 12 SPCs (SPC4 through SPC15) remain idle, resulting in an unbalanced computational load across the SPCs. Furthermore, most of the GPU's computing resources are not effectively utilized, significantly reducing hardware utilization and leading to low computational efficiency.

[0044] It's important to note that the Attention operator is a key operator in AI models, helping them filter out the most critical information for the task at hand from a vast array of information. However, the computational and memory complexity of the Attention operator increases quadratically with sequence length, becoming a major bottleneck for model performance. This is particularly true when processing long sequences, where computational cost and memory usage are particularly prominent. This is because the core mechanism of the Attention operator lies in calculating the score matrix, which is typically N×N in size, where N is the length of the input sequence. Since the size of the score matrix is ​​proportional to the square of the sequence length, when the input sequence is long, the score matrix consumes a large amount of memory, leading to insufficient memory. To mitigate memory constraints, algorithms such as Flash Attention employ a block-based computation strategy. This divides the input sequence into multiple chunks, computes the attention on each chunk, and merges all intermediate results to obtain the final result, thereby reducing memory requirements. It should be understood that in the traditional Attention operator computation process, when there are multiple, long batches of data, each batch is assigned to a SPC for processing, and each batch is further divided into multiple chunks for processing. For example, suppose that the data of batch 0 is assigned to SPC0 for processing. When the data of batch 0 is long, the batch data will be divided into multiple blocks, and SPC0 will calculate these blocks one by one in sequence.

[0045] Figure 1 This is a flow chart of the attention block calculation provided by the present invention, such as Figure 1As shown in the figure, during the Attention calculation process, the input sequence (such as a text sequence) is first converted into an embedded representation, that is, each token in the text sequence (referring to the smallest unit when the model processes text, which can be understood as a character, a word, or a symbol fragment) is mapped to a vector of fixed dimension through the embedding layer. Then, the vector is converted into a query vector matrix (Query, abbreviated as Q), a key vector matrix (Key, abbreviated as K), and a value vector matrix (Value, abbreviated as V) through linear transformation; then, the Q, K, and V matrices are split respectively to obtain their corresponding blocks; then, all Q blocks are traversed, and the currently traversed Q block (denoted as Qi) is loaded; for each Qi, all K and V blocks are traversed, and the currently traversed K and V blocks (denoted as Ki and Vi) are loaded for Attention calculation. During the calculation process, an MMA (Matrix Multiply Accumulate) operation is first performed on Qi and Ki to obtain the first result (i.e., a submatrix in the score matrix). A softmax operation is then performed on this first result to obtain the second result (i.e., a submatrix in the attention weight matrix). Finally, an MMA operation is performed on the second result and Vi to obtain the corresponding intermediate result. The calculation continues with the next K and V blocks until all K and V blocks have been traversed, obtaining the calculation result corresponding to the current block Qi. Repeat these steps until all Q blocks have been traversed to obtain the final result.

[0046] While chunking can optimize memory constraints and is an effective performance improvement, determining the optimal chunking strategy remains a key challenge. Different chunking strategies have varying performance impacts, necessitating experimentation to explore various possible strategies and find the optimal one. This traversal of various chunking strategies is labor-intensive and time-consuming, as it requires writing different code to implement different chunking methods, and conducting parallel performance testing and comparisons. In practice, as new network models and requirements constantly emerge, the need to optimize the chunking strategy for each new situation inevitably increases development complexity and workload.

[0047] To this end, the present invention provides an operator optimization method that can solve the performance optimization problem of executing the Attention operator on devices such as GPUs. Specifically, the present invention first proposes an optimization solution for the problem of not being able to fully utilize hardware resources in small batch data scenarios. By splitting the operator data in the batch dimension and the sequence dimension at the same time, when the data batch is small, the computing tasks can be reasonably allocated to each processing unit on the computing device, so that these processing units can participate in the calculation at the same time, avoiding idle hardware resources, and improving hardware resource utilization and overall computing efficiency. In addition, in response to the cumbersome splitting strategy traversal optimization process, the present invention designs a general splitting strategy determination scheme. Based on the batch size of the operator data, the mask mode and the hardware parameters of the computing device, the block size of the operator data in various dimensions such as the batch dimension and the sequence length dimension can be quickly determined, avoiding the cumbersome process of manually traversing and trying various splitting strategies, thereby overcoming the above-mentioned defects.

[0048] It should be noted that the execution entity of the operator optimization method provided by the present invention can be an artificial intelligence device such as GPU, GPGPU, TPU, etc. The following takes GPU as an example to briefly introduce the structure of the execution entity of the present invention.

[0049] Figure 2 Schematic diagram of the structure of the graphics processor provided by the present invention, such as Figure 2 As shown, the graphics processor 200 includes at least multiple streaming processor clusters (i.e., SPCs) 201 and video memory 202. Each streaming processor cluster 201 includes multiple computing units 203, and each computing unit 203 includes at least an on-chip cache 204 and a register 205. Video memory 202 can be off-chip High Bandwidth Memory (HBM) or other types of memory.

[0050] On-chip cache 204 is a temporary memory. For example, on-chip cache 204 may be a GMB (GemmMain Buffer). Its capacity is smaller than that of video memory 202, but its data exchange speed is faster than that of video memory 202. Compared to on-chip cache 204, register 205 has a smaller capacity than on-chip cache 204, but its data exchange speed is faster than that of on-chip cache 204. Register 205 may specifically be a TLR (Thread Local Register).

[0051] In addition to the aforementioned structures, the graphics processor 200 of the present invention may also include other structures, which are not specifically limited in the present invention. It should be understood that the operator data, data blocks, etc. involved in various embodiments of the present invention may be stored in the video memory 202. The processing unit involved in various embodiments of the present invention may be a streaming processor cluster 201.

[0052] based on Figure 2 The architecture diagram of the graphics processor shown in the figure, the present invention provides an operator optimization method, which can be applied to various scenarios, such as text processing, image processing, speech processing and other scenarios. In different application scenarios, operator data has different physical meanings. For example, in a text processing scenario, operator data can be text-related data used in tasks such as text generation and text recognition. For another example, in an image processing scenario, operator data can be image-related data used in tasks such as image preprocessing, image segmentation, and target detection. For another example, in a speech processing scenario, operator data can be speech-related data used in tasks such as speech recognition and speech synthesis. For ease of understanding, the technical solution provided by the present invention is introduced below by taking the Attention operator optimization and the operator data being text sequence data as an example.

[0053] Figure 3 It is a flow chart of the operator optimization method provided by the present invention, such as Figure 3 As shown, the method includes:

[0054] Step 310: Determine the block size of the operator data in each dimension based on the batch size and mask mode of the operator data and the hardware parameters of the computing device, wherein the dimensions include the batch dimension and the sequence length dimension.

[0055] Specifically, operator data refers to the input data involved in operator calculations. For example, in GPU-based Attention operator optimization scenarios, operator data is the core input data involved in Attention calculations and typically exists in the form of a multi-dimensional tensor. Taking natural language processing tasks as an example, operator data can be a three-dimensional tensor whose dimensions can be expressed as (batch_size, sequence_length, feature_size). Batch_size represents the batch size, which refers to the number of data samples input to the model for processing at one time during deep learning model training or inference. For example, when training a neural network, to improve computational efficiency, multiple samples are often fed into the model at once for calculation. This number of samples is the batch size. For example, batch_size = 32 means that 32 samples of data are processed simultaneously.

[0056] sequence_length indicates the sequence length. For sequence data (such as text sequences), the sequence length indicates the number of elements in each sample. For text, for example, the sequence length can be the number of words in a sentence. For example, if a sentence has 50 words, the sequence length is 50. In speech processing, the sequence length can be the number of audio frames.

[0057] feature_size refers to the feature dimension, indicating the number of features each element has. In text processing, the feature dimension can be the dimension of the word vector. For example, if the word vector dimension obtained using a pre-trained word vector model is 300, then the feature dimension is 300.

[0058] It's understandable that operator data is the input to the Attention operator, which uses this data to calculate the associations between different positions or features. During the Attention calculation process, operator data is used to generate the Q, K, and V matrices, which serve as the basis for calculating attention weights and performing weighted summations.

[0059] For example, consider a text classification task where the input is a batch of English sentences, each consisting of words represented by pre-trained word vectors. Assuming the batch size is 2 (batch_size=2), each sentence has 10 words (sequence_length=10), and the word vector dimension is 200 (feature_size=200), the operator data is a three-dimensional tensor with a shape of (2, 10, 200). This tensor contains two samples, each with 10 words, and each word is represented by a 200-dimensional vector. In the attention calculation, the Q, K, and V matrices are generated from this three-dimensional tensor. Attention weights are then calculated between different words to help the model better understand the relationships between words in a sentence, thereby improving text classification accuracy.

[0060] A masking pattern is a mechanism used to control which elements are included in a computation and which are ignored. In the Attention mechanism, masking patterns are often used to handle different scenarios in sequence data. Common masking patterns include padding masks and causal masks. Padding masks are used to handle data sequences of inconsistent lengths. For example, when processing variable-length sequences, padding is often performed to make them consistent for batch processing. The padding mask identifies which locations contain original data and which contain padding values, allowing the model to ignore the padding values ​​during computation and focus solely on the valid sequence portions.

[0061] Causal masking is primarily used for processing sequential data, such as text sequences in natural language processing. It ensures that the model, when generating a sequence, can only rely on previously generated data and cannot "see" future information, thereby ensuring that the generated sequence is causally sound. Here, "seeing" refers to the model's acquisition and utilization of information when processing sequential data. In sequential data processing, models typically encode the input sequence, converting each element in the sequence (such as a word in text) into a vector representation. Under standard attention mechanisms, the model comprehensively interacts with and reads information from all elements in the sequence. Causal masking, however, limits the model's access to sequential information. When a model "cannot see future information," it means that when processing the element at the current position, the model does not read the vector representations corresponding to elements after the current position in the sequence. For example, in a text generation task, when generating the nth word, the model can only make predictions based on previously generated words and does not "read" the vectors corresponding to words (n+1) and beyond.

[0062] It should be understood that the masking mode of operator data is typically predetermined by the developer based on the characteristics of the task when developing the operator or model. For example, if the task is autoregressive language generation, a causal mask is typically chosen; if the task involves processing sequence data of inconsistent lengths, a padding mask is typically chosen. Furthermore, for some complex systems or projects, operator or model configuration information, including the masking mode, may be stored in a configuration file. The program can then read the masking mode settings from the configuration file at runtime.

[0063] The hardware parameters of a computing device refer to various parameters that describe the performance and characteristics of the computing device, such as the number of cores, memory size, and limited buffer size. Here, the computing device refers to an AI device, which can be a GPU, GPGPU, TPU, etc. For a GPU, the number of cores can be the number of SPCs, or processing units (SPCs) on the computing device; the limited buffer size can be GMB, GSM (Group-Shared Memory), etc., but this is not specifically limited in this embodiment of the present invention.

[0064] According to the batch size and mask mode of the above-mentioned operator data, as well as the hardware parameters of the computing device, the block size of the operator data in each dimension can be calculated. The dimensions here mainly include the batch dimension and the sequence length dimension. It should be understood that the batch size will affect the block size of the data in the batch dimension. When the batch size of the data is small, by splitting the data in the batch dimension, each processing unit on the computing device can be assigned to the corresponding computing task, thereby participating in the calculation and avoiding idle hardware resources. The mask mode will affect the block size of the data in the sequence length dimension. For example, in the causal mask mode, a smaller block size can be selected to reduce unnecessary calculations; in the padding mask mode, a relatively larger block size can be selected to improve the overall computing efficiency.

[0065] Specifically, the batch size of operator data can be calculated based on the batch size of the operator data and the number of processing units (SPCs) on the computing device. For example, if there are 16 SPCs on the GPU and the batch size of the operator data is 4, the data can be divided into 4 parts along the batch dimension. Each batch of data is computed in parallel on 4 SPCs, with each SPC sharing 1 / 4 of the computational workload for each batch of data.

[0066] The chunk size of operator data along the sequence length dimension can be determined based on the operator data masking mode and the buffer size limit on the computing device. For example, in the chunked computation of the Attention operator, the sequence length dimension of the Q matrix and the sequence length dimensions of the K and V matrices are typically split. The chunk size (denoted as chunk_K) along the sequence length dimension of the K and V matrices is consistent. This can be determined based on the masking mode. In the causal masking mode, a smaller chunk size, such as 1024, can be selected to reduce inefficient computations in the upper triangular portion of the score matrix. In other masking modes, 2048 can be selected. A larger chunk_K reduces the number of repeated loadings of K and V, thereby improving computational efficiency. Since attention computation is implemented through Q, K, and V operations, the chunks of Q, K, and V must be stored simultaneously in the buffer during computation. The chunk size (denoted as chunk_Q) of the Q matrix along the sequence length dimension can be calculated based on the size of chunk_K and the buffer size on the SPC.

[0067] It's understood that the block size of operator data in each dimension refers to the size of each block when the data is partitioned along the corresponding dimension (such as the batch dimension or sequence length dimension). By using a reasonable block size, operator data can be divided into multiple smaller data blocks and distributed to multiple processing units on the computing device for parallel computing, thereby improving computational efficiency.

[0068] Step 320: Segment the operator data based on the block size of the operator data in each dimension to obtain multiple data blocks;

[0069] Specifically, after determining the block size of the operator data in each dimension, the operator data can be segmented in the corresponding dimension according to the block size in each dimension. For example, assuming there are 16 SPCs on the GPU and the batch size of the operator data is 4, the traditional method is to directly assign 4 batches of data to 4 SPCs for processing, causing the remaining SPCs to be idle, resulting in a waste of hardware resources. In the embodiment of the present invention, the 4 batches of data can be evenly distributed to the 16 SPCs, that is, every 4 SPCs process one batch of data. In other words, one batch of data is calculated in parallel on the 4 SPCs, and each SPC shares 1 / 4 of the calculation task of each batch of data.

[0070] For example, assuming there are four batches of data, batch0, batch1, batch2, and batch3, by splitting on the batch dimension, batch0 data can be assigned to the four SPC processes SPC0 to SPC3, batch1 data can be assigned to the four SPC processes SPC4 to SPC7, batch2 data can be assigned to the four SPC processes SPC8 to SPC11, and batch3 data can be assigned to the four SPC processes SPC12 to SPC15.

[0071] For each batch of data, it can be further split along the sequence length dimension to obtain multiple data blocks, and the computing tasks corresponding to each data block can be assigned to each processing unit. For example, for the batch0 data mentioned above, assuming that it is split into 16 data blocks along the sequence length dimension, namely chunk0~chunk15, then these 16 data blocks can be evenly distributed to the four SPCs SPC0~SPC3 for processing. For example, the computing tasks corresponding to the four data blocks chunk0~chunk3 are assigned to SPC0, the computing tasks corresponding to the four data blocks chunk4~chunk7 are assigned to SPC1, the computing tasks corresponding to the four data blocks chunk8~chunk11 are assigned to SPC2, and the computing tasks corresponding to the four data blocks chunk12~chunk15 are assigned to SPC3.

[0072] It can be understood that multiple data blocks refer to multiple subsets of data obtained by partitioning the original operator data according to the block size in each dimension. Each data block contains a portion of the original data, and the computational tasks corresponding to each data block can be performed independently, for example, assigned to different SPCs on a GPU for parallel computing. Although data blocks are subsets of the original data, they together constitute a complete representation of the original data. After the computation is complete, the results of each data block can be combined to obtain the final computational result for the original data.

[0073] Step 330: Allocate computing tasks corresponding to the multiple data blocks to multiple processing units on the computing device, and execute the computing tasks corresponding to the multiple data blocks in parallel based on the multiple processing units.

[0074] Specifically, in GPU-based Attention operator optimization scenarios, the computational tasks corresponding to multiple data blocks involve performing computational operations related to the Attention mechanism on each data block, including calculations of the score matrix, attention weight matrix, and weighted summation. After the operator data is partitioned, the computational tasks corresponding to each data block are assigned to a processing unit on the computing device, allowing for parallel computation across these processing units.

[0075] Specifically, taking the GPU as an example, the GPU contains multiple SPCs, and each SPC can be regarded as a processing unit. When assigning computing tasks, the computing tasks of each data block can be assigned to one or more thread blocks. Here, the thread block is the basic unit of parallel execution on the GPU. A thread block contains multiple threads. All threads in a thread block can access the same shared memory, which can quickly synchronize and communicate. Different thread blocks can be synchronized or asynchronous depending on the task. The thread blocks corresponding to all data blocks are combined into one or more thread grids. The grid is a collection of thread blocks. The GPU can simultaneously schedule multiple thread blocks to execute on different SPCs. In this way, the computing tasks of multiple data blocks can be assigned to multiple SPCs on the GPU for parallel execution.

[0076] It is understandable that if the size and computational complexity of the data blocks are relatively uniform, a uniform allocation strategy can be used to evenly distribute the data blocks to different SPCs. For example, for batch0 data calculated in parallel on SPC0-SPC3, the 16 data blocks corresponding to this batch of data can be evenly distributed to the four SPCs SPC0-SPC3, with each SPC responsible for processing four data blocks. For data blocks with uneven computational complexity, a dynamic allocation strategy can be used. The GPU can dynamically allocate data blocks to SPCs with lighter loads based on the current load conditions of each SPC, achieving a balanced distribution of computing tasks and thus balancing the load on each SPC.

[0077] Furthermore, each SPC includes multiple computing units, and each computing unit includes multiple cores, which can execute different threads. When the computational tasks of multiple data blocks are assigned to different thread blocks, these thread blocks can be executed in parallel on different SPCs. Each SPC can simultaneously schedule multiple thread blocks and utilize its multiple internal cores for computation. For example, after the computational tasks corresponding to the 16 data blocks after batch0 data is split are evenly distributed to the four SPCs SPC0 to SPC3, each SPC is responsible for processing four data blocks. For each SPC, it can use its multiple internal cores to process the computational tasks of these four data blocks, thereby obtaining the corresponding computational results.

[0078] The method provided by the embodiment of the present invention can quickly determine the block size of the operator data in various dimensions such as the batch dimension and the sequence length dimension based on the batch size of the operator data, the mask mode and the hardware parameters of the computing device. This step fully takes into account the data characteristics and hardware capabilities, and lays the foundation for subsequent efficient processing. By dividing the operator data according to the determined block sizes of each dimension, the operator data can be reasonably broken down into small units that are convenient for parallel processing, and finally the computing tasks corresponding to these divided data blocks are reasonably allocated to multiple processing units on the computing device for parallel execution. By dividing the data in the batch dimension and the sequence length dimension at the same time, the present invention enables multiple processing units on the computing device to participate in the calculation at the same time even in the scenario of small data batches, thereby avoiding idle hardware resources, making full use of the parallel computing capabilities of the hardware, significantly improving the utilization rate of hardware resources, and thus greatly improving the overall computing efficiency, and achieving efficient data processing and calculation.

[0079] Based on any of the above embodiments, step 310 specifically includes:

[0080] Step 311 : Determine the block size of the operator data in the batch dimension based on the batch size of the operator data and the number of processing units in the hardware parameters.

[0081] It should be noted that the number of processing units in the hardware parameters refers to the number of processing units on the GPU (or other computing device) that can be used to execute computing tasks in parallel. These processing units can be SPCs on the GPU, so the number of processing units can specifically be the number of SPCs. The purpose of step 311 is to determine the block size of the operator data in the batch dimension based on the batch size of the operator data and the number of processing units in the hardware parameters. The core idea is to evenly distribute computing tasks to each processing unit by reasonably dividing the batch dimension, avoiding idle processing units and thus improving computing efficiency. Step 311 specifically includes:

[0082] Step 3111, calculating the greatest common divisor of the batch size and the number of processing units;

[0083] Step 3112: Determine the block size of the operator data in the batch dimension based on the greatest common divisor and the number of processing units.

[0084] Specifically, in step 3111, by calculating the greatest common divisor of the batch size and the number of processing units, a suitable block granularity can be found so that the computing tasks are distributed as evenly as possible on each processing unit. Assuming that the batch size of the operator data is batch_size and the number of processing units on the computing device is core_num, the greatest common divisor of the two can be calculated using the following formula:

[0085] g_num = gcd(batch_size, core_num)

[0086] Where g_num is the greatest common divisor of the batch size and the number of processing units, and the gcd() function is used to calculate the greatest common divisor of two or more integers.

[0087] After calculating the greatest common divisor g_num, the number of batches processed on each processing unit, nPerCore, can be determined based on the greatest common divisor g_num and the number of processing units core_num, thereby obtaining the block size in the batch dimension. Specifically, the number of batches processed on each processing unit can be calculated using the following formula:

[0088] nPerCore = batch_size / g_num

[0089] At the same time, based on the number of processing units core_num and the greatest common divisor g_num, we can calculate the number of processing unit groups for parallel computing as group_num = core_num / g_num. In this way, nPerCore batches of data are computed in parallel on group_num processing units, with each processing unit sharing the computational workload of g_num / core_num for each batch of data.

[0090] For example, assuming the batch size of operator data is batch_size = 4 and the number of SPCs on the GPU is core_num = 16, the greatest common divisor g_num = 4 can be calculated according to the above formula. Therefore, the number of batches processed by each SPC is nPerCore = 1. One batch of data will be calculated in parallel on group_num = 4 SPCs, and each SPC will share 1 / 4 of the computational workload for each batch of data. It should be understood that since one batch of data is calculated in parallel on four SPCs, the above nPerCore = 1 means that each SPC processes not a complete batch of data, but only a portion of the batch (i.e., 1 / 4).

[0091] It can be understood that through the above steps, it can be ensured that the batch dimension segmentation can fully utilize hardware resources, avoid idle processing units, and thus improve computing efficiency.

[0092] Step 312: Determine the block size of the operator data in the sequence length dimension based on the mask mode of the operator data and the buffer size in the hardware parameters.

[0093] It's important to note that the buffer size in hardware parameters refers to the size of the memory area used for temporary data storage on the GPU or other parallel computing devices. During attention computations, data needs to be frequently accessed and manipulated. To improve computational efficiency, buffers are often used to cache intermediate results or frequently accessed data. These buffers can be part of global memory, GMB, GSM, and other memory locations. The buffer size affects data access latency and bandwidth, and thus the performance of the entire computational task.

[0094] The purpose of step 312 is to determine the chunk size of the operator data along the sequence length dimension based on the operator data mask pattern and the buffer size in the hardware parameters. During the Attention operator calculation process, the chunk size along the sequence length dimension can be divided into the row-wise chunk size (i.e., the chunk size of the Q matrix along the sequence length dimension, denoted as chunk_Q) and the column-wise chunk size (i.e., the chunk size of the K / V matrix along the sequence length dimension, denoted as chunk_K). The core concept is to appropriately split the sequence length dimension based on the mask pattern and buffer size constraints to optimize data access and computation efficiency.

[0095] It's understandable that during the Attention score matrix calculation, the ordinate corresponds to Q and the abscissa corresponds to K. Therefore, the row-wise block size refers to splitting the Q matrix into multiple blocks along the sequence length dimension, with each block containing a certain number of rows (i.e., query vectors). This allows for block-based data processing during the calculation, reducing data loading and storage overhead. The column-wise block size refers to splitting the K matrix into multiple blocks along the sequence length dimension, with each block containing a certain number of columns (i.e., key vectors). This block size allows you to control the number of query and key vectors processed in each calculation to accommodate buffer constraints.

[0096] Furthermore, the block size in the sequence length dimension includes the block size in the row direction and the block size in the column direction. Accordingly, step 312 specifically includes:

[0097] Step 3121: Determine the block size of the operator data in the column direction based on the mask pattern of the operator data;

[0098] Step 3122: Determine the block size of the operator data in the row direction based on the buffer size and the block size of the operator data in the column direction.

[0099] Specifically, the mask mode affects the range of valid data in attention calculations. For example, in causal mask mode, when calculating the attention for the current position, only the data for the previous position is considered, not the data for the subsequent position. Therefore, it is necessary to determine the appropriate column-wise block size based on the mask mode to reduce ineffective calculations.

[0100] Figure 4 is a schematic diagram of the causal mask provided by the present invention, such as Figure 4As shown in the Attention score matrix, the vertical axis (rows) corresponds to the Q vector, and the horizontal axis (columns) corresponds to the K vector. The gray part indicates which tokens in the K vector each token in the Q vector can see. In other words, each Q token and which K tokens need to calculate the attention weight. Figure 5 As can be seen in , the score matrix corresponding to the causal mask is usually a lower triangular matrix. The elements on the diagonal and below the diagonal of the matrix retain the actual calculated attention scores, while the elements above the diagonal need to be set to a minimum value or negative infinity. In this way, in the subsequent Softmax calculation, the weights corresponding to the future information positions will approach 0, thereby ensuring the causality of the final calculation result.

[0101] It is understandable that in the causal mask mode, the data in the upper triangular part of the score matrix will not participate in subsequent calculations. Therefore, in order to optimize the Attention calculation, the calculation of invalid data in the score matrix is ​​usually skipped, that is, the MMA operation of the Q block and K block corresponding to the upper triangular part is skipped. In this mode, in order to avoid invalid calculations in the upper triangular area, a smaller segmentation granularity is usually selected as the block size in the column direction. For example, chunk_K can be selected as 1024, because a smaller block size can better limit the calculation range and reduce unnecessary calculations. For other mask modes, there is no need to strictly limit the calculation range like the causal mask mode, so a relatively large segmentation granularity can be selected as the block size in the column direction. For example, in other mask modes, chunk_K can be selected as 2048. Such a larger block size can reduce the number of repeated loadings of K and V, thereby improving calculation efficiency, but the buffer limitation also needs to be considered.

[0102] In attention calculations, the rows of the Q matrix are typically used as the outer loop. This means that a larger row-wise block size can reduce repeated transfers of the K / V matrix, improving computational efficiency. However, the row-wise block size is also limited by the buffer size, as it is necessary to ensure that the relevant K / V matrix data can be stored in the buffer when processing a row-wise block. Therefore, the largest chunk_Q can be selected based on the buffer size and chunk_K.

[0103] Here, we can calculate the maximum number of Q matrix rows that can be stored in the buffer, i.e., the row-wise chunk size (chunk_Q), based on the buffer size and the already determined column-wise chunk size (chunk_K). While meeting the buffer limit, try to choose a larger row-wise chunk size to reduce repeated K / V matrix transfers. For example, if the buffer is large enough, we can choose a larger chunk_Q value to process more Q matrix rows in a single calculation, thereby reducing the number of K / V matrix loads.

[0104] In the embodiment of the present invention, by considering the limitations of the mask pattern and buffer size, the block sizes in the row and column directions along the sequence length dimension are reasonably determined to optimize the data access and computational efficiency of the Attention calculation.

[0105] Based on any of the above embodiments, in step 330, allocating computing tasks corresponding to the multiple data blocks to multiple processing units on the computing device includes:

[0106] Based on a preset allocation strategy, computing tasks corresponding to the multiple data blocks are allocated to multiple processing units on the computing device. The preset allocation strategy is any one of an equal distribution strategy and a load balancing strategy.

[0107] Specifically, when assigning computing tasks corresponding to multiple data blocks to multiple processing units on a computing device (such as multiple SPCs on a GPU), either an equal distribution strategy or a load balancing strategy can be adopted. Here, an equal distribution strategy refers to a strategy that simply and evenly distributes computing tasks across multiple SPCs, such as by evenly distributing tasks based on the number of tasks. The core idea is to ensure that each processing unit takes on roughly the same number of computing tasks, ideally achieving balanced computing load across all processing units and preventing some processing units from being idle.

[0108] For example, suppose there are computing tasks corresponding to 16 data chunks (e.g., chunk0-chunk15) to be assigned to four SPCs (e.g., SPC0-SPC4). Based on the equal distribution strategy, the number of tasks assigned to each SPC is calculated to be 4. Therefore, the computing tasks corresponding to chunks 0-chunk 3 can be assigned to SPC 0, the computing tasks corresponding to chunks 4-chunk 7 can be assigned to SPC 1, the computing tasks corresponding to chunks 8-chunk 11 can be assigned to SPC 2, and the computing tasks corresponding to chunks 12-chunk 15 can be assigned to SPC 3.

[0109] The above allocation strategy is relatively simple to implement and can quickly assign tasks to various processing units. However, this even-sharing strategy fails to account for the varying computational loads of different tasks. If some tasks are computationally intensive while others are less so, even if the number of tasks is evenly distributed, some SPCs may be overloaded while others are lightly loaded, failing to achieve true load balancing. To address this issue, embodiments of the present invention also propose a load-balanced allocation strategy.

[0110] Load balancing is a more flexible and intelligent task allocation strategy that dynamically distributes computing tasks based on the real-time load of each SPC. Its purpose is to ensure that the load on each SPC is as evenly distributed as possible, avoiding situations where some SPCs are idle while others are busy, thereby improving the computing efficiency and resource utilization of the entire computing device.

[0111] Specifically, to implement a load-balanced distribution strategy, a global task queue can be maintained, containing the computational tasks corresponding to all pending data blocks. The load of each SPC is monitored in real time. For example, the load is assessed by counting metrics such as the number of tasks currently executing, the number of completed tasks, and the estimated remaining computation time. When an SPC completes its current task and becomes idle, a task is selected from the task queue and assigned to it. Task selection is based on factors such as task priority and computational complexity to ensure optimal task allocation.

[0112] For example, assume there are five computing tasks corresponding to data blocks (Tasks A, B, C, D, and E), and two SPCs (SPC1 and SPC2). Initially, the task queue contains Tasks A, B, C, D, and E, and SPC1 and SPC2 are both idle. Assume that Tasks A and B are assigned to SPC1 and SPC2 for execution. During execution, it is observed that SPC1 completes Task A faster than SPC2 completes Task B. After SPC1 completes Task A, Task C is selected from the task queue and assigned to SPC1. SPC2 continues to execute Task B until it completes it, at which point Task D is selected from the task queue. If Task E is still in the task queue at this point, Task E is assigned to the idle SPC after SPC1 or SPC2 completes its current task.

[0113] As you can see, the load balancing strategy can dynamically adjust based on the real-time load of the SPCs, ensuring that the loads of each SPC are as evenly distributed as possible, thereby improving the overall performance and resource utilization of the computing device. This is especially true when tasks vary greatly in computational complexity, allowing each SPC to better utilize its computing power.

[0114] Based on any of the above embodiments, in step 330, allocating computing tasks corresponding to the multiple data blocks to multiple processing units on the computing device includes:

[0115] When the mask mode of the operator data is a causal mask, the computing tasks corresponding to the data blocks are allocated to the processing units in an alternating manner of positive order allocation and reverse order allocation based on the task numbers corresponding to the data blocks.

[0116] It should be noted that the Attention operator typically skips invalid upper triangular computations in causal mask mode. This splits the score matrix row-wise and assigns the computational tasks corresponding to each split block to each SPC for processing. This can easily lead to an imbalanced computational load across the SPCs, failing to achieve optimal performance. Although load balancing strategies can dynamically adjust based on the real-time load of the SPCs to ensure that the load across the SPCs is as balanced as possible, this strategy is relatively complex to implement and requires additional load monitoring and task scheduling mechanisms, increasing system overhead. Furthermore, dynamic allocation can lead to delays in task allocation due to the need for real-time load monitoring and decision-making.

[0117] In this regard, considering that in the causal mask mode, the amount of computing tasks corresponding to each data block increases along the row direction, in order to make the computing tasks allocated to each SPC more balanced while avoiding the overhead and delay introduced by dynamic allocation, an embodiment of the present invention proposes a new static allocation method, that is, numbering the tasks of each data block along the row direction, and according to these numbers, the computing tasks corresponding to each data block are allocated to the corresponding SPC in an alternating positive and reverse order.

[0118] Specifically, taking the allocation of the above 16 data blocks (i.e., chunks) to the four SPCs SPC0~SPC3 as an example, the above alternating positive and reverse order allocation strategy can be implemented through the following steps: First, set the initial allocation direction to positive order (i.e., from SPC0 to SPC3); then, traverse all chunks, and for each chunk, determine the SPC to be allocated based on the current allocation direction; under the positive allocation direction, the task number corresponding to the chunk (denoted as chunk_id) is modulo the number of SPCs (denoted as num_spcs) to obtain the corresponding SPC number, for example, chunk_id % num_spcs; under the reverse allocation direction, the task number corresponding to the chunk is modulo the number of SPCs, and the corresponding SPC number is calculated using (num_spcs - 1) - (chunk_id % num_spcs), so that reverse order allocation can be achieved starting from the last SPC. After each allocation round (i.e., after processing num_spcs chunks), the allocation direction switches: after forward allocation is complete, it switches to reverse allocation; after reverse allocation is complete, it switches back to forward allocation. This process repeats until all chunks have been allocated. This alternating forward and reverse allocation strategy achieves balanced load distribution while eliminating the need for additional load monitoring and task scheduling mechanisms, reducing overhead.

[0119] Figure 5 It is a schematic diagram of the computing task allocation provided by the present invention, such as Figure 5 As shown in the figure, the lower triangular matrix is ​​the score matrix for Attention calculation. This score matrix can be calculated based on a batch of input sequences. Assuming that the computation task corresponding to this score matrix is ​​assigned to four SPCs for parallel computation, after splitting the score matrix row-wise, each SPC is responsible for processing one or more blocks, each consisting of one or more rows. Here, since the score matrix is ​​obtained by performing an MMA operation on the Q and K matrices, the rows of the score matrix correspond to Q vectors and the columns correspond to K vectors. Splitting the score matrix row-wise means splitting the Q matrix along the sequence length dimension, resulting in multiple Q blocks (i.e., chunks Q). It should be noted that the chunk_K size selected in the causal mode is 1024. If the sequence length is less than 1024, the K matrix does not need to be split, so the score matrix only needs to be split row-wise.

[0120] Assume that the score matrix is ​​split into 16 blocks along the row direction. That is, the Q matrix is ​​split along the sequence length dimension, resulting in 16 chunks Q. The tasks corresponding to these 16 chunks Q are numbered 0 to 15. For each chunk Q, the corresponding computation task is to perform attention calculations based on the chunk Q and the K and V matrices. Different allocation strategies can be used to allocate the computation tasks corresponding to these 16 chunk Qs to the four SPCs (SPC0 to SPC3) for parallel processing.

[0121] For example, Figure 5 As shown in allocation method 1 in the figure, the computing tasks corresponding to these 16 chunkQs can be evenly distributed to each SPC, that is, computing tasks numbered 0 to 3 are assigned to SPC0, computing tasks numbered 4 to 7 are assigned to SPC1, computing tasks numbered 8 to 11 are assigned to SPC2, and computing tasks numbered 12 to 15 are assigned to SPC3. It should be understood that although this allocation method can quickly assign tasks to each SPC, because the upper triangular portion of the score matrix is ​​invalid, the amount of valid data in the blocks processed by different SPCs is different. For example, the valid data corresponding to number 15 is far more than the valid data corresponding to number 0. This will result in the computational workload of the task assigned to SPC0 being smaller, while the computational workload of the task assigned to SPC3 is larger, resulting in an unbalanced computational load among the SPCs, which in turn reduces overall computational efficiency and fails to achieve optimal performance.

[0122] For example, Figure 5 As shown in allocation method 2 in the figure, by traversing all chunkQs, the computing tasks corresponding to these chunkQs are assigned to each SPC in turn. That is, computing task numbered 0 is assigned to SPC0, computing task numbered 1 is assigned to SPC1, computing task numbered 2 is assigned to SPC2, computing task numbered 3 is assigned to SPC3, computing task numbered 4 is assigned to SPC0, and so on, until all chunkQs are traversed. It should be understood that compared with allocation method 1, the computing task load corresponding to allocation method 2 is relatively balanced, but it still does not achieve optimal load balancing distribution.

[0123] In this regard, the embodiment of the present invention proposes the following for the causal mask mode: Figure 5The allocation strategy shown in allocation method 3 adopts an alternating forward and reverse order allocation strategy to achieve optimal load balancing. Specifically, all chunkQs are traversed, and for each chunkQ, the SPC to be allocated is determined based on the current allocation direction. First, allocate in positive order. Take the SPC number modulo the task number corresponding to each chunkQ (i.e., chunk_id % num_spcs) to calculate the corresponding SPC number. For example, for chunkQ 0, since chunk_id = 0 and num_spcs = 4, the corresponding SPC number is 0, and the computing task numbered 0 is assigned to SPC0; for chunkQ 1, since chunk_id = 1 and num_spcs = 4, the calculated SPC number is 1, and the computing task numbered 1 is assigned to SPC1; for chunkQ 2, since chunk_id = 2 and num_spcs = 4, the calculated SPC number is 2, and the computing task numbered 2 is assigned to SPC2; for chunkQ 3, since chunk_id = 3 and num_spcs = 4, the calculated SPC number is 3, and the computing task numbered 3 is assigned to SPC3.

[0124] Since the task allocation for SPC0 to SPC3 has completed, it's time to switch the allocation direction from forward to reverse order. In reverse order, the corresponding SPC number can be calculated using the formula (num_spcs - 1) - (chunk_id % num_spcs). For example, for chunkQ 4, since chunk_id = 4 and num_spcs = 4, the calculated SPC number is 3, meaning that the compute task numbered 4 is assigned to SPC3, and so on. After each round of allocation (i.e., all four chunkQ tasks have been assigned), the allocation direction is switched, and the above steps are repeated until all chunkQ tasks have been assigned. This alternating forward and reverse order allocation strategy achieves balanced load distribution.

[0125] Based on any of the above embodiments, in step 330, the parallel execution of computing tasks corresponding to the multiple data blocks based on the multiple processing units includes:

[0126] For each processing unit, a data block related to the computing task of the processing unit is preloaded from an off-chip memory to a target memory area, and the data block is loaded from the target memory area to execute the computing task corresponding to the data block. The target memory area is any one of a non-uniform memory access area and an on-chip memory.

[0127] It should be noted that in order to further optimize operator performance and improve overall computing efficiency, if the buffer can carry the corresponding data block size through calculation, the corresponding data movement can be further optimized through preloading. That is, by loading data in advance, the data waiting time during the calculation process can be reduced, thereby further improving computing performance.

[0128] Specifically, for each processing unit (such as an SPC), while executing the current computing task, the data block corresponding to the next computing task can be pre-loaded from the off-chip memory into the target memory area to prepare the relevant data for the subsequent computing task. Here, for each SPC, the data block associated with a computing task of the SPC refers to the data block that the processing unit needs to access and process when executing the computing task. For example, in the Attention calculation, if an SPC is responsible for processing a query matrix block (Q block), then the data blocks associated with the data processing task of the SPC include the query matrix block and all key matrix blocks (K blocks) and value matrix blocks (V blocks) that need to be calculated with it.

[0129] It is understandable that after the operator data is split, the corresponding multiple data blocks are usually stored in an off-chip memory, such as in the UMA (Uniform Memory Access) area of ​​the HBM. For each processing unit, after determining the data blocks related to the computing task of the processing unit, these data blocks can be preloaded from the UMA area of ​​the HBM to the target memory area corresponding to the processing unit for subsequent data loading and processing. Once the relevant data blocks are preloaded into the target memory area, the processing unit can load the corresponding data blocks from the area for calculation. Here, uniform memory access means that each processing unit can access the same memory area, which makes it easy for multiple SPCs to access the UMA area of ​​the HBM to load data blocks at the same time, causing bandwidth competition problems, so that the corresponding SPCs become idle due to waiting for data loading and transmission, affecting the overall computing efficiency.

[0130] To this end, an embodiment of the present invention proposes to pre-load each data block related to the computing task of each processing unit into the corresponding target memory area. Here, the target memory area can be a non-uniform memory access (NUMA) area in HBM, which enables each processing unit to only access its corresponding target memory area and cannot access the target memory area corresponding to other processing units. In an embodiment of the present invention, by pre-loading each data block related to the computing task of each processing unit into the corresponding non-uniform memory access target memory area, the subsequent data loading operations of different processing units can be made independent of each other, thereby effectively avoiding bandwidth competition during the data loading process, ensuring that the data loading process can be carried out stably and efficiently, and thus improving the overall computing efficiency.

[0131] Alternatively, the target memory area can be on-chip memory within each processing unit, such as GMB. Off-chip memory is typically located outside the chip, with data transmission distances greater than the processing unit, resulting in higher access latency. On-chip memory, on the other hand, is located within the chip, physically closer to the processing unit, and offers faster data access. By preloading data into the target memory area, the processing unit can quickly retrieve data directly from that area, significantly reducing data access time and thus improving the efficiency of computing tasks.

[0132] The operator optimization device provided by the present invention is described below. The operator optimization device described below and the operator optimization method described above can refer to each other.

[0133] Based on any of the above embodiments, Figure 6 This is a schematic diagram of the structure of the operator optimization device provided by the present invention. Figure 6 As shown, the device includes:

[0134] A block determination module 610 is configured to determine the block size of the operator data in various dimensions, including a batch dimension and a sequence length dimension, based on the batch size and mask mode of the operator data and hardware parameters of the computing device.

[0135] A data segmentation module 620 is configured to segment the operator data based on the block size of the operator data in each dimension to obtain multiple data blocks;

[0136] The task execution module 630 is used to assign the computing tasks corresponding to the multiple data blocks to the multiple processing units on the computing device, and execute the computing tasks corresponding to the multiple data blocks in parallel based on the multiple processing units.

[0137] The device provided by the embodiment of the present invention can quickly determine the block size of the operator data in various dimensions such as the batch dimension and the sequence length dimension based on the batch size of the operator data, the mask mode and the hardware parameters of the computing device. This step fully takes into account the data characteristics and hardware capabilities, and lays the foundation for subsequent efficient processing. By dividing the operator data according to the determined block size of each dimension, the operator data can be reasonably broken down into small units that are convenient for parallel processing, and finally the computing tasks corresponding to these divided data blocks are reasonably allocated to multiple processing units on the computing device for parallel execution. By dividing the data in the batch dimension and the sequence length dimension at the same time, the present invention enables multiple processing units on the computing device to participate in the calculation at the same time even in the scenario of small data batches, thereby avoiding idle hardware resources, making full use of the parallel computing capabilities of the hardware, significantly improving the utilization rate of hardware resources, and thus greatly improving the overall computing efficiency, and achieving efficient data processing and calculation.

[0138] Based on any of the above embodiments, the block determination module 610 includes:

[0139] A first determining subunit is configured to determine a block size of the operator data in the batch dimension based on the batch size of the operator data and the number of processing units in the hardware parameters;

[0140] The second determining subunit is configured to determine a block size of the operator data in the sequence length dimension based on a mask pattern of the operator data and a buffer size in the hardware parameters.

[0141] Based on any of the foregoing embodiments, the first determining subunit is specifically configured to:

[0142] Calculating the greatest common divisor of the batch size and the number of processing units;

[0143] Based on the greatest common divisor and the number of processing units, a block size of the operator data in the batch dimension is determined.

[0144] Based on any of the foregoing embodiments, the block size in the sequence length dimension includes a block size in a row direction and a block size in a column direction, and the second determining subunit is specifically configured to:

[0145] determining a block size of the operator data in a column direction based on a mask pattern of the operator data;

[0146] The block size of the operator data in the row direction is determined based on the buffer size and the block size of the operator data in the column direction.

[0147] Based on any of the above embodiments, the task execution module 630 is specifically configured to:

[0148] Based on a preset allocation strategy, computing tasks corresponding to the multiple data blocks are allocated to multiple processing units on the computing device. The preset allocation strategy is any one of an equal distribution strategy and a load balancing strategy.

[0149] Based on any of the above embodiments, the task execution module 630 is specifically configured to:

[0150] When the mask mode of the operator data is a causal mask, the computing tasks corresponding to the data blocks are allocated to the processing units in an alternating manner of positive order allocation and reverse order allocation based on the task numbers corresponding to the data blocks.

[0151] Based on any of the above embodiments, the task execution module 630 is specifically configured to:

[0152] For each processing unit, a data block related to the computing task of the processing unit is preloaded from an off-chip memory to a target memory area, and the data block is loaded from the target memory area to execute the computing task corresponding to the data block. The target memory area is any one of a non-uniform memory access area and an on-chip memory.

[0153] Figure 7 An example of a physical structure diagram of an electronic device is shown below. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the operator optimization method, which includes: determining the block size of the operator data in each dimension based on the batch size and mask mode of the operator data and the hardware parameters of the computing device, the dimensions including the batch dimension and the sequence length dimension; dividing the operator data based on the block size of the operator data in each dimension to obtain multiple data blocks; allocating the computing tasks corresponding to the multiple data blocks to multiple processing units on the computing device, and executing the computing tasks corresponding to the multiple data blocks in parallel based on the multiple processing units.

[0154] Furthermore, the logic instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the relevant art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0155] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the operator optimization method provided by the above methods, which includes: based on the batch size and mask mode of the operator data, and the hardware parameters of the computing device, determining the block size of the operator data in each dimension, the dimensions including the batch dimension and the sequence length dimension; based on the block size of the operator data in each dimension, dividing the operator data to obtain multiple data blocks; allocating the computing tasks corresponding to the multiple data blocks to multiple processing units on the computing device, and executing the computing tasks corresponding to the multiple data blocks in parallel based on the multiple processing units.

[0156] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the operator optimization method provided by the above-mentioned methods, the method comprising: determining the block size of the operator data in each dimension based on the batch size and mask mode of the operator data, and the hardware parameters of the computing device, the dimensions including the batch dimension and the sequence length dimension; based on the block size of the operator data in each dimension, dividing the operator data to obtain multiple data blocks; allocating the computing tasks corresponding to the multiple data blocks to multiple processing units on the computing device, and executing the computing tasks corresponding to the multiple data blocks in parallel based on the multiple processing units.

[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0158] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An operator optimization method, characterized in that: include: Determining, based on the batch size and masking pattern of the operator data and hardware parameters of the computing device, the chunking size of the operator data in each dimension, including the batch dimension and the sequence length dimension; Based on the block size of the operator data in each dimension, the operator data is segmented to obtain multiple data blocks; Allocating computing tasks corresponding to the multiple data blocks to multiple processing units on the computing device, and executing the computing tasks corresponding to the multiple data blocks in parallel based on the multiple processing units; The step of determining the block size of the operator data in each dimension based on the batch size and mask mode of the operator data and the hardware parameters of the computing device includes: Determining a block size of the operator data in the batch dimension based on the batch size of the operator data and the number of processing units in the hardware parameters; Based on the mask mode of the operator data and the buffer size in the hardware parameters, a block size of the operator data in the sequence length dimension is determined.

2. The operator optimization method according to claim 1, characterized in that: The determining, based on the batch size of the operator data and the number of processing units in the hardware parameters, the block size of the operator data in the batch dimension includes: Calculating the greatest common divisor of the batch size and the number of processing units; Based on the greatest common divisor and the number of processing units, a block size of the operator data in the batch dimension is determined.

3. The operator optimization method according to claim 1, characterized in that: The block size in the sequence length dimension includes a block size in a row direction and a block size in a column direction, and determining the block size of the operator data in the sequence length dimension based on the mask mode of the operator data and the buffer size in the hardware parameter includes: determining a block size of the operator data in a column direction based on a mask pattern of the operator data; The block size of the operator data in the row direction is determined based on the buffer size and the block size of the operator data in the column direction.

4. The operator optimization method according to claim 1, characterized in that: Allocating computing tasks corresponding to the plurality of data blocks to a plurality of processing units on the computing device includes: Based on a preset allocation strategy, computing tasks corresponding to the multiple data blocks are allocated to multiple processing units on the computing device. The preset allocation strategy is any one of an equal distribution strategy and a load balancing strategy.

5. The operator optimization method according to claim 1, characterized in that: Allocating computing tasks corresponding to the plurality of data blocks to a plurality of processing units on the computing device includes: When the mask mode of the operator data is a causal mask, the computing tasks corresponding to the data blocks are allocated to the processing units in an alternating manner of positive order allocation and reverse order allocation based on the task numbers corresponding to the data blocks.

6. The operator optimization method according to any one of claims 1 to 5, characterized in that: The parallel execution of computing tasks corresponding to the multiple data blocks based on the multiple processing units includes: For each processing unit, a data block related to the computing task of the processing unit is preloaded from an off-chip memory to a target memory area, and the data block is loaded from the target memory area to execute the computing task corresponding to the data block. The target memory area is any one of a non-uniform memory access area and an on-chip memory.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the operator optimization method according to any one of claims 1 to 6 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the operator optimization method according to any one of claims 1 to 6 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the operator optimization method according to any one of claims 1 to 6 is implemented.

Images