Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of large models.The method comprises the steps that an input matrix is divided into a plurality of sub-matrixes, and the number and the position index of each sub-matrix are determined based on a grouping staggering method; executing corresponding sub-matrix operation based on at least one calculation thread block in the thread block network, and transmitting an operation result of the corresponding sub-matrix based on at least one communication thread block corresponding to the at least one calculation thread block; based on the serial number sequence corresponding to each sub-matrix, storing the operation result of each sub-matrix in a cache, and based on the serial number and the position index corresponding to each sub-matrix and the operation results of the sub-matrixes sequentially stored in the cache, determining an output matrix; therefore, the communication thread blocks are arranged between the calculation thread blocks, so that the calculation thread blocks can synchronously perform sub-matrix operation in the communication process of the communication thread blocks, and the resource utilization rate of the graphics processor is improved.

Description

Technical Field

[0001] This disclosure relates to the field of large model technology, and in particular to a data processing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) based on the Transformer architecture have made breakthroughs in fields such as natural language processing, computer vision, and multimodal understanding. In pursuit of stronger expressive power and generalization performance, the number of model parameters has rapidly increased from hundreds of millions in the early stages to tens or even hundreds of billions. This dramatic increase in parameter size means that the memory capacity and computing power of a single GPU can no longer meet the demands of model inference, making multi-GPU collaborative computing the mainstream solution for deploying large model inference.

[0003] In multi-GPU parallel inference, Tensor Parallelism (TP) is one of the most commonly used parallel strategies. This strategy splits the model's weight matrix along a specific dimension and distributes it across multiple GPUs. Each GPU independently performs local matrix multiplication, and then the computational results of each GPU are aggregated through a collective communication operation (such as AllReduce) to obtain the final output. Specifically, current mainstream large language models generally adopt an attention mechanism plus output mapping matrix multiplication (OutputGEMM) structure, as well as a two-layer matrix multiplication structure in a feedforward network (FFN). In tensor parallel mode, both of these structures require an AllReduce communication operation after the second layer of matrix multiplication to aggregate the partial sums from each GPU, i.e., the execution flow is: [Attention / GEMM1] → [GEMM2] → [AllReduce].

[0004] Currently, mainstream inference engines (such as vLLM and SGLang) generally adopt a serial execution mode based on Compute Unified Device Architecture (CUDA) streams: computation operators (such as matrix multiplication GEMM) and communication operators (such as AllReduce) are submitted to the same CUDA stream and executed sequentially. Each operator must wait for the previous operator to finish executing completely before it can start. This strict sequential execution mechanism cannot achieve overlap between computation and communication. The computation results of GEMM2 need to be fully ready before AllReduce communication can start, and subsequent computation layers cannot start until AllReduce communication is completed, forming a pipeline bubble of "computation waiting for communication". Since communication operators usually have very low usage of GPU computing units and memory bandwidth, this serial working mode reduces GPU resource utilization, causing inference engines to face performance bottlenecks under high load scenarios. Summary of the Invention

[0005] This disclosure provides a data processing method, apparatus, electronic device, and storage medium to at least solve the above-mentioned technical problems existing in the prior art.

[0006] According to a first aspect of this disclosure, a data processing method is provided, wherein a thread block network includes computation thread blocks and communication thread blocks, the communication thread blocks being disposed between the computation thread blocks, the computation thread blocks being used to perform submatrix operations, and the communication thread blocks being used to transmit the results of the submatrix operations, the method comprising:

[0007] The input matrix is ​​divided into multiple submatrices, and the number and position index of each submatrix are determined based on the grouping and interleaving method. The corresponding submatrix operation is performed based on at least one computation thread block in the thread block network, and the operation result of the corresponding submatrix is ​​transmitted based on at least one communication thread block corresponding to the at least one computation thread block; The operation results of each submatrix are stored in the cache according to the numbering order corresponding to each submatrix. The output matrix is ​​determined based on the number and position index corresponding to each submatrix and the operation results of the submatrix stored sequentially in the cache. The output matrix is ​​used to determine the processing result of the data to be processed corresponding to the input matrix. During the process of transmitting the operation result of the corresponding sub-matrix in the at least one communication thread block, other computing thread blocks in the thread block network perform the corresponding sub-matrix operation until the output matrix corresponding to the input matrix is ​​obtained.

[0008] According to a second aspect of this disclosure, a data processing apparatus is provided, wherein a thread block network includes computation thread blocks and communication thread blocks, the communication thread blocks being disposed between the computation thread blocks, the computation thread blocks being used to perform submatrix operations, and the communication thread blocks being used to transmit the results of the submatrix operations, the apparatus comprising: The partitioning unit is used to divide the input matrix into multiple sub-matrices, and the number and position index of each sub-matrix are determined based on the grouping and interleaving method; The processing unit is configured to perform corresponding submatrix operations based on at least one computation thread block in the thread block network, and to transmit the operation results of the corresponding submatrix based on at least one communication thread block corresponding to the at least one computation thread block; The cache unit is used to store the operation results of each submatrix in the cache according to the number order corresponding to each submatrix, and to determine the output matrix based on the number and position index corresponding to each submatrix and the operation results of the submatrix stored sequentially in the cache. The output matrix is ​​used to determine the processing result of the data to be processed corresponding to the input matrix. During the process of transmitting the operation result of the corresponding sub-matrix in the at least one communication thread block, other computing thread blocks in the thread block network perform the corresponding sub-matrix operation until the output matrix corresponding to the input matrix is ​​obtained.

[0009] According to a third aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods of this disclosure.

[0010] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this disclosure.

[0011] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the methods described in this disclosure.

[0012] The data processing method disclosed herein divides the input matrix into multiple sub-matrices, determines the number and position index of each sub-matrix based on a grouping and interleaving method, performs corresponding sub-matrix operations based on at least one computation thread block in the thread block network, and transmits the operation results of the corresponding sub-matrix based on at least one communication thread block corresponding to the at least one computation thread block, stores the operation results of each sub-matrix in a cache based on the number order corresponding to each sub-matrix, and determines the output matrix based on the number and position index corresponding to each sub-matrix and the operation results of the sub-matrix stored sequentially in the cache. In this way, by rearranging the input matrix through the grouping and interleaving method, the addresses of the sub-matrices calculated by each computation thread block are continuous, and by modifying the thread block network to set the communication thread blocks between the computation thread blocks, the computation thread blocks can perform sub-matrix operations synchronously during the communication process of the communication thread blocks, thereby improving the utilization of GPU resources.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0015] Figure 1 A schematic diagram of a first optional flow of the data processing method provided in this embodiment of the present disclosure is shown; Figure 2 A schematic diagram of a second optional flow of the data processing method provided in an embodiment of this disclosure is shown; Figure 3 A schematic diagram of a first optional submatrix numbering method provided in this disclosure embodiment is shown; Figure 4 A second optional schematic diagram of the submatrix numbering method provided in this disclosure embodiment is shown; Figure 5 A schematic diagram of a third optional flow of the data processing method provided in this embodiment of the present disclosure is shown; Figure 6 A schematic diagram of an optional structure of the data processing apparatus provided in an embodiment of this disclosure is shown; Figure 7 A schematic diagram of the composition structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0016] To make the objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0017] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0018] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0019] Unless otherwise defined, all technical and scientific terms used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used in this disclosure is for the purpose of describing embodiments of this disclosure only and is not intended to be limiting of this disclosure.

[0020] It should be understood that in the various embodiments of this disclosure, the sequence number of each implementation process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0021] Before providing a further detailed description of the embodiments of this disclosure, the nouns and terms involved in the embodiments of this disclosure will be explained, and the nouns and terms involved in the embodiments of this disclosure shall be interpreted as follows.

[0022] 1) Large-scale model inference: This refers to the process of using a pre-trained, large-scale deep learning model (such as a language model, visual model, or multimodal model with billions or more parameters) to perform forward computation on input data and generate prediction results or responses. This process typically does not involve model parameter updates, but only computationally intensive tensor operations and memory access operations. Large-scale model inference is often constrained by GPU memory bandwidth, computing power utilization, and data transfer efficiency. Optimizing the inference process is key to improving inference throughput and reducing latency.

[0023] 2) Graphics Processing Unit (GPU): A GPU is a specialized processor designed for massively parallel computing. Originally used for graphics rendering, it is now widely used in fields such as deep learning, scientific computing, and high-performance computing. GPUs typically consist of a large number of parallel computing cores (such as the Streaming Multiprocessor (SM) in the NVIDIA architecture). When executing a task, the GPU divides the overall computing task into a large number of logically identical subtasks and schedules these subtasks to different SMs for simultaneous execution.

[0024] Operators (Process Blocks): Operators are basic computational or communication units executed on the GPU, typically implemented as CUDA kernels, used to perform specific functional operations. During large model inference, the model's forward computation is decomposed into a series of combinations and scheduling of operators. Based on their function, operators can be divided into two categories: One type is computation operators (computation thread blocks), which are mainly responsible for numerical calculations within a single node, such as matrix multiplication, convolution, normalization, and activation functions. These operators typically have high computational density and can fully utilize the parallel computing resources and computing power of the GPU.

[0025] Another type is communication operators (communication thread blocks), used for sending, receiving, and aggregating tensors across multiple GPUs, such as broadcasting and all-reduce operations. Their underlying implementation typically relies on high-speed interconnect technologies such as Peripheral Component Interconnect Express (PCIe), NVIDIA High-Speed ​​Interconnect Bus (NVLink), and InfiniBand (IB). Generally, communication operators occupy only a small number of GPU computing units, and their performance bottleneck is mainly limited by interconnect bandwidth and network latency, rather than the GPU's computing power.

[0026] Block matrix multiplication: A matrix computation method that divides a large matrix into several smaller sub-matrices (blocks) and performs multiplication and addition operations at the block level. When performing matrix multiplication on a GPU, the matrix is ​​usually divided into fixed-size blocks along the row and column directions, and each thread block or computing unit (SM) is responsible for computing one or more sub-blocks in the output matrix.

[0027] Compute Unified Device Architecture (CUDA) Streams: In the CUDA programming model, these are execution queues used to organize and schedule a series of asynchronous operations (such as kernel function startup and memory copying) on ​​the GPU. Operations in each CUDA stream are executed in the order they are submitted, and operations in different streams can be executed concurrently, thereby achieving optimization effects such as overlapping computation and communication, and parallel execution of multiple kernel functions.

[0028] In existing technologies, some research has attempted to execute computation and communication in parallel on different streams using a dual-CUDA stream approach. However, dual-stream solutions require developers to manually manage inter-stream synchronization and are limited by the GPU hardware scheduler's scheduling strategy for multi-stream tasks. As a result, the actual overlap effect is often not ideal, and the parallel potential of the hardware cannot be fully utilized.

[0029] Therefore, how to achieve fine-grained overlap between matrix multiplication calculation and set communication within a single process block, and maximize the parallel utilization of GPU computing units and communication links, is a key technical problem that urgently needs to be solved to improve the efficiency of multi-GPU parallel inference.

[0030] Based on this, embodiments of this disclosure provide a data processing method that can eliminate the serial waiting overhead between matrix multiplication (GEMM) and collective communication (AllReduce) in multi-GPU tensor parallel inference scenarios, and achieve fine-grained overlap between computation and communication, thereby improving GPU resource utilization and inference throughput performance.

[0031] Figure 1 A schematic diagram of a first alternative flow of the data processing method provided in this disclosure embodiment is shown, and the steps will be described accordingly.

[0032] Step S101: Divide the input matrix into multiple submatrices and determine the number and position index of each submatrix based on the grouping and interleaving method.

[0033] In some embodiments, the carrier implementing the data processing method (hereinafter referred to as the carrier) partitions the input matrix into at least two sub-matrices of the same dimension. Each sub-matrix is ​​computed by a computation thread block.

[0034] In some embodiments, the carrier numbers all submatrices.

[0035] In specific implementation, the carrier can number the submatrices based on row and column order. For example, starting from the submatrix corresponding to the first row and first column, other submatrices in the same row are numbered sequentially along the row direction. After all submatrices in that row are numbered, the submatrices in the second row are numbered, and so on, until all submatrices are numbered. The position index of each submatrix is ​​determined based on its row and column positions in the input matrix, and the correspondence between the number and position index of each submatrix is ​​recorded.

[0036] Alternatively, in a specific implementation, the carrier can group all submatrices in the input matrix based on grouping parameters, making submatrices in at least two adjacent rows a matrix group, with the number of submatrices in each matrix group in the row direction being the same as the grouping parameters; then, the submatrices in each matrix group are numbered in column-first, row-second order, that is, the submatrices in the matrix group that are in the same column in the column direction are numbered, and then at least two submatrices in the next column of that column are numbered sequentially. After all the submatrices in the matrix group are numbered, the submatrices in the next matrix group are numbered in the same way until the numbering is completed.

[0037] Specific examples of submatrix numbering will be explained later.

[0038] The carrier can be a computer program, electronic circuit, database, mobile application, electronic device, cloud computing platform, distributed system, artificial intelligence framework, mathematical model, automation tool and microcontroller, etc., which are software or hardware capable of implementing algorithms and methods.

[0039] Step S102: Perform corresponding submatrix operations based on at least one computation thread block in the thread block network, and transmit the operation results of the corresponding submatrix based on at least one communication thread block corresponding to the at least one computation thread block.

[0040] In some embodiments, the thread block network includes computation thread blocks and communication thread blocks, with the communication thread blocks positioned between the computation thread blocks. The computation thread blocks are used to implement submatrix operations, and the communication thread blocks are used to transmit the results of the submatrix operations.

[0041] In specific implementation, the thread block network includes multiple thread groups, and each thread group includes multiple consecutive computation thread blocks and at least one communication thread block; the communication thread block is set after the multiple computation thread blocks; the last communication thread block of the previous thread group is adjacent to the first computation thread block of the adjacent next thread group; in the thread block network, the computation thread blocks correspond one-to-one with the submatrices of the input matrix.

[0042] In some embodiments, in each thread group, all communication thread blocks are used to transmit the calculation results of the corresponding submatrix by all calculation thread blocks in the thread group; if any thread group includes at least two communication thread blocks, then the at least two communication thread blocks are consecutive.

[0043] In some embodiments, at least one computation thread block in the first thread group performs a corresponding submatrix operation; in response to the execution of the corresponding submatrix operation by at least one computation thread block in the first thread group, at least one communication thread block in the first thread group transmits the operation result of the corresponding submatrix; while the at least one communication thread block in the first thread group is transmitting the operation result of the corresponding submatrix, at least one computation thread block in the second thread group performs the corresponding submatrix operation, and after the operation is completed, at least one communication thread block in the second thread group transmits the operation result of the corresponding submatrix, until all thread groups have completed the operation of the input matrix and the transmission of the operation result. In this way, communication and computation can be performed simultaneously, improving GPU resource utilization.

[0044] Step S103: Store the operation results of each submatrix in the cache according to the numbering order corresponding to each submatrix. Determine the output matrix based on the number and position index corresponding to each submatrix and the operation results of the submatrix stored sequentially in the cache.

[0045] In some embodiments, when the communication thread block performs communication operations, it requires that the data to be communicated be stored contiguously in memory so that the communication library (such as NCCL) can perform data transmission with maximum bandwidth. If communication is performed on non-contiguous memory, multiple small-granularity calls are required, which will significantly increase the communication startup overhead and reduce bandwidth utilization. Therefore, after the computation thread block performs operations on the submatrix and obtains the results, it stores the results sequentially in a cache based on the submatrix number, waiting for the communication thread block to transmit the sequentially stored results in the cache to other GPUs. After receiving the sequential results, the other GPUs rearrange the results based on the correspondence between the number and the position index to obtain the output matrix. The position index of the submatrix with the same number and its corresponding operation result is the same in both the input and output matrices.

[0046] The output matrix is ​​used to determine the processing result of the data to be processed corresponding to the input matrix.

[0047] Thus, by using the data processing method provided in this embodiment, the input matrix is ​​rearranged using a grouping and interleaving method, so that the addresses of the submatrices calculated by each computation thread block are continuous. By modifying the thread block network to set the communication thread block between the computation thread blocks, the computation thread blocks can perform submatrix operations synchronously during the communication process of the communication thread blocks, thereby improving the utilization of GPU resources.

[0048] Figure 2A second alternative flowchart of the data processing method provided in this disclosure embodiment is shown, and will be described according to each step.

[0049] Step S201: Determine the thread block network.

[0050] In some embodiments, the thread block network includes communication thread blocks and computation thread blocks. Optionally, the thread block network can be divided into multiple thread groups, each of which includes computation thread blocks and communication thread blocks. In each thread group, the computation thread blocks run first, and the communication thread blocks run after the computation thread blocks have run.

[0051] In some embodiments, computation thread blocks are responsible for performing block-based computations of matrix multiplication. Each computation thread block is responsible for the computation task of a submatrix (Tile), and there are a total of [number missing] thread blocks in the thread block network. (Number of submatrices) computation thread blocks. Local index of the computation thread block. The index of the target submatrix it is responsible for One-to-one correspondence and equality. Responsible for executing cross-GPU AllReduce collection communication. Each communication thread block is responsible for batch processing. The communication task of submatrices, with a total of [number] submatrices in the grid. A communication thread block. For index 1 The communication thread block, which is responsible for the index of the submatrix used for communication. The range is: Finally, the communication thread block queue... Inserted into the computation thread block queue according to the preset scheduling strategy. Together, they form a thread block network. (The specific insertion scheduling strategy will be described later.) The total number of thread blocks included in the thread block network is: .

[0052] In some alternative embodiments, each thread group includes a plurality of computation thread blocks and a plurality of communication thread blocks arranged consecutively; the plurality of computation thread blocks are arranged before the plurality of communication thread blocks.

[0053] In some embodiments, the number of communication thread blocks in each thread group is determined based on the communication task corresponding to each communication thread block and the calculation results of the corresponding submatrices of all computation thread blocks in the thread group; specifically, it can be determined based on the communication tasks that each communication thread block can complete and the communication tasks of the corresponding thread group; the communication tasks of the thread group are related to the number of computation thread blocks and the input matrix.

[0054] In some embodiments, in a thread block network, the interval between communication thread blocks of two adjacent thread groups is determined based on the number of computation thread blocks corresponding to each communication thread block and the number of communication thread blocks in each thread group.

[0055] In other embodiments, in a thread block network, the interval between the communication thread blocks of two adjacent thread groups is determined based on the number of computing thread blocks that the graphics processor can run concurrently.

[0056] Step S202: Divide the input matrix into multiple submatrices and determine the number of each submatrix based on the grouping and interleaving method.

[0057] In some embodiments, the carrier divides the input matrix into multiple submatrices, each with the same dimension; it determines at least one submatrix corresponding to each grouping parameter row in the input matrix as a matrix group; each matrix group includes the same number of submatrices, and each matrix group has the same dimension; each submatrix in each matrix group is numbered in column-major order so that adjacent numbered submatrices are physically adjacent in the column direction. For example, if the grouping parameter is 2, then every two rows of submatrices in the input matrix form a matrix group.

[0058] In practical implementation, the dimension of the input matrix is... For example, the input matrix is ​​divided into multiple submatrices according to rows and columns, and the height of each submatrix is... Width is Along the direction of travel ( The number of submatrices (direction) is That is, each row includes A matrix group; along The number of submatrices in the direction (column direction) is The total number of submatrices is .

[0059] Figure 3 A schematic diagram of a first alternative submatrix numbering method provided in an embodiment of this disclosure is shown.

[0060] In some embodiments, the carrier numbers each submatrix, such as... Figure 3 The input matrix shown is divided into 24 submatrices. The submatrices are numbered in row-major order, that is, by row, the submatrices in the first row are numbered sequentially, such as 0, 1, 2, 3, 4, 5. After the first row of submatrices is numbered, the submatrices in the second row are numbered sequentially, such as 6, 7, 8, 9, 10, 11, until all submatrices in the input matrix are numbered.

[0061] Figure 4 A second alternative schematic diagram of the submatrix numbering method provided in an embodiment of this disclosure is shown.

[0062] In some embodiments, the carrier numbers each submatrix, such as... Figure 4 The diagram shows the grouping and interleaving method number. The grouping parameter is 2, meaning that every two rows of submatrices form a matrix group. For example, the first and second rows of submatrices form matrix group 1, and the third and fourth rows of submatrices form matrix group 2.

[0063] The submatrices in the matrix group are numbered according to the column priority method. That is, the submatrices in the first column are numbered sequentially, such as 0, 1. After the submatrices in the first column are numbered, the submatrices in the second column are numbered sequentially, such as 2, 3, and so on, until all submatrices in the input matrix are numbered.

[0064] Specifically, adopt Figure 3 The numbering method shown indicates that in conventional matrix multiplication (GEMM) operations, the output matrix typically employs a row-major memory layout. In this layout, for a single submatrix, the data in each row is discretely distributed in the physical address space of video memory, and the address span between two adjacent rows is equal to the width (stride) of the input matrix. Furthermore, the storage locations of submatrices with different numbers in global video memory are not in the execution order. The memory is arranged in a continuous sequence. This kind of discreteness at the memory level severely hinders the efficient reading of large blocks of contiguous memory by the collection communication library, and greatly reduces the utilization of communication bandwidth.

[0065] use Figure 4 The numbering method shown ensures that, during each calculation (e.g., the calculation of submatrices 0, 1, 2, 3), their positions are adjacent in rows and columns, and continuous in the address space. Furthermore, after the calculation, the results are written to the cache according to the submatric numbering order, eliminating data gaps between rows within the submatrices and ensuring that a single communication targets a block of data with absolutely contiguous physical addresses. When other GPUs receive the computational data, they pre-assign numbers to each submatrix in global memory, based on the aforementioned grouping and interleaving numbering rules. Sequential allocation of contiguous memory space. The write-back base address of each submatrix is ​​uniquely mapped by its execution order, so that submatrix data with consecutive indices are contiguous in the rearranged cache, presenting a tightly packed contiguous memory form.

[0066] In some embodiments, the carrier determines the position index of each submatrix, specifically including: determining the number of submatrices in each matrix group based on grouping parameters and the number of submatrices corresponding to each row; determining the matrix group to which each submatrix belongs based on the number of each submatrix and the number of submatrices in each matrix group; determining the row identifier of each submatrix based on the matrix group to which each submatrix belongs and the grouping parameters; determining the column identifier of each submatrix based on the number of each submatrix, the grouping parameters, and the total number of submatrices included in each matrix group; and determining the position index of each submatrix based on the row identifier and column identifier of each submatrix.

[0067] In specific implementation, the carrier can be based on the matrix group to which each submatrix belongs. and grouping parameters Determine the initial row identifier for each submatrix. Based on the number of each submatrix and the grouping parameters, determine the row offset corresponding to each submatrix. The row identifier for each submatrix is ​​determined based on the initial row identifier and row offset for each submatrix. .

[0068] In practice, the carrier can determine the local index of each submatrix within the matrix group based on the number of each submatrix and the total number of submatrixes included in each matrix group. Based on the local index of each matrix within the matrix group and the grouping parameters, determine the column identifier corresponding to each submatrix. .

[0069] Step S203: Perform corresponding submatrix operations based on at least one computation thread block in the thread block network.

[0070] In some embodiments, submatrix operations and transmissions are performed on a per-thread-group basis within a thread block network; at least one computation thread block within a thread group performs operations on the corresponding submatrix and obtains the operation results of the submatrix.

[0071] In specific implementation, the carrier determines the operation result of the corresponding submatrix; based on the number corresponding to the submatrix, the operation results are sequentially stored in the cache; wherein, the numbers corresponding to the operation results in the cache are arranged in ascending order; in response to the operation result being stored in the cache, the value of a flag bit is incremented; wherein, the maximum value of the flag bit is the same as the number of computation thread blocks in the thread group. Increasing the value of the flag bit includes incrementing the flag bit by 1 for each operation result stored in the cache.

[0072] Step S204: Determine whether all computation thread blocks in the thread group have obtained the operation results of the submatrix.

[0073] In some embodiments, the communication thread block in the thread group determines whether the value of its corresponding flag bit is the same as the number of its corresponding computation thread blocks in the thread group; if the value of the flag bit is the same as the number of computation thread blocks corresponding to any communication thread block in the thread group, then it is determined that all computation thread blocks corresponding to any communication thread block have obtained the submatrix operation result; or, if the value of the flag bit is less than the number of computation thread blocks corresponding to any communication thread block in the thread group, then it is determined that at least one computation thread block among the computation thread blocks corresponding to any communication thread block has not obtained the submatrix operation result.

[0074] In some embodiments, if it is determined that all computation thread blocks corresponding to any communication thread block have obtained the operation result of the submatrix, then step S205 is executed; or, if it is determined that at least one computation thread block among the computation thread blocks corresponding to any communication thread block has not obtained the operation result of the submatrix, then the computation thread block continues to operate, and the communication thread block periodically determines the value of the flag bit.

[0075] Step S205: The communication thread block transmits the calculation results.

[0076] In some embodiments, the communication thread block is scheduled to a streaming multiprocessor; the underlying communication primitive is invoked to transmit the computation result corresponding to the communication thread block in the cache to other graphics processors; in response to the completion of the computation result transmission, the value of the flag bit corresponding to the communication thread block is set to a preset threshold; and the hardware resources of the streaming multiprocessor corresponding to the communication thread block are released. The preset threshold can be 0.

[0077] Step S206: Determine the output matrix based on the number and position index of each submatrix and the operation results of the submatrixes stored sequentially in the cache.

[0078] In some embodiments, among the computation results received by the other graphics processor, each computation result is sorted according to the number of the corresponding sub-matrix, and the position index corresponding to each computation result is determined based on the number of the sub-matrix corresponding to each computation result; the position index includes the row identifier and column identifier of the computation result in the output matrix; each computation result is sorted according to the position index corresponding to each computation result to obtain the output matrix.

[0079] In some embodiments, the carrier can determine the matrix group to which each operation result belongs based on the number of each operation result and the number of sub-matrices in each matrix group; determine the intra-group offset value of each operation result based on the number of each operation result and the number of sub-matrices in each matrix group; determine the column identifier of each operation result in its matrix group based on the intra-group offset value and grouping parameters; determine the row identifier of each operation result in its matrix group based on the intra-group offset value and grouping parameters; and sort each operation result according to its row identifier and column identifier in its matrix group to obtain an output matrix.

[0080] Specifically, the column identifier of each operation result in its matrix group is determined by rounding down the ratio of the intra-group offset value of each operation result to the grouping parameter; the row identifier of each operation result in its matrix group is determined by taking the modulo of the intra-group offset value of each operation result with respect to the grouping parameter.

[0081] Thus, by using the data processing method provided in this embodiment, the input matrix is ​​rearranged using a grouping and interleaving method, so that the addresses of the submatrices calculated by each computation thread block are continuous. By modifying the thread block network to set the communication thread block between the computation thread blocks, the computation thread blocks can perform submatrix operations synchronously during the communication process of the communication thread blocks, thereby improving the utilization of GPU resources.

[0082] Figure 5 A third alternative flowchart of the data processing method provided in this disclosure embodiment is shown, and will be described according to each step.

[0083] This disclosure proposes a GPU operator that integrates matrix multiplication and all-reduce communication (GEMM-AllReduce operator) to replace the existing two-step operation of "GEMM computation + AllReduce communication" in the inference process. The core idea of ​​this operator is to insert several communication thread blocks at specific positions between computation thread blocks in the thread block grid of matrix multiplication (GEMM). Utilizing the natural scheduling mechanism of the GPU hardware scheduler, dynamic overlapping execution of matrix multiplication computation and AllReduce communication is achieved. Furthermore, this disclosure introduces a fine-grained lock-free synchronization mechanism based on flag bit sets. When a computation thread block completes local computation, it notifies the communication thread block that data is ready by updating the flag bits, thereby triggering cross-GPU all-reduce communication. Thus, efficient parallelism of computation and communication can be achieved within a single operator.

[0084] In some embodiments, the original two-step operation "GEMM2+AllReduce" in the large language model inference engine is replaced with the AllReduce-GEMM single-step operator described in this disclosure. Specifically: (1) In the Attention module, replace [Attention] to [Output GEMM] to [AllReduce] with [Attention] to [AllReduce-GEMM]; (2) In the FFN module, replace [GEMM1] to [GEMM2] to [AllReduce] with [GEMM1] to [AllReduce-GEMM]. Through the above replacements, overlapping execution of computation and communication during inference is achieved without changing the semantics of model computation.

[0085] The specific steps of this embodiment are as follows: Step S501: Divide the input matrix into at least one submatrix.

[0086] In some embodiments, the dimension is Target input matrix Divide into several sub-matrices (tiles), each sub-matrix having a height of . Width is .along The number of submatrices in the direction is (Each column has) (submatrices), along The number of submatrices in the direction is (Each line has) (number of submatrices), the total number of submatrices is Each submatrix is ​​globally numbered. For example, the traditional row-matrix numbering method is as follows: Figure 3 As shown, through a two-dimensional index Its one-dimensional index can be obtained directly. .

[0087] The embodiments disclosed herein employ, as follows: Figure 4 The numbering method and grouping parameters shown That is, each A behavior is a matrix group, and each matrix group contains Submatrices, . Continuous Each submatrix is ​​divided into a tile group. The submatrices are divided into groups, and each group is numbered in column-major order, ensuring that adjacent submatrices remain physically adjacent along the column direction. (Two-dimensional position index of the submatrices.) Can be indexed by one dimension Grouping parameters Total number of lines and total number of columns The calculation is as follows: In some embodiments, the total number of submatrices contained within each matrix group is calculated. ; Determine the index of the matrix group to which the current submatrix belongs. And based on this, the starting row position of the matrix group in the row direction is obtained. ; Calculate the local index of the current submatrix within the group. Based on the column-priority numbering principle within the group, by adjusting the group size... Perform modulo and integer division operations to calculate the travel index. With column index .

[0088] Step S502: Construct the thread block network.

[0089] In some embodiments, the thread block grid of the traditional GEMM operator is extended from a single computation thread block to a hybrid grid containing the following two types of thread blocks: Compute Blocks: Responsible for performing submatrix calculations within matrix multiplication. Each compute block handles the calculation of one submatrix. There are a total of [number missing] compute blocks in the mesh. A computation thread block. In this invention, the local index of the computation thread block. Its responsible target submatrix index They correspond one-to-one and are equal.

[0090] Communication Block: Responsible for performing cross-GPU AllReduce collection communication. Each communication block is responsible for batch processing. Communication tasks for each Tile, with a total of [number] threads in the block network. A communication thread block. For index 1 The communication thread block, which is responsible for communication. The range is: .

[0091] Queue of communication thread blocks Inserted into the computation thread block queue according to the preset scheduling strategy. Together, they form a thread block network. The total length of the thread block queue is: .

[0092] In some embodiments, due to the fixed concurrency capacity of GPU streaming multiprocessors, the execution and completion of computation thread blocks on the hardware exhibit a typical "ladder-like" distribution. To enable communication thread blocks to start immediately following the completion of computation tasks and maximize the overlap between computation and AllReduce communication, embodiments of this disclosure propose a wave-interleaved insertion strategy that matches the hardware execution characteristics.

[0093] In practice, communication thread blocks are no longer inserted discretely and randomly, but rather according to... Each thread block is arranged in a batch, forming a single wave. Thread blocks within the same wave are connected in the thread block network. The threads are assigned consecutive indices to ensure they are continuously and centrally dispatched to idle SMs by the GPU scheduler, thereby fully aggregating the underlying network bandwidth and maximizing the peak throughput of aggregated communication. Within this group, the same wave of communication thread blocks and the multiple preceding computation thread blocks constitute a thread group.

[0094] In some embodiments, an insertion interval parameter is introduced. When constructing a thread block network, each permutation Each computation thread block, followed immediately by a communication thread block, is specifically responsible for handling this. The computation results of each computation thread block. This interleaved arrangement ensures that when a communication thread block is dispatched, the data it depends on is often ready, effectively eliminating the overhead of polling and waiting.

[0095] In some embodiments, for the insertion interval parameter This can be set to an integer multiple of the maximum number of computing thread blocks that the GPU can concurrently reside in. This strictly aligns with the hardware cycle where computing tasks are completed in a stepped, batch-like manner, achieving a seamless transition between releasing computing resources and starting communication tasks. Optionally, it can be based on the number of submatrices handled by a single communication thread block. and the number of communication thread blocks in each wave. To impose constraints, that is, to logically satisfy the supply and demand matching relationship. This is to ensure a balance between the rates of data production and communication consumption.

[0096] Step S503: The thread block in the thread block network performs submatrix operations.

[0097] In some embodiments, thread block function routing and identification: the entire mesh is divided into two main functional entities: computation and communication. When any thread block is hardware scheduled to execute on a streaming multiprocessor (SM), it first determines the execution based on the obtained thread block network index. In addition to the preset queue mapping rules, it dynamically determines whether it is a computation thread block or a communication thread block, and further derives its own local index. or .

[0098] If the thread block is a computation thread block, the submatrix operation process includes: performing submatrix operations, continuously writing back data, and notifying that the data is ready.

[0099] In practice, submatrix operations include using the standard GEMM calculation method to complete the indexing of the target matrix. Multiply-accumulate calculation of local fragments.

[0100] In practice, continuous data write-back includes, after the calculation is completed, expanding the calculation results in on-chip memory (such as registers or shared memory) and writing them into the continuously rearranged cache. Zhongyou Within the corresponding mapped contiguous physical address range.

[0101] Specifically, in conventional matrix multiplication (GEMM) operations, the input matrix typically uses a row-major memory layout. Figure 3 As shown in the diagram. In this layout, for a single submatrix (Tile), the data in each row is discretely distributed in the physical address space of video memory, and the address span between two adjacent rows is equal to the width of the input matrix. Furthermore, the storage locations of submatrices with different numbers in global video memory are not in execution order (as shown in the diagram). The memory is arranged in a continuous sequence. This kind of discreteness at the memory level severely hinders the efficient reading of large blocks of contiguous memory by the collection communication library, and greatly reduces the utilization of communication bandwidth.

[0102] To address this issue, embodiments of this disclosure design a contiguous reordering buffer specifically for "computation-communication" pipelines, which specifically includes: After completing its assigned tile computation task, the computation thread block no longer writes back data following the global stide layout of the original input matrix. Instead, it expands its internal results into a one-dimensional contiguous block of memory. This eliminates data gaps between rows within a tile, ensuring that a single AllReduce operation targets a single block of physically contiguous data. Combined with the aforementioned grouping and interleaving numbering rule, each tile is pre-assigned according to... Sequential allocation of contiguous memory space. The write-back base address of each tile is uniquely mapped by its execution order, so that the data of consecutively indexed tiles are arranged end-to-end in the rearranged cache, presenting a tightly packed contiguous memory form.

[0103] In practice, the notification that data is ready includes, after the memory write-back instruction is submitted, the computation thread block performs an atomic increment operation (atomicAdd) on the target flag `finish_flag[i]`. This represents the index of the communication thread block responsible for this tile. Relying on the visibility guarantees provided by the GPU memory model, the cache can be determined when the communication thread block determines that finish_flag[i] == T. The corresponding batch submatrix data has been completely written to disk and is globally visible.

[0104] Step S504: The communication thread block in the thread block network performs sub-matrix transport.

[0105] In some embodiments, the communication thread block execution logic, for a thread block routed as a communication task, is responsible for processing the first Tile index. , total continuous processing Communication per tile. This batch of data is continuously reordered in the cache. The physical offset range is: .

[0106] In some embodiments, the communication thread block performs polling wait, executes collection communication, and resource release steps.

[0107] In practice, the polling wait includes communication thread blocks scheduled to the SM, which continuously read the value of their corresponding finish_flag[CommBlockIdx] through spin-wait until the accumulated value reaches a threshold. .

[0108] In practice, executing collection communication includes indicating all communication threads under the jurisdiction of the current communication thread block when finish_flag[CommBlockIdx] == T is true. All submatrix calculations have been completed and stored in the cache. The communication thread block then calls the underlying communication primitives to access the cache. The continuous data in the process performs an AllReduce operation across GPUs.

[0109] In practice, resource release includes resetting finish_flag[CommBlockIdx] to 0 after AllReduce communication is completely completed, and then actively exiting execution, releasing the SM hardware resources it occupies. The released SM will be quickly allocated by the GPU hardware scheduler to other computing or communication thread blocks that are still waiting in the global queue, thereby achieving dynamic reuse of computing and communication resources.

[0110] Step S505: Determine the output matrix based on the computation data.

[0111] In some embodiments, after the AllReduce-GEMM operator completes, the cache R stores the reduction results arranged in submatrix number order, rather than the standard row-major order matrix layout. Subsequent operators (such as RMSnorm layer normalization, residual joins, and other element-wise operations) need to restore the data to the standard layout (i.e., the layout corresponding to the input matrix). This embodiment integrates the layout restoration operation into the immediately following downstream element-wise operator, rather than introducing a separate data copy operator.

[0112] Specifically, when the downstream element-wise operator reads the computational data, it no longer reads it in the standard matrix address order, but according to the mapping function F. Figure 4 The encoding read is shown. Experimental results show that the additional latency overhead of incorporating layout restoration into element-wise operators such as RMSNorm is approximately 8% to 10% of the operator's own latency, having a negligible impact on end-to-end performance.

[0113] In this way, the overhead of data rearrangement is distributed across the computation write-back stage and the downstream operator read stage. The layout restoration is completed through pure computation of the mapping function F. This eliminates the need for additional video memory to store the mapping table and does not introduce an independent data copy operator, allowing the communication block to focus on efficiently executing communication tasks and maximizing the benefits of computation-communication overlap.

[0114] Thus, by inserting communication blocks into the GEMM thread block grid, dynamic parallelism between computation and communication is achieved using the natural scheduling mechanism of the GPU hardware scheduler, resulting in a concise implementation with low intrusion into the GEMM operator. The AllReduce-GEMM operator can directly replace the GEMM+AllReduce operation in existing inference engines without changing the model's computational semantics, offering good compatibility and simple integration.

[0115] Figure 6 A schematic diagram of an optional structure of the data processing apparatus provided in an embodiment of this disclosure is shown, and the details will be described in terms of each part.

[0116] In some embodiments, the data processing apparatus includes a partitioning unit, a processing unit, and a caching unit.

[0117] The partitioning unit is used to divide the input matrix into multiple sub-matrices, and the number and position index of each sub-matrix are determined based on the grouping and interleaving method; The processing unit is configured to perform corresponding submatrix operations based on at least one computation thread block in the thread block network, and to transmit the operation results of the corresponding submatrix based on at least one communication thread block corresponding to the at least one computation thread block; The cache unit is used to store the operation results of each submatrix in the cache according to the number order corresponding to each submatrix, and to determine the output matrix based on the number and position index corresponding to each submatrix and the operation results of the submatrix stored sequentially in the cache. The output matrix is ​​used to determine the processing result of the data to be processed corresponding to the input matrix. During the process of transmitting the operation result of the corresponding sub-matrix in the at least one communication thread block, other computing thread blocks in the thread block network perform the corresponding sub-matrix operation until the output matrix corresponding to the input matrix is ​​obtained.

[0118] In some embodiments, the thread block network includes multiple thread groups, each thread group including multiple consecutive computation thread blocks and at least one communication thread block; the communication thread block is positioned after the multiple computation thread blocks. The last communication thread block of the previous thread group is adjacent to the first computation thread block of the adjacent next thread group; In a thread block network, each computation thread block corresponds one-to-one with a submatrix of the input matrix.

[0119] In some embodiments, in each thread group, all communication thread blocks are used to transmit the calculation results of the corresponding submatrix by all calculation thread blocks in the thread group; If any thread group includes at least two communication thread blocks, then the at least two communication thread blocks are consecutive.

[0120] In some embodiments, the number of communication thread blocks in each thread group is determined based on the communication task corresponding to each communication thread block and the calculation results of the corresponding sub-matrices of all computation thread blocks in the thread group. In a thread block network, the interval between the communication thread blocks of two adjacent thread groups is determined based on the number of computation thread blocks corresponding to each communication thread block and the number of communication thread blocks in each thread group.

[0121] In some embodiments, in a thread block network, the interval between the communication thread blocks of two adjacent thread groups is determined based on the number of computing thread blocks that the graphics processor can run concurrently.

[0122] The partitioning unit is specifically used to divide the input matrix into multiple sub-matrices, each sub-matrix having the same dimension. Each group of parameters in the input matrix is ​​defined as a matrix group, with each matrix group containing the same number of submatrices and having the same dimension. Each submatrix in each matrix group is numbered in column priority order so that adjacent numbered submatrixes are physically adjacent in the column direction.

[0123] The partitioning unit is specifically used to determine the position index corresponding to each submatrix based on the submatrix's number, grouping parameters, total number of rows in the input matrix, total number of columns in the input matrix, total number of rows in the submatrix, and total number of columns in the submatrix; specifically including: Based on the total number of rows in the input matrix and the total number of rows in the submatrixes, determine the number of submatrixes corresponding to each column of the input matrix; Based on the total number of columns in the input matrix and the total number of columns in the submatrices, determine the number of submatrices corresponding to each row of the input matrix; The number of submatrices in each matrix group is determined based on the grouping parameters and the number of submatrices corresponding to each row; Based on the number of each submatrix and the number of submatrixes in each matrix group, determine the matrix group to which each submatrix belongs; Based on the matrix group to which each submatrix belongs and the grouping parameters, determine the row identifier corresponding to each submatrix; Based on the number of each submatrix, the grouping parameters, and the total number of submatrixes included in each matrix group, determine the column identifier corresponding to each submatrix; Based on the row and column identifiers of each submatrix, determine the position index of each submatrix.

[0124] The partitioning unit is specifically used to determine the initial row identifier corresponding to each submatrix based on the matrix group to which each submatrix belongs and the grouping parameters; Based on the number of each submatrix and the grouping parameters, determine the row offset corresponding to each submatrix; The row identifier for each submatrix is ​​determined based on the initial row identifier and row offset of each submatrix.

[0125] The partitioning unit is specifically used to determine the local index of each submatrix within the matrix group based on the number of each submatrix and the total number of submatrixes included in each matrix group. Based on the local index of each matrix within the matrix group and the grouping parameters, the column identifier corresponding to each submatrix is ​​determined.

[0126] The processing unit is specifically used to perform the following operations on each thread group: At least one computation thread block in the thread group performs operations on the corresponding submatrix and obtains the operation result of the submatrix; In response to all computation thread blocks in the thread group obtaining the operation results of the submatrix, at least one communication thread block in the thread group transmits the operation results of the submatrix obtained by all computation thread blocks to other graphics processors.

[0127] The processing unit is specifically used to perform the following operations on each computation thread block in the thread group: Determine the result of the operation on the corresponding submatrix; The operation results are stored sequentially in the cache based on the corresponding numbers of the submatrices; the numbers of the operation results in the cache are arranged in ascending order. In response to the calculation result being stored in the cache, the value of the flag is incremented; The maximum value of the flag is the same as the number of computation thread blocks in the thread group.

[0128] The processing unit is specifically used to determine whether all corresponding computation thread blocks have obtained the submatrix operation results based on each communication thread block in the thread group, specifically including: Each communication thread block determines whether the value of its corresponding flag bit is the same as the number of its corresponding computation thread blocks in the thread group; If the value of the flag bit is the same as the number of computation thread blocks corresponding to any communication thread block in the thread group, then it is determined that all computation thread blocks corresponding to any communication thread block have obtained the operation result of the submatrix. Alternatively, if the value of the flag is less than the number of computation thread blocks corresponding to any communication thread block in the thread group, then it is determined that at least one computation thread block among the computation thread blocks corresponding to any communication thread block has not obtained the operation result of the submatrix.

[0129] The processing unit is specifically used to perform the following operations on each communication thread block in the thread group: Schedule the communication thread block to the streaming multiprocessor; Call the underlying communication primitive to transmit the calculation result corresponding to the communication thread block in the cache to other graphics processors; In response to the completion of the transmission of the calculation result, the value of the flag bit corresponding to the communication thread block is set to a preset threshold. Release the hardware resources of the streaming multiprocessor corresponding to the communication thread block.

[0130] The processing unit is specifically used to perform corresponding submatrix operations on at least one computation thread block in the first thread group. In response to at least one computation thread block in the first thread group performing a corresponding submatrix operation, at least one communication thread block in the first thread group transmits the operation result of the corresponding submatrix. During the process of transmitting the operation result of the corresponding submatrix in at least one communication thread block in the first thread group, at least one calculation thread block in the second thread group performs the operation of the corresponding submatrix, and after the operation is completed, at least one communication thread block in the second thread group transmits the operation result of the corresponding submatrix, until all thread groups have completed the operation of the input matrix and the transmission of the operation result.

[0131] The cache unit is specifically used in the calculation results received by the other graphics processors, where each calculation result is sorted according to the number of the corresponding sub-matrix, and the position index corresponding to each calculation result is determined based on the number of the sub-matrix corresponding to each calculation result; the position index includes the row identifier and column identifier of the calculation result in the output matrix; Sort each operation result according to its corresponding position index to obtain the output matrix.

[0132] The cache unit is specifically used to determine the matrix group to which each operation result belongs based on the number of each operation result and the number of sub-matrices in each matrix group. Based on the number of each operation result and the number of submatrices in each matrix group, determine the group offset value of each operation result. The column identifier of each operation result in its respective matrix group is determined based on the intra-group offset value and grouping parameters of each operation result. The row identifier of each operation result in its respective matrix group is determined based on the intra-group offset value and grouping parameters of each operation result. Sort each operation result according to its row and column indices in the matrix group to obtain the output matrix.

[0133] The cache unit is specifically used to determine the column identifier of each operation result in its respective matrix group based on the floor function of the ratio of the group offset value of each operation result to the grouping parameter.

[0134] The cache unit is specifically used to determine the row identifier of each operation result in its respective matrix group based on the result of taking the modulo of the grouping parameter with the group offset value of each operation result.

[0135] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0136] Figure 7 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0137] like Figure 7As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0138] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0139] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0140] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0144] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0145] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0146] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0147] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0148] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The thread block network includes computation thread blocks and communication thread blocks. Communication thread blocks are positioned between computation thread blocks. The computation thread blocks are used to implement submatrix operations, and the communication thread blocks are used to transmit the results of these submatrix operations. The method includes: The input matrix is ​​divided into multiple submatrices, and the number and position index of each submatrix are determined based on the grouping and interleaving method. The corresponding submatrix operation is performed based on at least one computation thread block in the thread block network, and the operation result of the corresponding submatrix is ​​transmitted based on at least one communication thread block corresponding to the at least one computation thread block; The operation results of each submatrix are stored in the cache according to the numbering order corresponding to each submatrix. The output matrix is ​​determined based on the number and position index corresponding to each submatrix and the operation results of the submatrix stored sequentially in the cache. The output matrix is ​​used to determine the processing result of the data to be processed corresponding to the input matrix. During the process of transmitting the operation result of the corresponding sub-matrix in the at least one communication thread block, other computing thread blocks in the thread block network perform the corresponding sub-matrix operation until the output matrix corresponding to the input matrix is ​​obtained.

2. The method according to claim 1, characterized in that, The thread block network satisfies one of the following: The thread block network includes multiple thread groups, and each thread group includes multiple consecutive computation thread blocks and at least one communication thread block; the communication thread block is set after the multiple computation thread blocks. The last communication thread block of the previous thread group is adjacent to the first computation thread block of the adjacent next thread group; In a thread block network, there is a one-to-one correspondence between computation thread blocks and submatrices of the input matrix; In each thread group, all communication thread blocks are used to transmit the calculation results of the corresponding submatrix by all calculation thread blocks in the thread group; If any thread group includes at least two communication thread blocks, then the at least two communication thread blocks are consecutive; The number of communication thread blocks in each thread group is determined based on the communication task corresponding to each communication thread block and the calculation results of the corresponding sub-matrices of all computation thread blocks in the thread group. In a thread block network, the interval between the communication thread blocks of two adjacent thread groups is determined based on the number of computation thread blocks corresponding to each communication thread block and the number of communication thread blocks in each thread group.

3. The method according to claim 1, characterized in that, The step of dividing the input matrix into multiple sub-matrices and determining the number of each sub-matrix based on a grouping and interleaving method includes: The input matrix is ​​divided into multiple submatrices, each with the same dimension; Each group of parameters in the input matrix is ​​defined as a matrix group, with each matrix group containing the same number of submatrices and having the same dimension. Each submatrix in each matrix group is numbered in column priority order so that adjacent numbered submatrixes are physically adjacent in the column direction.

4. The method according to claim 3, characterized in that, Based on the submatrix's index, grouping parameters, total number of rows in the input matrix, total number of columns in the input matrix, total number of rows in the submatrix, and total number of columns in the submatrix, determine the position index corresponding to each submatrix; specifically including: Based on the total number of rows in the input matrix and the total number of rows in the submatrixes, determine the number of submatrixes corresponding to each column of the input matrix; Based on the total number of columns in the input matrix and the total number of columns in the submatrices, determine the number of submatrices corresponding to each row of the input matrix; The number of submatrices in each matrix group is determined based on the grouping parameters and the number of submatrices corresponding to each row; Based on the number of each submatrix and the number of submatrixes in each matrix group, determine the matrix group to which each submatrix belongs; Based on the matrix group to which each submatrix belongs and the grouping parameters, determine the row identifier corresponding to each submatrix; Based on the number of each submatrix, the grouping parameters, and the total number of submatrixes included in each matrix group, determine the column identifier corresponding to each submatrix; Based on the row and column identifiers of each submatrix, determine the position index of each submatrix.

5. The method according to claim 4, characterized in that, The process of determining the row identifier for each submatrix based on the matrix group to which each submatrix belongs and the grouping parameters includes: Based on the matrix group to which each submatrix belongs and the grouping parameters, determine the initial row identifier corresponding to each submatrix; Based on the number of each submatrix and the grouping parameters, determine the row offset corresponding to each submatrix; The row identifier for each submatrix is ​​determined based on the initial row identifier and row offset of each submatrix.

6. The method according to claim 4, characterized in that, The process of determining the column identifier for each submatrix based on its number, grouping parameters, and the total number of submatrixes within each matrix group includes: Based on the number of each submatrix and the total number of submatrixes included in each matrix group, determine the local index of each submatrix within the matrix group; Based on the local index of each matrix within the matrix group and the grouping parameters, the column identifier corresponding to each submatrix is ​​determined.

7. The method according to claim 1 or 2, characterized in that, The process of performing corresponding submatrix operations based on at least one computation thread block in the thread block network, and transmitting the operation results of the corresponding submatrix based on at least one communication thread block corresponding to the at least one computation thread block, includes each thread group performing the following operations: At least one computation thread block in the thread group performs operations on the corresponding submatrix and obtains the operation result of the submatrix; In response to all computation thread blocks in the thread group obtaining the operation results of the submatrix, at least one communication thread block in the thread group transmits the operation results of the submatrix obtained by all computation thread blocks to other graphics processors.

8. The method according to claim 7, characterized in that, At least one computation thread block in the thread group performs operations on the corresponding submatrix and obtains the operation result of the submatrix, including each computation thread block in the thread group performing the following operations: Determine the result of the operation on the corresponding submatrix; The operation results are stored sequentially in the cache based on the corresponding numbers of the submatrices; the numbers of the operation results in the cache are arranged in ascending order. In response to the calculation result being stored in the cache, the value of the flag is incremented; The maximum value of the flag is the same as the number of computation thread blocks in the thread group.

9. The method according to claim 7, characterized in that, The method further includes: Each communication thread block in the thread group determines whether all its corresponding computation thread blocks have obtained the operation results of the submatrix, specifically including: Each communication thread block determines whether the value of its corresponding flag bit is the same as the number of its corresponding computation thread blocks in the thread group; If the value of the flag bit is the same as the number of computation thread blocks corresponding to any communication thread block in the thread group, then it is determined that all computation thread blocks corresponding to any communication thread block have obtained the operation result of the submatrix. Alternatively, if the value of the flag is less than the number of computation thread blocks corresponding to any communication thread block in the thread group, then it is determined that at least one computation thread block among the computation thread blocks corresponding to any communication thread block has not obtained the operation result of the submatrix.

10. The method according to claim 7, characterized in that, At least one communication thread block in the thread group transmits the calculation results of the submatrices obtained by all computation thread blocks to other graphics processors, including performing the following operations on each communication thread block in the thread group: Schedule the communication thread block to the streaming multiprocessor; Call the underlying communication primitive to transmit the calculation result corresponding to the communication thread block in the cache to other graphics processors; In response to the completion of the transmission of the calculation result, the value of the flag bit corresponding to the communication thread block is set to a preset threshold. Release the hardware resources of the streaming multiprocessor corresponding to the communication thread block.

11. The method according to claim 1, characterized in that, The step of performing corresponding submatrix operations based on at least one computation thread block in the thread block network, and transmitting the operation results of the corresponding submatrix based on at least one communication thread block corresponding to the at least one computation thread block, includes: At least one computation thread block in the first thread group performs the corresponding submatrix operation; In response to at least one computation thread block in the first thread group performing a corresponding submatrix operation, at least one communication thread block in the first thread group transmits the operation result of the corresponding submatrix. During the process of transmitting the operation result of the corresponding submatrix in at least one communication thread block in the first thread group, at least one calculation thread block in the second thread group performs the operation of the corresponding submatrix, and after the operation is completed, at least one communication thread block in the second thread group transmits the operation result of the corresponding submatrix, until all thread groups have completed the operation of the input matrix and the transmission of the operation result.

12. The method according to claim 1, characterized in that, The step of determining the output matrix based on the number and position index corresponding to each submatrix, and the operation results of the submatrixes stored sequentially in the cache, includes: Among the calculation results received by other graphics processors, each calculation result is sorted according to the number of the corresponding sub-matrix. Based on the number of the sub-matrix corresponding to each calculation result, the position index corresponding to each calculation result is determined. The position index includes the row identifier and column identifier of the calculation result in the output matrix. Sort each operation result according to its corresponding position index to obtain the output matrix.

13. A data processing apparatus, characterized in that, The thread block network includes computation thread blocks and communication thread blocks, with communication thread blocks positioned between computation thread blocks. The computation thread blocks are used to perform submatrix operations, and the communication thread blocks are used to transmit the results of these submatrix operations. The device includes: The partitioning unit is used to divide the input matrix into multiple sub-matrices, and the number and position index of each sub-matrix are determined based on the grouping and interleaving method; The processing unit is configured to perform corresponding submatrix operations based on at least one computation thread block in the thread block network, and to transmit the operation results of the corresponding submatrix based on at least one communication thread block corresponding to the at least one computation thread block; The cache unit is used to store the operation results of each submatrix in the cache according to the number order corresponding to each submatrix, and to determine the output matrix based on the number and position index corresponding to each submatrix and the operation results of the submatrix stored sequentially in the cache. The output matrix is ​​used to determine the processing result of the data to be processed corresponding to the input matrix. During the process of transmitting the operation result of the corresponding sub-matrix in the at least one communication thread block, other computing thread blocks in the thread block network perform the corresponding sub-matrix operation until the output matrix corresponding to the input matrix is ​​obtained.

14. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

15. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-12.

Images