Data processing method, parallel processor, electronic device, storage medium

Abstract

A data processing method, a parallel processor, an electronic device and a storage medium are applied to the field of data processing. The data processing method comprises: performing a calculation processing operation by using the parallel processor, wherein the calculation processing operation comprises a plurality of calculation operations executed in sequence, and performing the calculation processing operation by using the parallel processor comprises: for each calculation operation, determining at least one competitive work block in a plurality of work blocks according to the number of the plurality of work blocks included in the input data of a target kernel function corresponding to the calculation operation and the number of a plurality of calculation units; and processing the calculation task of the plurality of work blocks in a pipeline manner by using the plurality of calculation units to complete the calculation operation; wherein for the at least one competitive work block, the allocation relationship between the calculation unit and the calculation task of the at least one competitive work block is dynamically determined by using a competition mechanism. The method can eliminate the accumulated idle time of the long tail effect and improve the overall performance.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of data processing, and in particular to a data processing method, a parallel processor, an electronic device, and a non-transitory computer-readable storage medium. BACKGROUND

[0002] In the field of high-performance programming of parallel processors such as graphic processing units (GPUs), data parallelism is a core programming paradigm. Developers usually divide the data of a large-scale computing task (such as matrix multiplication, convolution, etc.) into multiple independent data blocks of the same shape, called work tiles. Subsequently, the parallel processor starts a large number of computing units to process these work tiles in parallel.

[0003] In distributed computing scenarios such as pre-training and post-training of large models, there is often a case of overlapping execution of communication and computation, that is, communication processing operations and computing processing operations are performed in parallel within the same time period. The relationship between communication and computation is embodied as data dependency and cooperation. The communication processing operation provides the distributed data required by the computation, and the computing processing operation performs local operations based on the data delivered by the communication. Alternatively, the communication processing operation transmits the data generated by the computing processing operation to a specified location. The communication processing operation and the computing processing operation form a pipeline-like cooperation through data flow.

[0004] Current data parallel processing may cause some computing units to be in an idle state in a wave when the number of work tiles and the number of computing units are not in an integer multiple relationship, reducing the utilization rate of the processor. And especially in the implementation mode of overlapping execution of communication and computation, this idling will cause the total idle time cost to increase exponentially, greatly reducing the performance of the computation and communication fusion. SUMMARY

[0005] The application provides a data processing method applied to a parallel processor, the parallel processor comprising a plurality of computing units, the data processing method comprising performing a computing processing operation by the parallel processor, wherein the computing processing operation comprises a plurality of computing operations executed sequentially, input data of the computing processing operation is divided into a plurality of data blocks, and each computing operation is used for performing processing of one data block in the plurality of data blocks, wherein performing the plurality of computing operations of the computing processing operation by the parallel processor comprises: for each computing operation, determining at least one competitive work block in a plurality of work blocks according to a number of the plurality of work blocks included in input data of a target kernel function corresponding to the computing operation and a number of the plurality of computing units; and processing, in a pipeline manner, computing tasks of the plurality of work blocks by the plurality of computing units to complete the computing operation; wherein for the at least one competitive work block, an allocation relationship between the plurality of computing units and the computing tasks of the at least one competitive work block is dynamically determined by a competition mechanism, the competition mechanism comprises that the plurality of computing units compete for execution qualifications of the computing tasks of the at least one competitive work block, and a computing unit obtaining the execution qualification executes a corresponding competitive work block.

[0006] For example, in the data processing method provided by at least one embodiment of the application, the at least one wave corresponding to the target kernel function comprises a competitive wave, and a work block in the competitive wave is used as the at least one competitive work block, for any computing unit in the plurality of computing units, after the computing unit completes a computing task of a target work block, the computing unit cyclically uses an atomic operation to compete for an execution qualification of the at least one competitive work block, wherein the target work block comprises a work block in a previous wave of a first competitive wave in the at least one wave; in response to that the any computing unit obtains the execution qualification of any competitive work block, the any computing unit executes the computing task of the any competitive work block; and in response to that the any computing unit does not obtain the execution qualification of any competitive work block, the any computing unit is scheduled to execute a subsequent processor task.

[0007] For example, in the data processing method provided in at least one embodiment of the present application, the allocation relationship between the plurality of computing units and the computing tasks of the at least one competition work block is dynamically determined through a competition mechanism, including: initializing a competition workspace, wherein the competition workspace includes R lock variables initialized in an unlocked state, R is a positive integer and represents the number of the at least one competition work block, and the R lock variables correspond one-to-one to the at least one competition work block; using an atomic operation to compete for the R lock variables in the competition workspace by the plurality of computing units in a loop to determine whether any lock variable can be updated from the unlocked state to a locked state; determining that a first computing unit that updates a lock variable to the locked state obtains the execution qualification of the computing task of the competition work block corresponding to the lock variable, and the first computing unit exits the competition and executes the computing task of the competition work block corresponding to the lock variable; determining that a second computing unit that does not update any lock variable to the locked state ends the execution operation of the target kernel function and schedules the second computing unit to execute a subsequent processor task.

[0008] For example, in the data processing method provided in at least one embodiment of the present application, using an atomic operation to compete for the R lock variables in the competition workspace by the plurality of computing units in a loop to determine whether any lock variable can be updated from the unlocked state to a locked state, including: for any computing unit in the plurality of computing units: loop through the R lock variables and perform the atomic operation with the R lock variables respectively, wherein the atomic operation includes, in response to determining that a current lock variable is in the unlocked state, exchanging an identification value corresponding to the any computing unit to the current lock variable and returning operation success, and in response to determining that the current lock variable is not in the unlocked state, returning operation failure; in response to the atomic operation performed with the current lock variable returning the operation success, determining that the any computing unit updates the current lock variable to the locked state, and in response to the atomic operation performed with the current lock variable returning the operation failure, determining that the any computing unit does not update the current lock variable to the locked state, and the any computing unit continues the atomic operation with a next lock variable until the R lock variables are looped through.

[0009] For example, in the data processing method provided by at least one of the embodiments of the present application, the atomic operation includes an atomic compare-and-swap operation, and the atomic compare-and-swap operation includes: comparing a value of the current lock variable with a preset value corresponding to the unlocked state, in response to the value of the current lock variable being the same as the preset value, determining that the current lock variable is in the unlocked state, and swapping an identification value corresponding to the any one of the plurality of computing units into the current lock variable, and returning the preset value to indicate that the operation is successful, in response to the value of the current lock variable being different from the preset value, determining that the current lock variable is not in the unlocked state, and returning the value of the current lock variable to indicate that the operation is unsuccessful, wherein the identification value corresponding to each of the plurality of computing units is different from the preset value.

[0010] For example, in the data processing method provided by at least one of the embodiments of the present application, initializing the contention workspace includes: allocating a global memory region on a memory of the parallel processor as the contention workspace; and setting the R lock variables in the contention workspace, wherein the R lock variables are initialized to the unlocked state.

[0011] For example, in the data processing method provided by at least one of the embodiments of the present application, determining at least one contention work block in the plurality of work blocks according to a number of work blocks included in input data of a target kernel function corresponding to the computing operation and a number of the plurality of computing units includes: determining a theoretical number W of at least one wave corresponding to the target kernel function according to a number N of the plurality of work blocks and a number M of the plurality of computing units, wherein each wave includes a parallel computing task of M work blocks performed by M computing units to complete one round of parallel task; determining a contention wave in the at least one wave according to the theoretical number W; and determining work blocks in the contention wave as the at least one contention work block, wherein M, N, and W are positive integers.

[0012] For example, in the data processing method provided by at least one of the embodiments of the present application, determining a contention wave in the at least one wave according to the theoretical number W includes: in response to W > 1, determining an mth executed wave and all waves after the mth executed wave in the W waves as the contention wave, m being a preset positive integer and being less than or equal to W, and in response to W = 1, determining the at least one wave as the contention wave.

[0013] For example, in the data processing method provided by at least one embodiment of the present application, the scheduling of the any one computing unit to execute the subsequent processor task includes: in response to the computing operation corresponding to the target kernel function not being the last computing operation executed in the plurality of computing operations, scheduling the any one computing unit to immediately execute the computing task in the kernel function corresponding to the next computing operation.

[0014] For example, in the data processing method provided by at least one embodiment of the present application, for the other work blocks in the plurality of work blocks except the at least one competitive work block, the allocation relationship between the plurality of computing units and the computing tasks of the other work blocks is determined according to the static preset mapping relationship.

[0015] For example, in the data processing method provided by at least one embodiment of the present application, the computing processing operation and the communication processing operation with data dependency are executed in parallel to implement a fused operator, the communication processing operation includes a plurality of data copy operations, and at least one computing operation in the plurality of computing operations is executed in parallel with at least one data copy operation.

[0016] For example, in the data processing method provided by at least one embodiment of the present application, the parallel processor is any one processor in a plurality of processors included in a multi-processor system, the plurality of processors are configured to jointly execute a processing task, and each processor is configured to execute a part of the processing task, the processing task includes executing the fused operator, and the computing processing operation includes a part of the processing task executed by the parallel processor in the fused operator.

[0017] At least one embodiment of the present application provides a parallel processor including a plurality of computing units, the parallel processor is configured to execute a computing processing operation, wherein the computing processing operation includes a plurality of computing operations executed in sequence, input data of the computing processing operation is divided into a plurality of data blocks, each computing operation is used for executing processing of a data block in the plurality of data blocks, and when the parallel processor executes each computing operation, the following operations are included: determining at least one competitive work block in a plurality of work blocks included in input data of a target kernel function corresponding to the computing operation according to a quantity of the plurality of work blocks and a quantity of the plurality of computing units; and processing computing tasks of the plurality of work blocks in a pipeline manner by the plurality of computing units to complete the computing operation; wherein for the at least one competitive work block, an allocation relationship between the plurality of computing units and the computing tasks of the at least one competitive work block is dynamically determined through a competitive mechanism, the competitive mechanism includes that the plurality of computing units compete for execution qualification of the computing tasks of the at least one competitive work block, and a computing unit obtaining the execution qualification executes the computing tasks of the corresponding competitive work block.

[0018] This application provides at least one embodiment of an electronic device, including a parallel processor as described in at least one embodiment of this disclosure.

[0019] This application provides at least one embodiment of an electronic device, including: a memory that non-transitoryly stores computer-executable instructions; and a processor configured to run the computer-executable instructions, wherein the computer-executable instructions are executed by the processor to implement the data processing method according to at least one embodiment of this disclosure.

[0020] This invention application discloses at least one embodiment of a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the data processing method according to at least one embodiment of this disclosure.

[0021] The data processing method provided in this invention application introduces a competition wave and a dynamic competition mechanism when the competition wave takes effect. The competition mechanism dynamically determines the allocation relationship between the computing unit and the computing tasks corresponding to the work block, transforming the cumulative long-tail effect of multiple computing operations into a load problem that can be dynamically balanced, thereby eliminating the accumulated idle time and improving the overall performance. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0023] Figure 1 This is a schematic structural diagram of a general-purpose graphics processor;

[0024] Figure 2 This is a schematic diagram illustrating task allocation for a computing unit.

[0025] Figure 3A This is a schematic diagram of a converged computing scenario;

[0026] Figure 3B for Figure 3A The diagram shows the execution flow when performing matrix multiplication on the activation tensor and the weight tensor.

[0027] Figure 3C A schematic diagram illustrating task allocation for another type of computing unit;

[0028] Figure 4 A schematic flowchart illustrating a data processing method provided in at least one embodiment of this disclosure;

[0029] Figure 5A task allocation diagram of a computing unit provided for at least one embodiment of the present disclosure;

[0030] Figure 6 A schematic structural diagram of a parallel processor provided for at least one embodiment of the present disclosure;

[0031] Figure 7 A schematic structural diagram of a multi-processor system provided for at least one embodiment of the present disclosure;

[0032] Figure 8 A schematic structural diagram of an electronic device provided for at least one embodiment of the present disclosure;

[0033] Figure 9 A schematic block diagram of another electronic device provided for an embodiment of the present disclosure;

[0034] Figure 10 A schematic diagram of a non-transitory computer readable storage medium provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are some, but not all of the embodiments of the present disclosure. Based on the described embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the scope of the present disclosure.

[0036] Unless otherwise defined, technical terms or scientific terms used in the present disclosure shall have the ordinary meaning commonly understood by one of ordinary skill in the art to which the present disclosure belongs. The terms “first”, “second” and similar terms used in the present disclosure do not denote any order, quantity or importance, but are used to distinguish different components. The terms “include” or “contain” and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms “connect” or “connected” and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “up”, “down”, “left”, “right” and the like only represent relative positional relationships, and when the absolute positions of the described objects are changed, the relative positional relationships may also be changed accordingly. In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits the detailed description of some known functions and known components.

[0037] Figure 1 A schematic structural diagram of a general-purpose graphics processor.

[0038] As shown in Figure 1 , the general-purpose graphics processor is actually an array of programmable multi-processors, for example, the programmable multi-processors can be streaming processor clusters (SPC), for example, including Figure 1 a plurality of streaming processor clusters as shown in , the plurality of streaming processor clusters are of the same structure. In the general-purpose graphics processor, one streaming processor cluster processes one computing task, or multiple streaming processor clusters process one computing task. The multiple streaming processor clusters share data through a global cache or a global memory.

[0039] As shown in Figure 1 , one streaming processor cluster includes a plurality of computing units, and the plurality of computing units are of the same structure. Each computing unit (CU) is used to perform arithmetic logic operations, such as accumulation, reduction, general addition, subtraction, multiplication, division, etc. One computing unit includes a plurality of cores, each core including an arithmetic logic unit (ALU), a floating point computing unit, etc., and the core is used to perform a specific computing task. In addition, the computing unit also includes a register (for example, a register file in Figure 1 ) and a shared memory, which are used to store source data and destination data related to the computing task in a hierarchical manner, and the shared memory in one computing unit is used to share data among the cores in the computing unit.

[0040] As shown in Figure 1 , each computing unit is also provided with a tensor core, which is used to perform tensor-related calculations, for example, related tensor operations such as tensor contraction operations, and the tensor core can accelerate tensor operations such as matrix multiplication. The tensor cores in the plurality of computing units can be uniformly scheduled and controlled.

[0041] As shown in Figure 1 , each streaming processor cluster is also provided with a buffer, which is used for data caching of the plurality of computing units in the streaming processor cluster.

[0042] In parallel computing, a computing task is generally executed by a plurality of threads. These threads are divided into a plurality of thread blocks before being executed in the general-purpose graphics processor (or parallel computing processor), and then distributed through a thread block distribution module (TBD) Figure 1(Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0043] In each computing unit, the thread beam scheduling / distribution module ( Figure 1 (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can execute concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory-executed instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches, global caches, or global memory (e.g., [example cache]). Figure 1 High Bandwidth Memory (HBM) is used for read and write operations.

[0044] As mentioned above, the computing units of a graphics processor do not process work blocks one by one, but are scheduled in batches or waves. The GPU scheduler organizes computing tasks into multiple waves for execution, allowing as many computing units as possible to process tasks simultaneously in each wave.

[0045] In parallel processor programming (such as GPUs), a kernel is a piece of parallel code that runs on the device side and is initiated by the host (such as the central processing unit, CPU). It is the core logical unit for the parallel processor to execute computational tasks. The kernel defines the specific operations that each thread in the parallel processor needs to perform and is the key carrier for achieving data parallel computing. An operator may consist of one or more kernel functions, which are the carriers on the GPU where the operator is executed.

[0046] For a single kernel function, the number of working blocks in its input data is often not divisible by the number of computing units. For example, if there are 12 working blocks and 5 computing units, the parallel processor will process them in 3 waves: the first two waves each have 5 computing units processing 5 working blocks, the last wave has 2 computing units processing 2 working blocks, and the remaining 3 computing units are idle in this wave.

[0047] Therefore, for the above case, the utilization rate of the computing units of the last wave is extremely low. This resource idling at the end of the kernel function execution due to load imbalance is called long tail effect. It directly reduces the utilization rate of the parallel processor hardware and prolongs the execution time of a single operator.

[0048] Figure 2 A schematic diagram for task allocation of a computing unit.

[0049] In Figure 2 , the input data of the kernel function is divided into 12 work blocks, namely Figure 2 work block 0, work block 1, work block 2, …, work block 10, and work block 11 in

[0050] For example, in the example of Figure 2 , the data blocks are allocated in a static fixed mapping manner when launching the parallel processor to execute the kernel function. For example, computing unit 0 fixedly executes the processing of data block 0, data block 5, and data block 10, computing unit 1 fixedly executes the processing of data block 1, data block 6, and data block 11, computing unit 2 fixedly executes the processing of data block 2 and data block 7, computing unit 3 fixedly executes the processing of data block 3 and data block 8, and computing unit 4 fixedly executes the processing of data block 4 and data block 9. In this way, the launch operation is simplified.

[0051] However, in this case, for an input data, only 2 computing units are in working state in the third wave, and the other 3 computing units are in idle state, reducing the hardware resource utilization rate of the graphics processor.

[0052] In distributed computing scenarios such as pre-training and post-training of large models, sequence parallel and tensor parallel can be used to split tensors. Among them, sequence parallel splits input sequences according to the token dimension, and each processor only processes a part of tokens to reduce memory occupation and computation; tensor parallel splits the weight tensor of the model according to the dimension to obtain multiple sub-tasks for processing multiple split tensors, and the multiple sub-tasks are distributed on different processors and executed in parallel.

[0053] Therefore, by sequence parallel or tensor parallel, the computing task can be divided into multiple sub-tasks, each sub-task performs a part of the tensor computation operation, and multiple processors can execute multiple sub-tasks in parallel to complete the computing task, thereby improving the throughput and shortening the completion time of the computing task.

[0054] There are often scenarios in which communication and computation are executed in parallel in large models. For example, the AllGather + GEMM mode or the GEMM + ReduceScatter mode. Current data communication and computation are performed in a serial manner on a pipeline.

[0055] Fusion is an optimization technique that implements the overlap of computation and communication. A large computation kernel function is split into multiple small kernel functions, which are interleaved with communication kernel functions to achieve the overlap of computation and communication, thereby improving the overall hardware utilization and program throughput.

[0056] Figure 3A FIG. 1 is a schematic diagram of a fusion scenario.

[0057] As shown in FIG. 1, the activation tensor is divided into four parts, which are denoted as tensor A0, tensor A1, tensor A2, and tensor A3. Figure 3A As shown in FIG. 1, the activation tensor is divided into four parts, which are denoted as tensor A0, tensor A1, tensor A2, and tensor A3. Figure 3A As shown in FIG. 1, the activation tensor is divided into four parts, which are denoted as tensor A0, tensor A1, tensor A2, and tensor A3. Figure 3A As shown in FIG. 1, the activation tensor is divided into four parts, which are denoted as tensor A0, tensor A1, tensor A2, and tensor A3.

[0058] To hide the communication time consumption in the computation time consumption as much as possible, the computation processing operation and the communication processing operation are usually divided into multiple sub-operations, and are executed in parallel.

[0059] Figure 3B FIG. 2 is an execution flow diagram of matrix multiplication of the activation tensor and the weight tensor shown in FIG. 1. Figure 3A As shown in FIG. 2, the parallel processor a includes one communication stream and one computation stream. The computation stream includes multiple matrix multiplication operations that are sequentially executed, which are W0xA0, W0xA1, W0xA2, and W0xA3, respectively. The communication stream of the parallel processor a includes multiple data copy operations that are sequentially executed, which are copying tensor A0, copying tensor A1, copying tensor A2, and copying tensor A3, respectively.

[0060] Figure 3B As shown in FIG. 2, the parallel processor a includes one communication stream and one computation stream. The computation stream includes multiple matrix multiplication operations that are sequentially executed, which are W0xA0, W0xA1, W0xA2, and W0xA3, respectively. The communication stream of the parallel processor a includes multiple data copy operations that are sequentially executed, which are copying tensor A0, copying tensor A1, copying tensor A2, and copying tensor A3, respectively.

[0061] As shown in FIG. 2, the parallel processor a includes one communication stream and one computation stream. The computation stream includes multiple matrix multiplication operations that are sequentially executed, which are W0xA0, W0xA1, W0xA2, and W0xA3, respectively. The communication stream of the parallel processor a includes multiple data copy operations that are sequentially executed, which are copying tensor A0, copying tensor A1, copying tensor A2, and copying tensor A3, respectively.

[0062] ​The same applies to parallel processors c and d, which will not be elaborated here.

[0063] exist Figure 3B In this context, the horizontal direction represents the time axis, while overlapping matrix multiplication and data copying operations in the vertical direction are executed in parallel, such as the matrix multiplication operation W0×A0 and the operation of copying tensor A1.

[0064] Therefore, the matrix multiplication between the activation tensor and the weight tensor is divided into several smaller matrix multiplication operations, namely W0×A0, W0×A1, W0×A2, W0×A3, etc., and the communication operations and computation operations are executed in parallel as much as possible to improve the utilization of hardware resources and hide the time consumption caused by communication.

[0065] As mentioned above, in the context of integrated computing, a large kernel function is broken down into multiple consecutive smaller kernel functions, for example... Figure 3A The matrix multiplication between the activation tensor and the weight tensor executed by the parallel processor 0 shown is broken down into multiple consecutively executed matrix multiplication operations, namely W0×A0, W0×A1, W0×A2, and W0×A3, etc. Each matrix multiplication operation can be regarded as an independently executed kernel function.

[0066] Figure 3C A schematic diagram illustrating task allocation for another type of computing unit.

[0067] like Figure 3C As shown, for Figure 3A In the computational scenario shown, for parallel processor 0, four matrix multiplication operations are performed, namely W0×A0, W0×A1, W0×A2, and W0×A3, to complete the computational tasks of the portions of the activation tensor and weight tensor allocated to processor 0.

[0068] For each small kernel function (e.g., W0×A0), the input data is divided into, for example, 12 working blocks. Figure 3C In the diagram, A0_0-A0_11 represents the 12 working blocks of tensor A0, A1_0-A1_11 represents the 12 working blocks of tensor A1, A2_0-A2_11 represents the 12 working blocks of tensor A2, and A3_0-A3_11 represents the 12 working blocks of tensor A3.

[0069] In the case of static allocation, computation unit 0 is fixed to perform work block 0 for each input data (e.g., Figure 3C A0_0, A1_0, A2_0, A3_0 in the code), working block 5 (e.g. Figure 3C A0_5, A1_5, A2_5, A3_5 in the example), working block 10 (e.g. Figure 3CA0_10, A1_10, A2_10, A3_10) of the input data. The computing unit 2 is fixed to perform the computing task of each input data's work block 2 (e.g. Figure 3C A0_2, A1_2, A2_2, A3_2) of the input data. The computing unit 4 is fixed to perform the computing task of each input data's work block 4 (e.g. Figure 3C A0_7, A1_7, A2_7, A3_7) of the input data. The computing unit 6 is fixed to perform the computing task of each input data's work block 6 (e.g. Figure 3C A0_4, A1_4, A2_4, A3_4) of the input data. The computing unit 8 is fixed to perform the computing task of each input data's work block 8 (e.g. Figure 3C A0_9, A1_9, A2_9, A3_9) of the input data. And so on.

[0070] As shown in Figure 3C As shown in Figure 3A The computing operation of the activation tensor and the weight tensor shown in the above figure will independently produce a long tail effect in each small-scale kernel function (1 matrix multiplication operation) during execution. These long tail effects will be superimposed on each other, resulting in a total idle time that increases exponentially. This is because in parallel computing, the task load cannot be completely evenly distributed, resulting in a large number of computing units being in an idle state at the end of the task execution, and only a small number of units being in a working state, causing overall resource waste. The final result is that the cumulative execution time of multiple small kernel functions will significantly exceed the execution time of a single large kernel function, which is contrary to the original intention of performance improvement in the fusion, and has become a serious performance bottleneck on this technical path.

[0071] The data processing method provided by at least one embodiment of the present disclosure is applied to a parallel processor, and the parallel processor includes a plurality of computing units. The data processing method includes performing a computing processing operation by the parallel processor, where the computing processing operation includes a plurality of computing operations performed sequentially, and input data of the computing processing operation is divided into a plurality of data blocks, and each computing operation is used to perform processing of one data block in the plurality of data blocks. The performing of the plurality of computing operations included in the computing processing operation by the parallel processor includes: for each computing operation, determining at least one competitive work block in a plurality of work blocks according to a number of the plurality of work blocks included in input data of a target kernel function corresponding to the computing operation and a number of the plurality of computing units; and performing, by the plurality of computing units in a pipeline manner, a computing task of the plurality of work blocks to complete the computing operation. For the at least one competitive work block, an allocation relationship between the plurality of computing units and the computing task of the at least one competitive work block is dynamically determined by a competition mechanism, and the competition mechanism includes that the plurality of computing units compete for execution qualification of the computing task of the at least one competitive work block, and a computing unit obtaining the execution qualification performs a computing task of a corresponding competitive work block.

[0072] The data processing method provided by at least one embodiment of the present disclosure converts the cumulative long tail effect of the plurality of computing operations included in the computing processing operation into a load problem that can be dynamically balanced by introducing a competitive work block, applying a dynamic competition mechanism to the competitive work block, and dynamically determining the allocation relationship between the computing unit and the computing task of the competitive work block by the competition mechanism, so as to eliminate the cumulative idle time and improve the overall performance.

[0073] Specifically, the data processing method provided by at least one embodiment of the present disclosure no longer passively accepts uneven load at the end of each kernel function, but actively performs peak shaving, and in the competitive work block of each kernel function, instead of pre-designating several computing units to process the remaining work blocks, the plurality of computing units compete for the work load by a fair competition mechanism. In this way, the idle computing unit in the first kernel function can obtain the work of the competitive work block in the second kernel function, and the idle computing unit in the second kernel function can obtain the work of the competitive work block in the third kernel function. From a macro perspective, the work load of all computing units in the entire task becomes more balanced, and the overall idle time is compressed, so as to eliminate the cumulative long tail effect and improve the hardware utilization rate and execution efficiency.

[0074] The embodiments of the present disclosure are described in detail below with reference to the drawings, but the present disclosure is not limited to these specific embodiments.

[0075] The data processing method provided in at least one embodiment of the present disclosure can be applied to a parallel processor, and the parallel processor includes a graphics processor, a general-purpose graphics processor, a tensor processing unit (TPU), a data processing unit (DPU), a neural processing unit (NPU), and the like, which can perform parallel processing on data, and the present disclosure does not make a specific limitation in this regard.

[0076] For example, in some embodiments, the parallel processor can be a graphics processor or a general-purpose graphics processor, and the hardware architecture thereof can refer to the related description of Figure 1 .

[0077] For example, the parallel processor includes a plurality of computing units, which can perform the same or different computing tasks in parallel, for example, can perform the same processing on different input data.

[0078] The data processing method provided in at least one embodiment of the present disclosure includes performing a computing processing operation by using a parallel processor.

[0079] For example, the computing processing operation includes a plurality of computing operations performed in sequence, and the input data of the computing processing operation is divided into a plurality of data blocks, and each computing operation is used to perform processing on one data block in the plurality of data blocks.

[0080] The computing processing operation can be any computing operation that needs to be performed by the parallel processor, for example, a matrix multiplication operation, and the like, and the present disclosure does not make a specific limitation in this regard.

[0081] For example, in the foregoing Figure 3A and Figure 3B examples, the computing processing operation can be a matrix multiplication processing performed by the parallel processor a, the plurality of computing operations can be a matrix multiplication operation A0xW0, a matrix multiplication operation A1xW0, a matrix multiplication operation A2xW0, and a matrix multiplication operation A3xW0, and the plurality of data blocks can be a tensor A0, a tensor A1, a tensor A2, and a tensor A3. Of course, the present disclosure is not limited thereto, and for different operators, the computing processing operation, the computing operation, and the data block can be different, and the present disclosure does not make a specific limitation in this regard.

[0082] For example, in one example, the computing processing operation is performed in parallel with a communication processing operation with data dependency to implement a compute communication fused operator. For example, the compute communication fused operator can be an operator in which communication and computation are performed in overlap, for example, in an AllGather + GEMM manner, or a GEMM + ReduceScatter manner, and the like, and the present disclosure does not make a specific limitation in this regard.

[0083] The communication processing operation includes a plurality of data copy operations, and at least one of the plurality of computing operations is executed in parallel with at least one of the data copy operations.

[0084] For example, referring to the example of Figure 3B , the communication processing operation includes copying tensor A0, copying tensor A1, copying tensor A2, and copying tensor A3, and in the execution period of the computing operation A0xW0, copying tensor A1 is also executed in parallel, in the execution period of the computing operation A1xW0, copying tensor A2 is also executed in parallel, and so on. The fused operator splits a large computing processing operation into a plurality of small computing operations, and executes the data copy operations in between, to achieve the overlap of computing and communication, and improve the overall hardware utilization and program throughput.

[0085] It should be noted that the present disclosure is not limited thereto, and the present disclosure does not make specific limitations on the parallel execution manner of the communication processing operation and the computing processing operation.

[0086] For example, in one example, the parallel processor is any one of a plurality of processors included in a multi-processor system, the plurality of processors are configured to jointly execute a processing task, and each processor is configured to execute a part of the processing task, the processing task including executing the fused operator, and the computing processing operation includes the part of the processing task executed by the parallel processor in the fused operator.

[0087] For example, in the example of Figure 3B , the parallel processor can be any one of the parallel processor a, the parallel processor b, the parallel processor c, and the parallel processor d in Figure 3B . The four parallel processors are configured to jointly complete the matrix multiplication of the weight tensor and the activation tensor in Figure 3A , and the computing processing operation can include the four matrix multiplication operations executed by the parallel processor.

[0088] For example, the plurality of processors can be a graphics processor, a general-purpose graphics processor, etc.

[0089] For example, the multi-processor system can be a multi-card (multi-GPU) system, which integrates multiple GPU cards, and is an architecture for completing large-scale parallel computing tasks by cooperative work, and is commonly used in, for example, artificial intelligence training, scientific computing, high-performance graphics rendering, etc.

[0090] Figure 4 The schematic flowchart of the data processing method provided by at least one embodiment of the present disclosure is shown.

[0091] For example, the computing processing operation is executed by the parallel processor, including executing a plurality of computing operations by the parallel processor, and when each computing operation is executed, it can includeFigure 4 Steps S10 and S20 in the process.

[0092] In step S10, based on the number of multiple working blocks and the number of multiple computing units included in the input data of the target kernel function corresponding to the computation operation, at least one competing working block among the multiple working blocks is determined.

[0093] For example, with Figure 3A For example, when the parallel processor is parallel processor a, the target kernel function can be kernel1 corresponding to the computation operation A0×W0, kernel2 corresponding to the computation operation A1×W0, kernel3 corresponding to the computation operation A2×W0, or kernel4 corresponding to the computation operation A3×W0. The same principle applies to other parallel processors, and will not be elaborated further here.

[0094] For example, taking the target kernel function used to perform the computation operation A0×W0 as an example, the input data are A0 and W0. The input data can be divided into multiple working blocks, and the tensors A0 and W0 can also be divided into multiple working blocks, for example... Figure 3C For example, tensor A0 can be divided into 12 working blocks, including tensors A0_0 to A0_11. Similarly, tensor W0 can be divided into 12 working blocks of the same size. Each computational unit executes the computational task of one working block, for example, executing the computational tasks of the corresponding working blocks in tensors A0_0 and W0. The same principle applies when the target kernel function is used for other computational operations, which will not be elaborated further here.

[0095] The number of computing units can be the total number of computing units included in a parallel processor for performing computational operations.

[0096] For example, in some embodiments, step S10 may include: determining the theoretical number W of at least one wave corresponding to the target kernel function based on the number N of the plurality of working blocks and the number M of the plurality of computing units, wherein each wave includes the computational task of M working blocks performed in parallel by M computing units to complete one round of parallel task; determining the competing wave among the at least one wave based on the theoretical number W; and determining the working blocks in the competing wave as the at least one competing working block, wherein M, N, and W are all positive integers.

[0097] The input data of the target kernel function is divided into multiple working blocks. Each computing unit executes a computing task for one working block at a time. Each wave consists of multiple computing units performing multiple computing tasks in parallel to complete one round of parallel tasks.

[0098] In a parallel processor such as a graphics processor, a wave is essentially a set of thread collections that execute the same instruction synchronously, and is a logical grouping of compute units for a parallel processor to schedule to perform a task. The compute units in a wave are usually launched synchronously and perform the same or different tasks.

[0099] In a wave, M compute units perform M computing tasks in parallel to complete a round of parallel tasks. For example, a parallel processor includes 5 compute units to perform computing operations, in a wave, the 5 compute units perform computing tasks of 5 work blocks in parallel, for example, in the first wave, the 5 compute units perform computing tasks of work block A0_0, work block A0_1, work block A0_2, work block A0_3, and work block A0_4 in parallel.

[0100] It should be noted that the at least one wave corresponding to the target kernel function refers to the theoretical number of waves obtained when the plurality of compute units are used to execute only the target kernel function. For example, the theoretical number of the at least one wave corresponding to the target kernel function W can be determined according to the following formula: W = ceil(N / M). Wherein, ceil() is a rounding up function, N represents the number of work blocks, and M represents the number of compute units.

[0101] For example, referring to Figure 3C , the at least one wave corresponding to the kernel function kernel1 can be the first wave to the third wave (the first column to the third column from the left), the at least one wave corresponding to the kernel function kernel2 can be the fourth wave to the sixth wave (the fourth column to the sixth column from the left), the at least one wave corresponding to the kernel function kernel3 can be the seventh wave to the ninth wave (the seventh column to the ninth column from the left), and the at least one wave corresponding to the kernel function kernel4 can be the tenth wave to the twelfth wave (the tenth column to the twelfth column from the left).

[0102] In actual execution of the parallel processor, for the first computing operation, the at least one wave corresponding to the target kernel function is also actually executed, but for subsequent computing operations, since some computing tasks can be started in advance, the wave actually executed by the parallel processor and the at least one wave corresponding to the target kernel function can not be exactly the same.

[0103] The at least one wave is executed in a pipelined sequence, so the at least one wave has a predetermined execution order. Even if some computing tasks are started in advance in actual execution, the relative order in the kernel function still maintains the predetermined execution order.

[0104] For example, in some embodiments, determining the competitive wave in the at least one wave according to the theoretical number of the at least one wave can include: in response to W > 1, determining the mth executed wave in the W waves and all waves after the mth executed wave as the competitive wave, m being a preset positive integer and being less than or equal to W, and in response to W = 1, determining the one wave as the competitive wave.

[0105] For example, in response to W > 1, the mth executed wave in the W waves is determined as the first wave, and the competitive wave includes the first wave and all waves executed after the first wave (if any), that is, a certain wave and all waves after the certain wave (if any) can be regarded as the competitive wave.

[0106] m is a preset value, and the same m can be used for different kernel functions of the calculation processing operation, for example, m = W, and only the last wave is regarded as the competitive wave; or m = 2, and the second wave and all waves after the second wave are regarded as the competitive wave. The setting of m can be set according to actual needs, and the present disclosure does not make specific limitations thereto.

[0107] For example, in some embodiments, the competitive wave is determined according to the theoretical number of the at least one wave. Figure 3C For example, in the example shown in FIG. 6, N = 12, M = 5, W = ceil(12 / 5) = 3, m = W = 3, and since W > 1, the third wave is regarded as the competitive wave, and the work blocks A0_10 and A0_11 in the competitive wave are regarded as the competitive work blocks.

[0108] If W = 1, there is only one wave, and the wave is directly regarded as the competitive wave, that is, all work blocks are regarded as competitive work blocks.

[0109] For example, in some embodiments, the competitive work blocks can be specified, for example, the work blocks A0_10 and A0_11 are specified as the competitive work blocks.

[0110] For example, in some embodiments, only the last wave in the at least one wave is regarded as the competitive wave, and the work blocks in the competitive wave are regarded as the competitive work blocks. In this embodiment, the overhead of the competition mechanism and the atomic operation is only generated in the last wave with the least calculation load, that is, only the last wave is regarded as the competitive wave, and for other waves which are the main part, the execution manner is the same as that of the traditional method, without any additional overhead. This design ensures that the core performance bottleneck is solved while not causing significant negative impact on the overall calculation performance, the performance overhead is controllable, and fine optimization is achieved.

[0111] In step S20, a plurality of calculation units process a plurality of work blocks in a pipeline manner to complete a calculation operation.

[0112] For the at least one competing work block, a competition mechanism is used to dynamically determine the allocation relationship between multiple computing units and the computing tasks of the at least one competing work block. The competition mechanism includes the competition among the multiple computing units for the execution qualification of the computing tasks of the at least one competing work block. The computing unit that obtains the execution qualification executes the computing tasks of the corresponding competing work block. Each computing unit executes the computing tasks of one competing work block at a time.

[0113] Therefore, unlike the pre-specified approach for competing work blocks, this method allows all computing units to compete for these workloads through a fair competition mechanism. This transforms the cumulative long-tail effect caused by multiple small-scale kernel functions in operators such as general computing fusion into a dynamically balanced load problem, thereby eliminating accumulated idle time and improving overall performance.

[0114] The other work blocks (e.g., non-contested work blocks) among multiple work blocks are still assigned according to the static preset mapping relationship, which determines the allocation relationship between the computing unit and the computing task corresponding to the work block.

[0115] For example, such as Figure 3C As shown, non-contested work blocks can be work blocks A0_0-A0_9, A1_0-A1_9, A2_0-A2_9, and A3_0-A3_9, while contested work blocks can be work blocks A0_10-A0_11, A1_10-A1_11, A2_10-A2_11, and A3_10-A3_11.

[0116] For example, for these working blocks located in normal wavelets, refer to Figure 3C For example, each computing unit performs the computing task corresponding to a fixed work block. For instance, computing unit 3 performs the computing task for each input data work block 3 (e.g., ...). Figure 3C A0_3, A1_3, A2_3, A3_3 in the example), working block 8 (e.g. Figure 3C The computational task of A0_8, A1_8, A2_8, A3_8 in the dataset.

[0117] For example, at least one wave corresponding to the target kernel function includes a competing wave, and the working blocks in the competing wave serve as the at least one competing working block. For any one of the plurality of computing units, after the computing unit completes the computation task of the target working block, atomic operations are used cyclically to compete for the execution qualification of the at least one competing working block, wherein the target working block includes working blocks located in the previous wave that is before and adjacent to the first competing wave; in response to any computing unit obtaining the execution qualification of any competing working block, the computing unit executes the computation task of the competing working block; in response to any computing unit not obtaining the execution qualification of any competing working block, the computing unit is scheduled to execute subsequent processor tasks.

[0118] For example, in response to the number of the at least one wave being greater than 1, the previous wave belongs to the at least one wave; in response to the number of the at least one wave being equal to 1, the previous wave does not belong to the at least one wave, for example, it belongs to a wave in the previous computational operation or a wave in the previous computational processing operation.

[0119] For example, in one example, still taking a work block consisting of A0_0-A0_11 and a computation unit count of 5, work blocks A0_10 and A0_11 in the third wave are competing work blocks, while the other work blocks A0_0-A0_9 are non-competitive work blocks. Since the number of waves is greater than 1, each computation unit still allocates the non-competitive work blocks in the first two waves according to a static preset mapping relationship. For example, computation unit 3 is fixed to process work block 3 for each input data (e.g., ...). Figure 3C A0_3, A1_3, A2_3, A3_3 in the example), working block 8 (e.g. Figure 3C The computational task of A0_8, A1_8, A2_8, A3_8 in the dataset.

[0120] The competition wave is the third wave, and the working blocks in the previous wave are working blocks 5-9.

[0121] For computation unit 0, the target working block can be working block 5 for each input data (e.g., A0_5, A1_5, etc.). For computation unit 1, the target working block can be working block 6 for each input data (e.g., A0_6, A1_6, etc.). For computation unit 2, the target working block can be working block 7 for each input data (e.g., A0_7, A1_7, etc.). For computation unit 3, the target working block can be working block 8 for each input data (e.g., A0_8, A1_8, etc.). For computation unit 4, the target working block can be working block 9 for each input data (e.g., A0_9, A1_9, etc.).

[0122] After the target work block is executed by the computing unit, the atomic operation is used to compete for execution qualification of the at least one competition work block.

[0123] For example, if a computing unit obtains execution qualification of any competition work block, the computing unit executes the computing task of the competition work block; if a computing unit does not obtain execution qualification of any competition work block, the computing unit is scheduled to execute a subsequent processor task.

[0124] For example, in some embodiments, scheduling the any computing unit to execute the subsequent processor task can include: in response to the computing operation corresponding to the target kernel function not being the last executed computing operation in the plurality of computing operations, scheduling the any computing unit to immediately execute the computing task in the kernel function corresponding to the next computing operation.

[0125] That is, the subsequent processor task can be the computing task in the kernel function corresponding to the next computing operation. For example, the computing task can be the computing task determined according to the static preset mapping relationship.

[0126] The computing task in the kernel function corresponding to the next computing operation is performed in parallel with the corresponding computing task executed by the computing unit that obtains execution qualification of the competition work block.

[0127] In this embodiment, since the kernel functions corresponding to the computing operations are independent of each other, the computing task of the next kernel function can be started in advance, without waiting for the current kernel function to be executed, thereby reducing the overall idle waiting time.

[0128] If the computing operation corresponding to the target kernel function is the last executed computing operation in the plurality of computing operations, after the computing task of the competition work block is executed, the computing unit is scheduled to start the task of the next computing processing operation synchronously.

[0129] For example, in some embodiments, the dynamically determining, by the competition mechanism, the allocation relationship between the computing units and the computing tasks of the at least one competition work block can include: initializing a competition workspace, wherein the competition workspace includes R lock variables initialized in an unlocked state, R being a positive integer and representing the number of the at least one competition work block, and the R lock variables corresponding one-to-one to the at least one competition work block; using, by the plurality of computing units, an atomic operation to compete for the R lock variables in the competition workspace to determine whether any lock variable can be updated from the unlocked state to a locked state; determining that a first computing unit that updates one lock variable to the locked state obtains the execution qualification of the computing task of the competition work block corresponding to the one lock variable, the first computing unit exiting the competition and executing the computing task of the competition work block corresponding to the one lock variable; and determining that a second computing unit that does not update any lock variable to the locked state ends the execution operation of the target kernel function and schedules the second computing unit to execute a subsequent processor task.

[0130] For example, if the last wave is the competition wave, R = N - M × (W - 1).

[0131] A lock variable is a basic synchronization mechanism for implementing mutual exclusion in concurrent programming, and its core purpose is to ensure the mutual exclusion of multiple threads / processes accessing shared resources (i.e., only one thread / process can enter the critical section at the same time), avoiding data competition or inconsistency problems.

[0132] By binding the lock variable to the unprocessed work block, the consistency of access is guaranteed, and it is ensured that only one computing unit obtains the execution qualification of the work block.

[0133] For example, in some embodiments, initializing the competition workspace can include: allocating a global memory region on the memory of the parallel processor as the competition workspace; and setting R lock variables in the competition workspace, wherein the R lock variables are initialized in an unlocked state.

[0134] For example, in some embodiments, the competition workspace can include R lock variables, and the R lock variables are initialized in an unlocked state. Figure 1 As shown in the high-bandwidth memory of the graphics processor, a global memory region is pre-allocated as the competition workspace corresponding to the competition wave, which includes R lock variables initialized in an unlocked state, for example, initialized to 0, before the execution of the target kernel function.

[0135] All currently available computing units participate in the competition, and each computing unit tries to obtain a lock in the competition workspace by using an atomic operation through a loop. Here, the currently available computing units refer to the computing units in an idle state in the current wave actually executed by the parallel processor.

[0136] If the computing unit can update one lock variable in the competition workspace from the unlocked state to the locked state, it means that the computing unit successfully wins the lock, and determines that the computing unit has the execution qualification of the computing task of the work block corresponding to the lock variable, and the computing unit exits the competition loop and starts to execute the computing task of the work block obtained.

[0137] If the computing unit fails to successfully obtain any one of the R lock variables after traversing all the lock variables, i.e., fails to update any one of the lock variables from the unlocked state to the locked state, it means that all the remaining work of the current competition round has been claimed / allocated by other computing units, and the computing unit can directly end the execution of the current target kernel function and can be scheduled to execute subsequent processor tasks, such as the computing task of the next kernel function.

[0138] An atomic operation is a core concept in concurrent programming and computer architecture, which refers to a minimum operation unit that cannot be interrupted, and the execution process of the operation cannot be interrupted by other threads / processes or hardware, and the operation is either completely executed or not executed at all, and there is no intermediate state. The atomic operation can ensure that only one computing unit can change the lock variable at the same time, and other computing units can execute the operation on the lock variable only after the computing unit ends the operation, ensuring the consistency of access and ensuring that only one computing unit obtains the execution qualification of the work block.

[0139] For example, the atomic operation used by the computing unit can be one or a combination of multiple atomic operations, as long as the function of the atomic operation defined in the present disclosure can be implemented, and the present disclosure does not make specific limitations on specific atomic operations.

[0140] In this embodiment, the allocation relationship between the computing unit and the computing task corresponding to the competition work block is dynamically determined through the competition mechanism, the cumulative long tail effect caused by splitting the computing processing operation into multiple computing operations is eliminated, and the overall performance is improved: through the dynamic competition mechanism, the dispersed and fixed long tail load in multiple kernel functions is redistributed, so that the load of all computing units in the entire fusion task is more balanced, which fundamentally solves the cumulative long tail effect problem.

[0141] As described above, in order to realize parallel communication processing operation and computing processing operation, the multi-computing operator is executed in parallel through multiple computing operations and data copying operations, and the long tail effect caused in this process leads to a fold increase in idle time, which significantly reduces the performance of the multi-computing operator. The embodiment of the present disclosure can eliminate the cumulative long tail effect, and the total execution time of multiple computing operations can approach or even reach the performance of a single computing processing operation, ensuring the effectiveness of the multi-computing technology and improving the overall hardware utilization and program throughput.

[0142] For example, in some embodiments, the circulating, by the plurality of computing units, to contend for the R lock variables in the contention workspace, the atomic operation, to determine whether any of the lock variables can be updated from the unlocked state to the locked state, can comprise, for any of the plurality of computing units: looping through the R lock variables, performing the atomic operation with the R lock variables respectively, wherein the atomic operation comprises, in response to determining that a current lock variable is in the unlocked state, swapping an identification value corresponding to the any of the computing units to the current lock variable and returning operation success, and in response to determining that the current lock variable is not in the unlocked state, returning operation failure; in response to the atomic operation performed with the current lock variable returning the operation success, determining that the any of the computing units updates the current lock variable to the locked state, and in response to the atomic operation performed with the current lock variable returning the operation failure, determining that the any of the computing units does not update the current lock variable to the locked state, the any of the computing units continuing the atomic operation with a next lock variable until looping through the R lock variables.

[0143] For example, in some embodiments, the atomic operation comprises an atomic compare-and-swap operation, the atomic compare-and-swap operation comprising: comparing a value of a current lock variable with a preset value corresponding to the unlocked state, in response to the value of the current lock variable being the same as the preset value, determining that the current lock variable is in the unlocked state, and swapping an identification value corresponding to the any of the computing units to the current lock variable, and returning the preset value to indicate operation success, and in response to the value of the current lock variable being different from the preset value, determining that the current lock variable is not in the unlocked state, and returning the value of the current lock variable to indicate operation failure, wherein the identification value corresponding to each of the plurality of computing units is different from the preset value.

[0144] The identification value corresponding to each of the plurality of computing units can be the same or different from each other.

[0145] For example, the atomic compare-and-swap operation can be implemented using atomicCAS (&workspace_lock[j], value_unlock, my_unique_id ), where workspace_lock[j] represents the jth lock variable in the R lock variables, j is from 0 to R-1, my_unique_id is an identification value corresponding to the computing unit, and value_unlock represents a preset value corresponding to the unlocked state, for example, 0. atomicCAS represents an indivisible atomic operation, and the specific operation is to read the current value of workspace_lock[j]; compare whether the current value is equal to value_unlock; if equal, write my_unique_id to workspace_lock[j] and return value_unlock, and if not equal, directly return the current value in workspace_lock[j].

[0146] The R lock variables workspace_lock[0]-workspace_lock[R-1] correspond to the R competition work blocks one by one.

[0147] For example, each of the currently available computing units loops through the R lock variables (workspace_lock[0] to workspace_lock[R-1]). For example, for the ith computing unit (for example, i is from 0 to M-1), it performs an atomic compare-and-swap operation on workspace_lock[j], such as atomicCAS (&workspace_lock[j], value_unlock, my_unique_id ). If value_unlock is returned, it indicates that the operation is successful, and the index value my_unique_id corresponding to the ith computing unit is written into workspace_lock[j]. The ith computing unit successfully updates the lock variable workspace_lock[j] to the locked state, determines that the ith computing unit obtains the execution qualification of the computing task of the competition work block corresponding to workspace_lock[j], the ith computing unit exits the competition, and executes the computing task of the competition work block corresponding to workspace_lock[j]. If value_unlock is not returned, it indicates that the operation fails, and the atomic compare-and-swap operation with the next lock variable is continued to be tried until the R lock variables are traversed.

[0148] If the i-th computing unit fails to update any of the R lock variables to the locked state after traversing all the R lock variables, for example, each atomic compare-and-swap operation returns value_unlock, it indicates that all the remaining computing tasks of the competing work blocks have been assigned to other computing units, and it is determined that the i-th computing unit ends the execution of the target kernel function, and the i-th computing unit is scheduled to execute a subsequent processor task, for example, to continue the next kernel function.

[0149] It should be noted that the present disclosure is not limited to using the atomic compare-and-swap operation atomicCAS to implement atomic operations, and those skilled in the art can also use other atomic operations or combinations of multiple atomic operations to implement the above process, and the present disclosure does not make specific limitations thereon.

[0150] The following describes the specific process of applying the data processing method provided by at least one embodiment of the present disclosure, taking the scenario shown in FIGS. 1 and 2 as an example. Figure 3A and Figure 3B The following describes the specific process of applying the data processing method provided by at least one embodiment of the present disclosure, taking the scenario shown in FIGS. 1 and 2 as an example.

[0151] Figure 5 The following describes the specific process of applying the data processing method provided by at least one embodiment of the present disclosure, taking the scenario shown in FIGS. 1 and 2 as an example.

[0152] For example, the following describes the specific process of the computing and processing operation performed by the parallel processor a, and the same applies to the parallel processor b to the parallel processor d, which will not be described here.

[0153] Referring to Figure 3B The computing and processing operation performed by the parallel processor a includes four sequentially executed computing operations, which are A0×W0, A1×W0, A2×W0, and A3×W0. Figure 5 The task allocation manner of the four computing operations is shown in FIG. 4, and the ranges of the four kernel functions are divided by black thick lines.

[0154] Figure 5 The work block in gray shadow in FIG. 4 is a competing work block, and the work block in white background is a non-competing work block.

[0155] First, the kernel function kernel1 corresponding to the computing operation A0×W0 is performed.

[0156] Specifically, referring to step S10, at least one competing work block in the plurality of work blocks is determined according to the number of the plurality of work blocks included in the input data of the kernel function kernel1 and the number of the plurality of computing units.

[0157] As described above, the tensor A0 is divided into 12 work blocks, and the number of computing units is 5, so the theoretical number of at least one wave corresponding to the kernel function kernel1 for performing A0×W0 is 3, and the last wave is taken as a competing wave, and referring toFigure 3C The competing work blocks include work blocks A0_10 and A0_11 in the third wave.

[0158] Other non-contested work blocks include work blocks A0_0-A0_9. Non-contested work blocks are assigned tasks according to a static preset mapping relationship.

[0159] like Figure 5 As shown, in the first and second waves, the computational tasks of the work blocks are allocated to the computational units according to a static preset mapping relationship. In these two waves, each of the five computational units is assigned a work block according to a preset mapping rule and performs computations in parallel. For example, as... Figure 5 As shown, in the first two waves (such as the first and second columns from the left in 5), calculation unit 0 performs the calculation tasks of working blocks A0_0 and A0_5, calculation unit 1 performs the calculation tasks of working blocks A0_1 and A0_6, calculation unit 2 performs the calculation tasks of working blocks A0_2 and A0_7, and so on.

[0160] In the third wave, the contention wave, kernel1 still has 12 - 5 × 2 = 2 working blocks yet to be processed. The initialized contention workspace includes two lock variables, each corresponding to one of the two working blocks A0_10 and A0_11. The two lock variables are initialized to 0 to indicate an unlocked state. Each of the five computation units has a unique corresponding identifier value; for example, computation units 0 to 4 correspond to identifier values ​​1 to 5, respectively.

[0161] Each of the five computation units uses atomic operations in a loop to compete for two lock variables in an attempt to update either lock variable from an unlocked state to a locked state.

[0162] For example, computation unit 0 iterates through two lock variables and performs atomic comparison and swap operations on the lock variables, such as atomicCAS. For details, please refer to the aforementioned embodiment; further explanation is omitted here. If the atomic comparison and swap operation with the lock variable corresponding to work block A0_10 returns 0, it indicates that the operation was successful. Computation unit 0 has written the identifier value 1 to the lock variable corresponding to work block A0_10, confirming that computation unit 0 has obtained the qualification to execute the computation task of work block A0_10. Computation unit 0 then exits the competition and executes the computation task of work block A0_10.

[0163] At this time, the value of the lock variable corresponding to work block A0_10 is 1, which is a locked state. When other computing units perform atomic comparison and swap operations, the value of the lock variable corresponding to work block A0_10 is different from 0, and all return operation failure.

[0164] For example, after computation unit 1 performs an atomic comparison and swap operation with the lock variable corresponding to work block A0_11, it returns an operation failure. Computation unit 1 then performs another atomic comparison and swap operation with the lock variable corresponding to work block A0_11. For example, if it returns 0, it means that the operation was successful. Computation unit 1 writes the identifier value 2 into the lock variable corresponding to work block A0_11, confirming that computation unit 1 has obtained the qualification to execute the computation task of work block A0_11. Computation unit 1 withdraws from the competition and executes the computation task of work block A0_11.

[0165] At this time, the value of the lock variable corresponding to work block A0_11 is 2, which is a locked state. When other computing units perform atomic comparison and swap operations, the value of the lock variable corresponding to work block A0_11 is different from 0, and all return operation failure.

[0166] After traversing two lock variables, computation units 2-4 failed to update either lock variable to a locked state. Therefore, computation units 2-4 ended the execution of the target kernel function kernel1 and scheduled computation units 2-4 to immediately execute subsequent processor tasks, such as executing the operation of the next kernel function kernel2.

[0167] like Figure 5 As shown, in the third wave of execution by the parallel processor, computing units 0 and 1 obtain the execution qualification for work blocks A0_10 and A0_11. After computing units 0 and 1 complete the computation tasks of work blocks A0_10 and A0_11, they determine to end the execution operation for the target kernel function kernel1 and can schedule computing units 0 and 1 to execute subsequent processor tasks.

[0168] like Figure 5 As shown, in the third wave of execution by the parallel processor, computation units 2-4 did not qualify to execute the remaining working blocks in kernel1. Therefore, computation units 2-4 terminated the execution of the target kernel function kernel1 and could directly begin executing the computation tasks of working blocks A1_2-A1_4 in kernel function kernel2. The computation tasks executed by computation units 2-4 can be determined according to a static preset mapping relationship.

[0169] Next, the execution process of the kernel function kernel2 corresponding to the computation operation A1×W0 is described.

[0170] Specifically, referring to step S10, at least one competing working block is determined from the multiple working blocks based on the number of multiple working blocks and the number of multiple computing units included in the input data of the target kernel function kernel1.

[0171] As mentioned earlier, tensor A1 is divided into 12 working blocks, and the number of computational units is 5. Therefore, the theoretical number of waves corresponding to at least one kernel function kernel2 used to execute A1×W0 is 3. Taking the last wave as the competing wave, refer to... Figure 3C The competing working blocks include working blocks A1_10 and A1_11 in the third wave.

[0172] Other non-contested work blocks include work blocks A0_1-A1_9. Tasks are assigned to non-contested work blocks according to a static preset mapping relationship.

[0173] like Figure 5 As shown, in the third wave of execution by the parallel processor, computing units 2-4, having not obtained any qualification to execute any competing working blocks, directly begin executing working blocks A1_2-A1_4 of kernel2. In the fourth wave of execution by the parallel processor, computing unit 0 executes the computation task of working block A1_0 according to the preset mapping relationship, computing unit 2 executes the computation task of working block A1_7, computing unit 4 executes the computation task of working block A1_9, and so on.

[0174] In the fifth wave actually executed by the parallel processor, computing unit 0 executes non-contested work block A1_5 according to the preset mapping relationship, and computing unit 1 executes non-contested work block A1_6 according to the preset mapping relationship.

[0175] exist Figure 5 In the fifth wave, for non-contested work blocks A1_10 and A1_11, the initialized contested workspace includes two lock variables that correspond one-to-one with the two work blocks A1_10 and A1_11. The lock variables are initialized to 0 to indicate an unlocked state.

[0176] After computation units 2-4 have completed their respective target work blocks, each of the three computation units uses atomic operations in a loop to compete for two lock variables in an attempt to update either lock variable to a locked state.

[0177] For example, computation unit 2 iterates through the two lock variables and performs atomic comparison and swap operations on the lock variables, such as atomicCAS. For details, please refer to the aforementioned embodiment; further explanation is omitted here. If the atomic comparison and swap operation with the lock variable corresponding to work block A1_10 returns 0, it indicates that the operation was successful. Computation unit 2 has written the identifier value 3 into the lock variable corresponding to work block A1_10, confirming that computation unit 2 has obtained the qualification to execute the computation task of work block A1_10. Computation unit 2 then exits the competition and executes the computation task of work block A1_10.

[0178] At this time, the value of the lock variable corresponding to work block A1_10 is 3, which is a locked state. When other computing units perform atomic comparison and swap operations, the value of the lock variable corresponding to work block A1_10 is different from 0, and all return operation failure.

[0179] For example, after performing an atomic comparison and swap operation with the lock variable corresponding to work block A1_10, calculation unit 3 returns an operation failure. Then, calculation unit 3 performs an atomic comparison and swap operation with the lock variable corresponding to work block A1_11. For example, if it returns 0, it means that the operation is successful. Calculation unit 3 writes the identifier value 4 into the lock variable corresponding to work block A1_11 to determine that calculation unit 3 has obtained the qualification to execute the calculation task of work block A1_11. Calculation unit 3 withdraws from the competition and executes the calculation task of work block A1_11.

[0180] At this time, the value of the lock variable corresponding to work block A1_11 is 4, which is a locked state. When other computing units perform atomic comparison and swap operations, the value of the lock variable corresponding to work block A1_11 is different from 0, and all return operation failure.

[0181] After traversing the two lock variables, computation unit 4 did not update either lock variable to a locked state. Therefore, computation unit 4 terminates the execution of the target kernel function kernel2 and schedules computation unit 4 to execute subsequent processor tasks, such as directly starting the execution of the next kernel function kernel3. Figure 5 In the fifth wave, computing unit 4 executes the computation task of work block A2_4. The computation task executed by computing unit 4 at this time is also determined according to the static preset mapping relationship.

[0182] like Figure 5 As shown, in the 5th wave, computing units 2 and 3 obtain the execution qualification of work blocks A1_10 and A1_11. After computing units 2 and 3 finish executing the computing tasks of work blocks A1_10 and A1_11, they determine to end the execution operation on the target kernel function kernel2 and can schedule computing units 2 and 3 to execute subsequent processor tasks.

[0183] After computation units 0 and 1 finish executing the target working block, for example, after computation unit 0 finishes executing working block A1_5 and computation unit 1 finishes executing A1_6, computation units 0 and 1 also perform the above-mentioned operation of traversing the two lock variables. Since both lock variables have been updated to the locked state, computation units 0 and 1 do not update either lock variable to the locked state. It is determined that computation units 0 and 1 end the execution operation of the target kernel function kernel2, and schedule computation unit 4 to execute the subsequent processor task, such as executing the operation of the next kernel function kernel3.

[0184] Next, the execution process of the kernel function kernel3 corresponding to the calculation operation A2xW0 is described.

[0185] Similar to the process of the foregoing kernel function, the competitive work blocks include work blocks A2_10 and A2_11, and the other non-competitive work blocks include work blocks A2_1-A2_9. The non-competitive work blocks are assigned tasks according to the preset static mapping relationship.

[0186] As shown in FIG. 2B, for the non-competitive work blocks A2_1-A2_9, tasks are still assigned according to the foregoing preset static mapping relationship, for example, as shown in FIG. 2B, the calculation unit 0 executes the non-competitive work blocks A2_0 and A2_5, the calculation unit 1 executes the non-competitive work blocks A2_1 and A2_6, and so on. Figure 5 Figure 5 As shown in FIG. 2B, for the non-competitive work blocks A2_1-A2_9, tasks are still assigned according to the foregoing preset static mapping relationship, for example, as shown in FIG. 2B, the calculation unit 0 executes the non-competitive work blocks A2_0 and A2_5, the calculation unit 1 executes the non-competitive work blocks A2_1 and A2_6, and so on.

[0187] After the calculation unit 4 finishes executing the target work block A2_9, the calculation unit 4 cyclically uses an atomic operation to compete for 2 lock variables corresponding to the competitive work blocks A2_10 and A2_11, to attempt to update any one of the lock variables to a locked state.

[0188] For example, the calculation unit 4 cyclically traverses the 2 lock variables and performs an atomic compare-and-swap operation, such as atomicCAS, on the lock variables. Details can be referred to the foregoing embodiments, which will not be described herein. If the atomic compare-and-swap operation on the lock variable corresponding to the work block A2_10 returns 0, indicating that the operation is successful, the calculation unit 4 has written the identification value 5 to the lock variable corresponding to the work block A2_10, and determines that the calculation unit 4 obtains the execution qualification of the calculation task of the work block A2_10. The calculation unit 4 exits the competition and executes the calculation task of the work block A2_10.

[0189] After the calculation units 0-3 finish executing the respective target work blocks, each of the 4 calculation units cyclically uses an atomic operation to compete for 2 lock variables, to attempt to update any one of the lock variables to a locked state.

[0190] For example, the calculation unit 0 returns an operation failure after performing the atomic compare-and-swap operation on the lock variable corresponding to the work block A2_10. The calculation unit 0 performs the atomic compare-and-swap operation on the lock variable corresponding to the work block A2_11 again, for example, returns 0 indicating that the operation is successful. The calculation unit 0 writes the identification value 1 to the lock variable corresponding to the work block A2_11, determines that the calculation unit 0 obtains the execution qualification of the calculation task of the work block A2_11, exits the competition, and executes the calculation task of the work block A2_11.

[0191] ​At this time, the value of the lock variable corresponding to work block A2_10 is 5, and the value of the lock variable corresponding to work block A2_11 is 1, both of which are in the locked state. When the atomic compare-and-swap operation is performed by the other computing units 1-3, the operation fails. It is determined that the computing units 1-3 end the execution of the target kernel function kernel3, and the computing units 1-3 are scheduled to execute subsequent processor tasks, for example, directly starting the operation of the next kernel function kernel4, as shown in FIG. 8. Figure 5 In the eighth wave, the computing units 1-3 perform the computing tasks of work blocks A3_1-A3_3.

[0192] After the computing unit 4 completes the execution of work block A2_10, it is determined that the operation of the current kernel function kernel3 ends, and the operation of the next kernel function kernel4 starts, as shown in FIG. 8. Figure 5 In the eighth wave, the computing unit 4 performs the computing task of work block A3_4.

[0193] Next, the execution process of the kernel function kernel4 corresponding to the computing operation A3xW0 is described.

[0194] Similar to the process of the foregoing kernel function, the competing work blocks include work blocks A3_10 and A3_11, and the other non-competing work blocks include work blocks A3_1-A3_9. The non-competing work blocks are assigned tasks according to the preset static mapping relationship.

[0195] As shown in FIG. 8, for the non-competing work blocks A3_1-A3_9, the tasks are still assigned according to the foregoing preset static mapping relationship, for example, as shown in FIG. 8, the computing unit 0 executes the non-competing work blocks A3_0 and A3_5, the computing unit 1 executes the non-competing work blocks A3_1 and A3_6, and so on. Figure 5 Figure 5 As shown in FIG. 8, for the non-competing work blocks A3_1-A3_9, the tasks are still assigned according to the foregoing preset static mapping relationship, for example, as shown in FIG. 8, the computing unit 0 executes the non-competing work blocks A3_0 and A3_5, the computing unit 1 executes the non-competing work blocks A3_1 and A3_6, and so on.

[0196] After the computing unit 1 completes the execution of the target work block A3_6, the computing unit 1 cyclically uses the atomic operation to compete for the two lock variables corresponding to the work blocks A3_10 and A3_11, to attempt to update any lock variable to the locked state.

[0197] For example, the computing unit 1 cyclically traverses the two lock variables and performs the atomic compare-and-swap operation, such as atomicCAS, on the lock variables. For details, reference can be made to the foregoing embodiments, which are not described herein again. If the atomic compare-and-swap operation on the lock variable corresponding to the work block A3_10 returns 0, it indicates that the operation is successful, the computing unit 1 has written the identification value 2 to the lock variable corresponding to the work block A3_10, and it is determined that the computing unit 1 obtains the execution qualification of the computing task of the work block A3_10. The computing unit 1 exits the competition and executes the computing task of the work block A3_10.

[0198] ​After computation unit 2 completes the execution of target work block A3_7, it uses atomic operations cyclically to compete for two lock variables, attempting to update either lock variable to a locked state. Referring to a similar process described above, if the atomic comparison and swap operation with the lock variable corresponding to work block A3_11 returns 0, it indicates success, confirming that computation unit 2 has acquired the qualification to execute the computation task of work block A3_11. Computation unit 2 then exits the competition and executes the computation task of work block A3_11.

[0199] After completing their respective target working blocks, computation units 0, 3, and 4 repeatedly use atomic operations to compete for two lock variables. At this point, working blocks A3_10 and A3_11 are both locked. When computation units 0, 3, and 4 attempt to perform atomic comparison and swap operations, all operations return failure. It is determined that computation units 0, 3, and 4 have finished executing the target kernel function kernel4, and are scheduled to execute subsequent processor tasks. Since kernel4 is the last computational operation, the computation units wait for the current computational operation to finish before starting other processor tasks.

[0200] like Figure 3C As shown, in this embodiment, the computational processing can be completed in just 10 waves, compared to... Figure 6 The reduction of two waves from the 12 waves effectively compresses the cumulative long-tail effect of multiple kernel functions in the fusion computing process. This results in a more balanced load across all computing units in the entire fusion task, allowing the total execution time of multiple small kernel functions to approach or even reach the performance of a single large kernel function. This ensures the effectiveness of the fusion computing technology and improves overall hardware utilization and program throughput. Furthermore, in this embodiment, the overhead of the competition mechanism and atomic operations only occurs in the last wave, which has the least computational load. For the majority of normal waves, the execution method is no different from the traditional method, with no additional overhead. This design ensures that while addressing core performance bottlenecks, it does not have a significant negative impact on overall computing performance.

[0201] At least one embodiment of this disclosure also provides a parallel processor. Figure 6 A schematic structural diagram of a parallel processor provided for at least one embodiment of this disclosure.

[0202] like Figure 1 As shown, the parallel processor 100 includes multiple computing units 101.

[0203] The parallel processor can be, for example, a graphics processing unit (GPU) or a general-purpose GPU. When the parallel processor is a GPU or a general-purpose GPU, its specific architecture can be found in [reference needed]. Figure 7 The relevant descriptions will not be repeated here.

[0204] The plurality of computing units can perform the same or different computing tasks in parallel.

[0205] The parallel processor 100 is configured to perform a computing processing operation, wherein the computing processing operation comprises a plurality of computing operations performed sequentially, and input data of the computing processing operation is divided into a plurality of data blocks, each computing operation is configured to perform processing of one data block of the plurality of data blocks.

[0206] When the parallel processor 100 performs each computing operation, the parallel processor 100 comprises the following operations: determining at least one competitive work block in the plurality of work blocks according to a number of the plurality of work blocks included in input data of a target kernel function corresponding to the computing operation and a number of the plurality of computing units; and performing computing tasks of the plurality of work blocks in a pipeline manner by the plurality of computing units to complete the computing operation.

[0207] For the at least one competitive work block, an allocation relationship between the plurality of computing units and the computing tasks of the at least one competitive work block is dynamically determined by a competition mechanism, the competition mechanism comprises that the plurality of computing units compete for execution qualifications of the computing tasks of the at least one competitive work block, and the computing unit obtaining the execution qualification performs the computing tasks of the corresponding competitive work block.

[0208] For example, in some embodiments, at least one wave corresponding to the target kernel function comprises a competitive wave, and a work block in the competitive wave is the at least one competitive work block, for any computing unit of the plurality of computing units, after the computing unit completes the computing tasks of a target work block, the target work block comprises a work block in a previous wave of the first competitive wave, the execution qualification of the at least one competitive work block is cyclically competed by using an atomic operation, and in response to that the any computing unit obtains the execution qualification of any competitive work block, the any computing unit performs the computing tasks of the any competitive work block, and in response to that the any computing unit does not obtain the execution qualification of any competitive work block, the any computing unit is scheduled to perform a subsequent processor task.

[0209] For example, in some embodiments, when the parallel processor dynamically determines the allocation relationship between the plurality of computing units and the computing tasks of the at least one competition work block through a competition mechanism, the parallel processor includes the following operations: initializing a competition workspace, wherein the competition workspace includes R lock variables initialized as an unlocked state, R is a positive integer and represents the number of the at least one competition work block, and the R lock variables correspond to the at least one competition work block one by one; the plurality of computing units cyclically compete for the R lock variables in the competition workspace using an atomic operation to determine whether any lock variable can be updated from the unlocked state to a locked state; a first computing unit that updates a lock variable to the locked state obtains the execution qualification of the computing task of the competition work block corresponding to the lock variable, and the first computing unit exits the competition and executes the computing task of the competition work block corresponding to the lock variable; a second computing unit that does not update any lock variable to the locked state ends the execution operation of the target kernel function, and schedules the second computing unit to execute a subsequent processor task.

[0210] For the specific process of dynamically determining the allocation relationship between the computing unit and the computing task corresponding to the work block by the parallel processor through the competition mechanism, reference can be made to the description of the related embodiments in the foregoing data processing method, which will not be repeated here.

[0211] For example, the computing processing operation and the communication processing operation with data dependency are executed in parallel to implement a compute-in-mem fusion operator, the communication processing operation includes a plurality of data copy operations, and at least one computing operation in the plurality of computing operations is executed in parallel with at least one data copy operation. For related content of the compute-in-mem fusion operator, reference can be made to the description of the related embodiments in the foregoing data processing method, which will not be repeated here.

[0212] Figure 7 A schematic structural diagram of a multi-processor system provided by at least one embodiment of the present disclosure is shown.

[0213] As shown in the multi-processor system Figure 7 , a plurality of computing nodes are shown. Of course, it can be understood that the multi-processor system can only include one computing node, which can be set as needed by those skilled in the art.

[0214] For example, as shown in Figure 7 , each processor can have its own memory, and each processor independently manages its own memory.

[0215] The parallel processor can be any one of the plurality of processors included in the multi-processor system.

[0216] As shown in Figure 8As shown, each processor can access the memory of other processors through the bus, and multiple processors can be interconnected through the bus.

[0217] The multiple processors in the computing node are configured to jointly perform a processing task, and each processor is configured to perform a part of the processing task.

[0218] For example, in at least one embodiment of the present disclosure, the computing task can be split in a tensor parallel or sequence parallel manner, and the complete activation tensor and weight tensor are split into multiple sub-tensors, each sub-task performs related operations on the corresponding sub-tensor, and multiple sub-tasks are distributed to multiple processors for parallel execution. Of course, the present disclosure is not limited to this, and those skilled in the art can also split the computing task in other ways and distribute it to multiple processors for parallel execution.

[0219] It should be noted that in at least one embodiment of the present disclosure, the multi-processor system can include more or fewer circuits or units, and the connection relationship between the circuits or units is not limited, and can be determined according to actual needs. The specific construction manner of each circuit or unit is not limited, which can be constituted by analog devices according to the circuit principle, or can be constituted by digital chips, or constituted in other applicable manners.

[0220] For example, the multi-processor system can be implemented in hardware, software, or a combination of hardware and software, and the present disclosure does not make specific limitations thereto.

[0221] In some embodiments, the multi-processor system provided by at least one embodiment of the present disclosure can be in the form of single-machine multi-card, integrating multiple GPU cards in a single server or workstation, and realizing card-to-card communication through a bus (such as a PCIe bus, an AMD bus, etc.). For example, the multi-processor system can also be multi-machine multi-card, and multiple single-machine multi-card servers are interconnected through a high-speed network to form a cluster, and each card can not only communicate with other cards on the local machine, but also interact with cards on other nodes through the network. For example, the multi-processor system can also be a TPU cluster, a multi-FPGA acceleration card system, etc., and the present disclosure does not make specific limitations thereto.

[0222] For example, the multi-processor system breaks through the single-card computing power or memory limit through parallel computing, and is widely used in scenarios requiring large-scale data processing, high-throughput computing, or low-latency parallel tasks. For example, deep learning and large model training, high-performance computing, large-scale data processing and AI inference, real-time rendering and visualization, etc., which can be applied to the fields of large model training, scientific computing, high-concurrency AI services, real-time rendering, etc.

[0223] The parallel processor provided by at least one embodiment of the present disclosure can effectively compress the cumulative long tail effect of a plurality of kernel functions in the fusion calculation, so that the load of all computing units in the entire fusion task is more balanced, and the total execution time of a plurality of small kernel functions can approach or even reach the performance of a single large kernel function, thereby guaranteeing the effectiveness of the fusion calculation technology and improving the overall hardware utilization and program throughput.

[0224] Figure 8 A schematic structural diagram of an electronic device provided by at least one embodiment of the present disclosure is provided.

[0225] As Figure 9 shown, the electronic device 200 provided by at least one embodiment of the present disclosure includes a parallel processor 100. The related description of the parallel processor 100 can refer to the foregoing introduction of the parallel processor, and will not be repeated here.

[0226] For example, according to actual needs, the electronic device can be a deep learning workstation, a data center server, a special AI server, etc. equipped with a multi-processor system. Alternatively, the electronic device can also be a supercomputing center cluster, a cloud service provider AI cluster, etc. The present disclosure does not make specific limitations on this.

[0227] The electronic device can be provided with other hardware, software or firmware for cooperative work with the multi-processor system according to needs, and the present disclosure does not make specific limitations on this.

[0228] The technical effects of the electronic device can refer to the related description of the parallel processor of the foregoing embodiments, which will not be repeated here.

[0229] Figure 9 A schematic block diagram of another electronic device provided by an embodiment of the present disclosure is provided. As Figure 9 shown, the electronic device 300 is suitable for implementing the data processing method provided by the embodiments of the present disclosure, for example. It should be noted that Figure 9 The components of the electronic device 300 shown are only exemplary and are not limiting. The electronic device 300 can also have other components according to actual application needs.

[0230] As Figure 9 shown, the electronic device 300 can include a processing device 301 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to non-transitory computer readable instructions stored in a memory to achieve various functions.

[0231] For example, when the computer readable instructions are executed by the processing device 301, one or more steps of the data processing method according to any of the above embodiments can be performed. It should be noted that the detailed description of the processing process of the data processing method can refer to the related description in the embodiments of the data processing method.

[0232] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory, etc., whereby computer-readable instructions can be loaded from storage device 308 into RAM 303 to execute. Non-volatile memory may include, for example, read-only memory (ROM) 302, hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB storage, flash memory, etc. Various applications and various data, such as various data used and / or generated by applications, may also be stored in the computer-readable storage medium.

[0233] For example, the processing device 301, the read-only memory (ROM) 302, and the random access memory (RAM) 303 are interconnected via a bus 304. The input / output (I / O) interface 305 is also connected to the bus 304.

[0234] Typically, the following devices can be connected to the input / output (I / O) interface 305: input devices 306 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 308 including, for example, magnetic tape, hard disk, flash memory, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 10 An electronic device 300 with various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown, and the electronic device 300 may alternatively implement or possess more or fewer devices. For example, a processing device 301 can control other components in the electronic device 300 to perform desired functions. The processing device 301 may be a device with data processing capabilities and / or program execution capabilities, such as a central processing unit (CPU), a tensor processor (TPU), or a graphics processing unit (GPU). The central processing unit (CPU) may be an x86, ARM, RISC-V architecture, etc. The GPU may be directly integrated into the SOC, directly integrated onto the motherboard, or built into the northbridge chip of the motherboard.

[0235] Figure 10 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of this disclosure. For example, such as ​As shown, the storage medium 400 can be a non-transitory computer-readable storage medium, and one or more computer-readable instructions 401 can be non-transitorily stored in the storage medium 400. For example, the one or more computer-readable instructions 401 can perform one or more steps of the data processing method according to the above description when executed by the processor.

[0236] For example, the storage medium 400 can be applied in the electronic device 300. For example, the storage medium 400 can include the storage apparatus 308 in the electronic device 300.

[0237] For example, the storage apparatus can include any combination of one or more computer program products. The computer program product can include various forms of computer-readable storage media for storing the computer-readable instructions, such as volatile memory (e.g., random access memory (RAM) and / or cache) and / or non-volatile memory (e.g., read-only memory (ROM), hard disk, erasable programmable ROM (EPROM), compact disc read-only memory (CD-ROM), USB memory, flash memory, and / or the like). One or more computer-readable instructions can be stored in the computer-readable storage medium, and the processor can execute the computer-readable instructions to implement various functions of the processor. Various application programs and various data and the like can also be stored in the storage medium.

[0238] For example, the storage medium can include a memory card of a smart phone, a cache component of a tablet computer, a hard disk of a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable ROM (EPROM), compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage medium, or other applicable storage medium.

[0239] The flowcharts and block diagrams in the attached drawings illustrate the possible architectural, functional, and operational scenarios of systems, methods, and computer program products in accordance with various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can represent a module, a segment, or a portion of code that comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that, in some alternative implementations, the functions noted in the blocks can occur in a different order than that noted in the attached drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, or they can be executed in reverse order, depending on the involved functions. It should also be noted that each block in the block diagrams and / or flowcharts, and a combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or they can be implemented by a combination of dedicated hardware and computer instructions.

[0240] The units described in the embodiments of the present disclosure can be implemented in the form of software, or can be implemented in the form of hardware. In some cases, the name of the unit does not constitute a limitation on the unit itself.

[0241] The functions described above in the present disclosure can be performed at least in part by one or more hardware logic components. For example, non-limiting example types of hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0242] The above description is merely the preferred embodiments of the present disclosure and the explanation of the principles of the applied technology. It should be understood by those skilled in the art that the disclosed range of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and also covers other technical solutions formed by any combinations of the above technical features or equivalent features without departing from the disclosed concept. For example, the technical solutions formed by replacing the above features with the technical features disclosed in the present disclosure (but not limited to) having similar functions.

[0243] In addition, although each operation is described in a particular order, this should not be understood as requiring the operations to be performed in the specific order shown or in a sequential order. In certain circumstances, multitasking and parallel processing can be advantageous. Similarly, although several implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be combined in a single embodiment. Conversely, various features described in the context of a single embodiment can also be separated and implemented in multiple embodiments.

[0244] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely exemplary forms of implementing the claims.

[0245] For the present disclosure, the following points need to be explained:

[0246] (1) The drawings of the embodiments of the present disclosure only involve the structures involved in the embodiments of the present disclosure, and other structures can be referred to the general design.

[0247] (2) In the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined to obtain new embodiments.

[0248] The above description is merely that of a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized in that, Applied to a parallel processor, the parallel processor comprising multiple computing units, The data processing method includes executing computational processing operations through the parallel processor, wherein the computational processing operations include multiple computational operations executed sequentially, the input data of the computational processing operations are divided into multiple data blocks, and each computational operation is used to process one of the multiple data blocks. The plurality of computational operations included in the computational processing operations performed by the parallel processor include: For each computational operation: Based on the number of multiple working blocks included in the input data of the target kernel function corresponding to the computational operation and the number of multiple computational units, at least one competing working block is determined among the multiple working blocks, wherein at least one wave corresponding to the target kernel function includes a competing wave, and the working block in the competing wave is the at least one competing working block; The computational tasks of the multiple work blocks are processed in a pipelined parallel manner by the multiple computing units to complete the computational operation; Specifically, for the at least one competing work block, a competition mechanism is used to dynamically determine the allocation relationship between the multiple computing units and the computing tasks of the at least one competing work block. The competition mechanism includes the multiple computing units competing for the qualification to execute the computing tasks of the at least one competing work block, and the computing unit that obtains the qualification to execute the computing tasks of the corresponding competing work block.

2. The data processing method according to claim 1, characterized in that, For any one of the plurality of computing units, after the computing unit has completed the computing task of the target working block, atomic operations are used cyclically to compete for the execution qualification of at least one competing working block, wherein the target working block includes the working block in the wave preceding the first competing wave in the at least one wave. In response to any computing unit obtaining the qualification to execute any competing work block, the computing unit executes the computing task of the competing work block. In response to any computing unit failing to qualify for execution of any competing work block, the computing unit is scheduled to execute subsequent processor tasks.

3. The data processing method according to claim 1, characterized in that, The allocation relationship between the computational tasks of the multiple computing units and the at least one competing work block is dynamically determined through a competition mechanism, including: Initialize a contention workspace, wherein the contention workspace includes R lock variables initialized to an unlocked state, where R is a positive integer and represents the number of the at least one contention work block, and the R lock variables correspond one-to-one with the at least one contention work block; The plurality of computing units cyclically use atomic operations to compete for the R lock variables in the competitive workspace to determine whether any lock variable can be updated from the unlocked state to the locked state; The first computing unit that updates a lock variable to the locked state is granted the qualification to execute the computing task of the competing work block corresponding to the lock variable. The first computing unit withdraws from the competition and executes the computing task of the competing work block corresponding to the lock variable. Once it is determined that the second computing unit has not updated any lock variable to the locked state, the execution operation of the target kernel function ends, and the second computing unit is scheduled to execute subsequent processor tasks.

4. The data processing method according to claim 3, characterized in that, The plurality of computing units cyclically use atomic operations to compete for the R lock variables in the competitive workspace to determine whether any lock variable can be updated from the unlocked state to the locked state, including: For any one of the plurality of computing units: The R lock variables are iterated over in a loop, and the atomic operation is performed on each of the R lock variables. The atomic operation includes, in response to determining that the current lock variable is in the unlocked state, swapping the identifier value corresponding to any one of the computing units to the current lock variable and returning a success operation, and in response to determining that the current lock variable is not in the unlocked state, returning a failure operation. In response to the atomic operation performed with the current lock variable returning a success value, it is determined that either computational unit updates the current lock variable to the locked state. In response to the atomic operation performed with the current lock variable returning an operation failure, and determining that any one of the computation units has not updated the current lock variable to the locked state, the computation unit continues to perform the atomic operation with the next lock variable until all R lock variables have been traversed.

5. The data processing method according to claim 4, characterized in that, The atomic operations include atomic comparison and exchange operations. The atom comparison and exchange operation includes: Compare the current value of the lock variable with the preset value corresponding to the unlocked state. In response to the fact that the value of the current lock variable is the same as the preset value, the current lock variable is determined to be in the unlocked state, and the identifier value corresponding to any one of the computing units is swapped to the current lock variable. The preset value is then returned to indicate that the operation was successful. In response to the fact that the value of the current lock variable is different from the preset value, it is determined that the current lock variable is not in the unlocked state, and the value of the current lock variable is returned to indicate that the operation has failed. The identifier value corresponding to each of the plurality of computing units is different from the preset value.

6. The data processing method according to claim 3, characterized in that, Initialize the contention workspace, including: A global memory region is allocated on the memory of the parallel processor to serve as the contention workspace; In the contested workspace, the R lock variables are set, wherein the R lock variables are initialized to the unlocked state.

7. The data processing method according to claim 1, characterized in that, Based on the number of multiple working blocks included in the input data of the target kernel function corresponding to the computational operation and the number of multiple computational units, at least one competing working block among the multiple working blocks is determined, including: Based on the number N of the plurality of working blocks and the number M of the plurality of computing units, the theoretical number W of at least one wave corresponding to the target kernel function is determined, wherein each wave includes the computing tasks of M working blocks performed in parallel by M computing units to complete one round of parallel tasks. Based on the theoretical quantity W, determine the competing wavelet among the at least one wavelet; The working block in the competitive wave is identified as the at least one competing working block. Where M, N, and W are all positive integers.

8. The data processing method according to claim 7, characterized in that, Determining the competing waves among the at least one wavelet based on the theoretical quantity W includes: In response to W > 1, the m-th wave executed out of W waves and all waves following the m-th wave are determined as the competing waves, where m is a preset positive integer less than or equal to W. In response to W=1, the at least one wavelet is determined to be the competing wavelet.

9. The data processing method according to claim 2, characterized in that, Scheduling any one of the computing units to execute subsequent processor tasks includes: In response to the fact that the computation operation corresponding to the target kernel function is not the last computation operation executed among the plurality of computation operations, the computation unit is scheduled to immediately execute the computation task in the kernel function corresponding to the next computation operation.

10. The data processing method according to claim 1, characterized in that, For the other work blocks among the plurality of work blocks, excluding the at least one competing work block, the allocation relationship between the plurality of computing units and the computing tasks of the other work blocks is determined according to a static preset mapping relationship.

11. The data processing method according to any one of claims 1-10, characterized in that, The computational processing operations are executed in parallel with the communication processing operations that have data dependencies to realize the integrated computing operator. The communication processing operation includes multiple data copy operations, and at least one of the multiple computation operations is executed in parallel with at least one data copy operation.

12. The data processing method according to claim 11, characterized in that, The parallel processor is any one of a plurality of processors included in a multiprocessor system, the plurality of processors being configured to jointly execute a processing task, and each processor being configured to execute a portion of the processing task. The processing task includes executing the general computing fusion operator, and the computational processing operation includes a portion of the processing task in the general computing fusion operator that is executed by the parallel processor.

13. A parallel processor, characterized in that, Includes multiple computing units, The parallel processor is configured to perform computational processing operations, wherein the computational processing operations include multiple computational operations executed sequentially, the input data of the computational processing operations is divided into multiple data blocks, and each computational operation is used to process one of the multiple data blocks. When the parallel processor executes each computational operation, it includes performing the following operations: Based on the number of multiple working blocks included in the input data of the target kernel function corresponding to the computational operation and the number of multiple computational units, at least one competing working block is determined among the multiple working blocks, wherein at least one wave corresponding to the target kernel function includes a competing wave, and the working block in the competing wave is the at least one competing working block; The computational tasks of the multiple work blocks are processed in a pipelined parallel manner by the multiple computing units to complete the computational operation; Specifically, for the at least one competing work block, a competition mechanism is used to dynamically determine the allocation relationship between the multiple computing units and the computing tasks of the at least one competing work block. The competition mechanism includes the multiple computing units competing for the qualification to execute the computing tasks of the at least one competing work block, and the computing unit that obtains the qualification to execute the computing tasks of the corresponding competing work block.

14. An electronic device, characterized in that, Includes the parallel processor as described in claim 13.

15. An electronic device, characterized in that, include: Memory stores computer-executable instructions non-transiently; The processor is configured to run computer-executable instructions. The computer-executable instructions are executed by the processor to implement the data processing method according to any one of claims 1-12.

16. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by a processor, they implement the data processing method according to any one of claims 1-12.

Images