Task scheduling method and device, training method and system, equipment and medium

Abstract

The embodiment of the invention provides a task scheduling method and device, a training method and system, equipment and a medium. The task scheduling method is applied to a training system comprising a plurality of multi-chip acceleration cards, each multi-chip acceleration card comprises a plurality of chips integrated in a single package, and the method comprises the following steps: dividing a to-be-trained model into a plurality of stages, a first assembly line and a second assembly line which have opposite data flow directions and are used for processing different training batches are constructed based on the multiple stages, and each stage comprises at least one layer of the to-be-trained model; the first assembly line and the second assembly line are mapped to the core particles of the multiple multi-core-particle acceleration cards, so that each core particle in the at least one multi-core-particle acceleration card bears two stages, and the two stages are used for executing the corresponding training tasks of the first assembly line and the second assembly line respectively. According to the method, the physical overlapping of the forward propagation task and the back propagation task is effectively realized, and the training efficiency and the expansion capability of a large-scale model on a device cluster are remarkably improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of artificial intelligence, and in particular to a task scheduling method and device, a training method and system, equipment and a medium. BACKGROUND

[0002] With the development of large-scale artificial intelligence models, the model parameter quantity and the computing complexity continue to rise, which poses severe challenges to the training system in terms of computing power, memory and communication. The training process not only needs massive high-quality data and high-performance hardware cluster support, but also faces problems such as slow convergence speed, low resource utilization, and high training cost. Therefore, under the limited computing resources and time budget, how to effectively improve the training efficiency of large-scale models has become one of the technical problems to be solved. SUMMARY

[0003] At least one embodiment of the present disclosure provides a task scheduling method applied to a training system including a plurality of multi-core particle accelerator cards, the multi-core particle accelerator card including a plurality of core particles integrated in a single package, wherein the method comprises: dividing a to-be-trained model into a plurality of stages, and constructing a first pipeline and a second pipeline based on the plurality of stages respectively, the first pipeline and the second pipeline having opposite data flow directions and being used for processing different training batches, wherein each stage includes at least one layer of the to-be-trained model; mapping the first pipeline and the second pipeline to the core particles of the plurality of multi-core particle accelerator cards, so that each core particle in at least one multi-core particle accelerator card carries two stages, the two stages being used for executing corresponding training tasks of the first pipeline and the second pipeline respectively.

[0004] In the task scheduling method provided by at least one embodiment of the present disclosure, the to-be-trained model is divided into a plurality of stages, and a first pipeline and a second pipeline having opposite data flow directions and being used for processing different training batches are constructed based on the plurality of stages, comprising: dividing the to-be-trained model into N consecutive stages, and constructing the first pipeline and the second pipeline based on the N consecutive stages, wherein N is a positive integer, the first pipeline includes the N consecutive stages and schedules data in a first direction between execution units, the second pipeline includes the N consecutive stages and schedules data in a second direction opposite to the first direction between execution units, and the execution units include core particles.

[0005] In the task scheduling method provided by at least one embodiment of the present disclosure, the first pipeline and the second pipeline are mapped to the core particles of the plurality of multi-core particle accelerator cards, comprising: for at least one multi-core particle accelerator card in the plurality of multi-core particle accelerator cards, performing the following operation: sequentially assigning stages to each core particle of the multi-core particle accelerator card according to the stage arrangement order corresponding to the first pipeline and the second pipeline.

[0006] In the task scheduling method provided in at least one embodiment of the present disclosure, the stages are sequentially assigned to each corelet of the multi-corelet acceleration card according to the stage arrangement order corresponding to the first pipeline and the second pipeline, including: for the first pipeline, stages are sequentially assigned to each corelet of the multi-corelet acceleration card according to the forward stage arrangement order; and for the second pipeline, stages are sequentially assigned to each corelet of the multi-corelet acceleration card according to the reverse stage arrangement order.

[0007] In the task scheduling method provided in at least one embodiment of the present disclosure, the mapping of the first pipeline and the second pipeline to the corelets of the plurality of multi-corelet acceleration cards further includes: in response to all corelets of a multi-corelet acceleration card currently assigned with stages being assigned with stages, assigning stages to corelets in a next multi-corelet acceleration card.

[0008] In the task scheduling method provided in at least one embodiment of the present disclosure, the plurality of multi-corelet acceleration cards includes M multi-corelet acceleration cards, each of which includes L corelets, , and 0≤i i and S N-i-1 corresponding to the second pipeline are respectively assigned to D j,k , where , , S i represents the i+1th stage in the N consecutive stages, S N-i-1 represents the N-i th stage in the N consecutive stages, D j,k represents the k+1th corelet in the j+1th multi-corelet acceleration card, represents the floor function, and % represents the remainder.

[0009] In the task scheduling method provided in at least one embodiment of the present disclosure, the training task includes a forward propagation calculation task and a backward propagation calculation task, and the method further includes: during at least part of the training process, for at least one multi-corelet acceleration card in the plurality of multi-corelet acceleration cards, performing the following operations: determining that the forward propagation calculation task is performed by a first corelet in the multi-corelet acceleration card and the backward propagation calculation task is performed by a second corelet in the multi-corelet acceleration card; or determining that the backward propagation calculation task is performed by the first corelet in the multi-corelet acceleration card and the forward propagation calculation task is performed by the second corelet in the multi-corelet acceleration card, so that the forward propagation calculation task and the backward propagation calculation task performed by the first corelet and the second corelet respectively at least partially overlap in time.

[0010] The task scheduling method provided in at least one embodiment of the present disclosure further includes: for at least one multi-core particle acceleration card in the plurality of multi-core particle acceleration cards, performing the following operation: determining that an activation value obtained by a first core particle in the multi-core particle acceleration card performing a forward propagation calculation task is transmitted to a second core particle in the multi-core particle acceleration card through an inter-core particle interconnection link, wherein a stage corresponding to a forward propagation calculation task of the second core particle is arranged in a forward order after a stage corresponding to a forward propagation calculation task of the first core particle.

[0011] In the task scheduling method provided in at least one embodiment of the present disclosure, the plurality of multi-core particle acceleration cards perform data transmission through direct links, and the plurality of core particles perform data transmission through inter-core particle interconnection links.

[0012] In the task scheduling method provided in at least one embodiment of the present disclosure, the multi-core particle acceleration card is a dual-core particle acceleration card, and the dual-core particle acceleration card includes two core particles integrated in a single package.

[0013] At least one embodiment of the present disclosure provides a training method applied to a training system including a plurality of multi-core particle acceleration cards, wherein each core particle of at least one multi-core particle acceleration card in the plurality of multi-core particle acceleration cards performs the following operation: performing corresponding training tasks of a first pipeline and a second pipeline according to two stages carried by the core particle to train a to-be-trained model, wherein the to-be-trained model is divided into a plurality of stages, each stage includes at least one layer of the to-be-trained model, the first pipeline and the second pipeline are mapped to the core particles of the plurality of multi-core particle acceleration cards, the first pipeline and the second pipeline are constructed based on the plurality of stages, and the first pipeline and the second pipeline have opposite data flow directions and are used to process different training batches.

[0014] In the training method provided in at least one embodiment of the present disclosure, performing corresponding training tasks of a first pipeline and a second pipeline according to two stages carried by the core particle includes: during at least part of the training process, when a first core particle in the multi-core particle acceleration card performs a forward propagation calculation task, a second core particle in the multi-core particle acceleration card performs a backward propagation calculation task; or during at least part of the training process, when a first core particle in the multi-core particle acceleration card performs a backward propagation calculation task, a second core particle in the multi-core particle acceleration card performs a forward propagation calculation task, so that the forward propagation calculation task and the backward propagation calculation task performed by the first core particle and the second core particle respectively at least partially overlap in time.

[0015] The training method provided in at least one embodiment of the present disclosure further includes: for at least one multi-core particle accelerator card in the plurality of multi-core particle accelerator cards, a first core particle in the multi-core particle accelerator card transmits an activation value obtained by performing a forward propagation calculation task to a second core particle in the multi-core particle accelerator card for storage through an inter-core interconnection link, wherein a stage corresponding to a forward propagation calculation task of the second core particle is arranged in a forward order after a stage corresponding to a forward propagation calculation task of the first core particle.

[0016] In the training method provided in at least one embodiment of the present disclosure, data transmission is performed between the plurality of multi-core particle accelerator cards through a direct link, and data transmission is performed between the plurality of core particles through an inter-core interconnection link.

[0017] The present disclosure provides a task scheduling device, applied to a training system including a plurality of multi-core particle accelerator cards, wherein each multi-core particle accelerator card includes a plurality of core particles integrated in a single package. The device includes: a division module configured to divide a to-be-trained model into a plurality of stages, and construct a first pipeline and a second pipeline based on the plurality of stages, respectively, wherein each stage includes at least one layer of the to-be-trained model, and the first pipeline and the second pipeline have opposite data flow directions and are used to process different training batches; and an allocation module configured to map the first pipeline and the second pipeline to core particles of the plurality of multi-core particle accelerator cards, so that each core particle in at least one multi-core particle accelerator card carries two stages, and the two stages are used to execute corresponding training tasks of the first pipeline and the second pipeline, respectively.

[0018] The present disclosure provides a training system including a plurality of multi-core particle accelerator cards, wherein each multi-core particle accelerator card includes a plurality of core particles integrated in a single package. Each core particle of at least one multi-core particle accelerator card in the plurality of multi-core particle accelerator cards is configured to execute corresponding training tasks of a first pipeline and a second pipeline, respectively, according to two stages carried by the core particle, to train a to-be-trained model, wherein the to-be-trained model is divided into a plurality of stages, each stage includes at least one layer of the to-be-trained model, the first pipeline and the second pipeline are mapped to core particles of the plurality of multi-core particle accelerator cards, the first pipeline and the second pipeline are constructed based on the plurality of stages, and the first pipeline and the second pipeline have opposite data flow directions and are used to process different training batches.

[0019] At least one embodiment of this disclosure provides an electronic device, including: at least one processor; at least one memory including one or more computer program modules; wherein the one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the method provided by at least one embodiment of this disclosure.

[0020] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by at least one processor, perform the method provided in at least one embodiment of this disclosure.

[0021] The task scheduling method, task scheduling device, training method, training system, electronic device, and storage medium provided in at least one embodiment of this disclosure align the logical dual-pipeline with the physical multi-core architecture, utilizing the inherent multi-core physical structure of multi-core hardware to support the bidirectional data flow structure of the dual-pipeline. Since each core has independent computing resources and execution capabilities, multiple cores in a multi-core accelerator card can execute different operators in parallel. This allows two cores in the same multi-core accelerator card to execute forward propagation and backward propagation tasks simultaneously, effectively transforming the "logical overlap" of forward and backward propagation tasks into "physical parallelism," significantly improving the training efficiency and scalability of large-scale models on device clusters. Compared to time-sharing multiplexing of single-core accelerator cards, the theoretical computational capacity for model training achieved using the above strategy is doubled, and the operator switching overhead within single-core accelerator cards is effectively avoided. This solution is suitable for steady-state training scenarios with a sufficient number of micro-batches (typically ≥ 2 × the number of pipeline stages). Task scheduling can be used to achieve overlap of pipeline stages in physical space and deep overlap of computation and D2D communication in the time dimension, thereby maximizing hardware resource utilization and hiding communication latency. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure, and are not intended to limit this disclosure.

[0023] Figure 1A This is a schematic diagram of a pipelined parallel training process.

[0024] Figure 1B This is a schematic diagram of a dual-path pipeline parallel training process.

[0025] Figure 1C This is a schematic structural diagram of a single-chip general-purpose graphics processor provided for at least one embodiment of the present disclosure.

[0026] Figure 1D A schematic structural diagram of a dual-core particle general-purpose graphics processor is provided for at least one embodiment of the present disclosure.

[0027] Figure 2 A schematic flow diagram of a task scheduling method is provided for at least one embodiment of the present disclosure.

[0028] Figure 3 A schematic flow diagram of a training method is provided for at least one embodiment of the present disclosure.

[0029] Figure 4A A schematic diagram of a training system is provided for at least one embodiment of the present disclosure.

[0030] Figure 4B A schematic diagram of a task scheduling method is provided for at least one embodiment of the present disclosure.

[0031] Figure 5 A schematic block diagram of a task scheduling apparatus is provided for at least one embodiment of the present disclosure.

[0032] Figure 6 A schematic block diagram of a training system is provided for at least one embodiment of the present disclosure.

[0033] Figure 7 A schematic block diagram of an electronic device is provided for at least one embodiment of the present disclosure.

[0034] Figure 8 A schematic block diagram of another electronic device is provided for at least one embodiment of the present disclosure.

[0035] Figure 9 A schematic block diagram of a non-transitory computer-readable storage medium is provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without any creative effort fall within the protection scope of the present disclosure.

[0037] Unless otherwise defined, technical terms or scientific terms used in the present disclosure shall have the ordinary meaning as understood by a person having ordinary skill in the art to which the present disclosure pertains. The terms "first", "second", and similar terms are used herein to distinguish one element from another, and are not necessarily used in a sequence or order to indicate importance or chronology. The terms "comprise", "comprising", "include", "including", and the like are used herein to mean including but not limited to. The terms "connected", "coupled", and the like are used herein to indicate either a direct connection, an indirect connection, or an electrical connection between two elements, and can include a wired or wireless connection. The terms "upper", "lower", "left", "right", and the like are used herein only to represent relative positions, and can change accordingly when the absolute positions of the described objects change.

[0038] The present disclosure will be described below through several specific embodiments. In order to keep the following description of the embodiments of the present disclosure clear and concise, detailed description of known functions and known elements can be omitted. When any element of the embodiments of the present disclosure appears in more than one figure, the element is denoted by the same or similar reference numeral in each figure.

[0039] Since a large-scale model (e.g., a model based on a Transformer architecture) can have up to hundreds of billions of parameters, a single machine and a single card cannot handle the training task of such a large-scale model, so distributed training is widely used in large-scale model training in artificial intelligence. Distributed training refers to the use of multiple machines working together to speed up the training process of large deep learning models. In distributed training, the original model, dataset, and training process are decomposed and distributed to multiple machines for simultaneous processing, thereby effectively utilizing more computing resources and shortening the training time. In the model training process, in order to save memory occupancy and improve computing efficiency, various parallel strategies can be used, such as data parallelism (DP), pipeline parallelism (PP), and tensor parallelism (TP).

[0040] Data parallelism refers to splitting the training dataset of a model into multiple sub-datasets, and the multiple sub-datasets after splitting are allocated to multiple computing devices, each computing device holding a complete model copy, so that it can independently train the sub-dataset allocated to it, for example, each computing device can independently complete the forward propagation and backward propagation calculations of the sub-dataset.

[0041] Tensor parallelism refers to splitting one layer (or operator) of a model and placing parts of the weights of the operator on different computing devices, thereby reducing the memory footprint on each computing device (e.g., GPU). A neural network model includes multiple layers, and each layer can be understood as a function that can perform a specific mathematical operation on an input tensor. In the tensor parallelism mode, the tensor of one layer is split into multiple parts, and the multiple part tensors after splitting are respectively allocated to multiple computing devices, each of which is only responsible for a part of the computation of the model and exchanges necessary intermediate results through communication.

[0042] Pipeline parallelism is one of the important techniques for training large-scale models. In the pipeline parallelism mode, the network layers of a model are sequentially divided into multiple stages, each stage containing a number of consecutive layers and being allocated to different computing devices, thereby forming a computing pipeline across devices. For example, in the forward propagation process, each computing device passes the activation value to the computing device where the next pipeline stage is located, and in the backward propagation process, each computing device passes the gradient back to the computing device where the previous pipeline stage is located.

[0043] To improve the pipeline throughput, a global training batch is usually divided into multiple micro-batches. Each micro-batch flows into the pipeline in turn and is pushed forward between different stages, forming a “pipeline” execution mode. As the number of micro-batches increases, the pipeline tends to be “full”, and the pipeline bubble (i.e., the idle time of the computing device due to waiting for data) is compressed, thereby significantly improving the computing efficiency. Through this mechanism, multiple computing devices can concurrently process data of different micro-batches in the same training cycle, realizing computing overlap in the time dimension, effectively improving the utilization of hardware resources and the overall training efficiency.

[0044] Figure 1A A schematic diagram of a pipeline parallel training process.

[0045] Figure 1A The pipeline parallel training process shown adopts a One Forward pass followed by One Backward pass (1F1B) scheduling strategy. The core idea of the 1F1B strategy is that in each pipeline stage, after completing the forward propagation calculation of one micro-batch in turn, the corresponding backward propagation calculation is immediately started, without waiting for all micro-batches to complete the forward propagation, thereby improving the training efficiency and throughput. Under this scheduling mechanism, after receiving the output of the previous stage, the computing device of each stage immediately starts processing and produces output, which is then passed to the computing device of the next stage.

[0046] For example, such as Figure 1A As shown, the multiple layers of the model to be trained are divided into four stages (stages A to D, not shown in the figure), which are assigned to computing devices A through D for computation. Each stage includes, for example, eight micro-batches of data. Fi in the row corresponding to computing device A represents the forward propagation of the i-th micro-batch of data in stage A, Bx in the row corresponding to computing device D represents the backward propagation of the x-th micro-batch of data in stage D, and so on. During forward computation, the computation of the next stage depends on the computation of the previous stage. For example, the forward propagation F1 of the first micro-batch of data on computing device B (corresponding to stage B) can only begin after the F1 computation on computing device A (corresponding to stage A) is completed. Similarly, during backward computation, the computation of the previous stage depends on the computation of the next stage. For example, the backward propagation B1 of the first micro-batch of data on computing device C (corresponding to stage C) can only begin after the B1 computation on computing device D (corresponding to stage D) is completed.

[0047] Due to this dependency, each computing device will be idle while waiting for input data or gradients. For example, in the early stages of training, only computing device A starts processing F1, while computing devices B, C, and D are all waiting. This process results in a large number of wasted computation cycles, forming significant pipeline bubbles (e.g., ...). Figure 1A (As shown in the blank box in the middle), this phenomenon is particularly severe when there are many stages or few micro-batches, which restricts the utilization of hardware resources and training throughput.

[0048] To reduce pipeline bubbles, a dual-pipeline parallel approach can be employed, logically constructing two pipelines with opposite data flows but sharing hardware resources. During training, these two pipelines run concurrently, processing different micro-batches of data and each performing its own complete forward and backward propagation. Thanks to this design, while the first pipeline is processing the forward computation of one micro-batch, the second pipeline can simultaneously process the forward computation of another micro-batch. Once a micro-batch completes its entire forward stage in its respective pipeline, its backward propagation can begin immediately without waiting for all forward tasks in other pipelines to complete. Because the two pipelines advance alternately, forward and backward propagation achieve closer overlap on the timeline, significantly reducing pipeline bubbles.

[0049] Figure 1B This is a schematic diagram of a dual-path pipeline parallel training process.

[0050] like Figure 1BAs shown, the model to be trained is divided into 8 consecutive stages (Stage 0~Stage 7), each stage including one or more consecutive model layers. When performing training, any one mini-batch of data is sequentially forward-propagated from Stage 0 to Stage 7; its corresponding backward propagation starts from Stage 7 and propagates the gradient back to Stage 0. For example, assuming the model to be trained includes 40 model layers (Layer 0~Layer 39), each stage can include five consecutive model layers. The correspondence between stages and model layers can be referred to the following example: Stage 0 includes Layer 0~Layer 4; Stage 1 includes Layer 5~Layer 9; Stage 2 includes Layer 10~Layer 14; and so on.

[0051] As shown, the model to be trained is divided into 8 consecutive stages (Stage 0~Stage 7), each stage including one or more consecutive model layers. When performing training, any one mini-batch of data is sequentially forward-propagated from Stage 0 to Stage 7; its corresponding backward propagation starts from Stage 7 and propagates the gradient back to Stage 0. For example, assuming the model to be trained includes 40 model layers (Layer 0~Layer 39), each stage can include five consecutive model layers. The correspondence between stages and model layers can be referred to the following example: Stage 0 includes Layer 0~Layer 4; Stage 1 includes Layer 5~Layer 9; Stage 2 includes Layer 10~Layer 14; and so on. Figure 1B As shown, the 8 stages of the model to be trained can be logically organized into two independent and data flow opposite pipelines (first pipeline Stream A and second pipeline Stream B), each pipeline including the complete 8 stages Stage 0~Stage 7, and the corresponding stages in the two pipelines are paired and allocated to the same computing device. The arrows in the figure respectively show the forward propagation direction of Stream A and Stream B. Figure 1B 8 computing devices (Card0~Card 7) are shown. Specifically, Stage i of the first pipeline and Stage 7-i of the second pipeline can be allocated to computing device Card i (where 0≤i<8, i is a non-negative integer).

[0052] For example, stage 0 of the first pipeline and stage 7 of the second pipeline are allocated to Card 0, stage 1 of the first pipeline and stage 6 of the second pipeline are allocated to Card 1, and so on. During training, the first pipeline and the second pipeline run simultaneously, each processing a different mini-batch of data and each performing a full forward pass and backward pass. When the first pipeline performs a forward pass on a first mini-batch, it computes in forward order by stage from Stage 0 to Stage 7, corresponding to sequentially through Cards 0 to 7 on the physical device; the second pipeline can simultaneously perform a forward pass on a second mini-batch, computing in forward order by stage from Stage 0 to Stage 7, but corresponding to sequentially through Cards 7 to 0 on the physical device. Similarly, when the first pipeline performs a backward pass on the first mini-batch, it computes in backward order by stage from Stage 7 to Stage 0, corresponding to sequentially through Cards 7 to 0 on the physical device; when the second pipeline performs a backward pass on the second mini-batch, it computes in backward order by stage from Stage 7 to Stage 0, corresponding to sequentially through Cards 0 to 7 on the physical device. In this way, when training reaches a steady state, logical overlap of forward and backward passes in the two pipelines can be achieved.

[0053] The training system can include a plurality of computing nodes, each of which can include one or more hosts, each of which can be connected to one or more computing devices, and the hosts and computing devices in the computing nodes can cooperatively complete computing tasks. The number of computing nodes included in the training system, the number of hosts and computing devices included in each computing node are not limited by the embodiments of the present disclosure. Within the computing node, data transmission can be achieved between the host and the computing device in various ways. The computing device (also referred to as a "slave device") can be directly connected to the host through a hardware interface (for example, connected through a peripheral component interconnect express (PCI Express, PCIe) bus) to achieve low-latency data transmission; or it can work with the host in a software-defined manner (such as virtualization technology).

[0054] For example, the host described above can include a Central Processing Unit (CPU), and the computing device can include a Graphics Processing Unit (GPU), a General-purpose Graphics Processing Unit (GPGPU), a Tensor Processing Unit (TPU), a Deep learning Processing Unit (DPU), an Accelerated Processing Unit (APU), a Neural network Processing Unit (NPU), an Application-Specific Integrated Circuit (ASIC), or a Field-Programmable Gate Array (FPGA), etc.

[0055] Figure 1C A schematic structural diagram of a single-core general-purpose graphics processing unit is provided for at least one embodiment of the present disclosure.

[0056] As shown in Figure 1C , the general-purpose graphics processing unit is actually an array of programmable multi-processors, for example, the programmable multi-processors can be Streaming Processor Clusters (SPCs), for example, including Figure 1C Streaming Processor Cluster 1,..., Streaming Processor Cluster M as shown in . M is a positive integer greater than 1. In the general-purpose graphics processing unit, one Streaming Processor Cluster processes a computing task, or multiple Streaming Processor Clusters process a computing task. Multiple Streaming Processor Clusters share data through a global cache or a global memory. The Streaming Processor Cluster can also be referred to as a computing core.

[0057] As shown in Figure 1C , taking Streaming Processor Cluster 1 as an example, one Streaming Processor Cluster includes multiple computing units, for example Figure 1C Computing Unit 1, Computing Unit 2,..., Computing Unit N in . N is a positive integer. Each computing unit (Compute Unit, abbreviated as CU) is used to perform arithmetic logic operations, such as accumulation, reduction, regular addition, subtraction, multiplication, division, etc. A computing unit includes multiple cores, and each core includes an arithmetic logic unit (ALU), a floating point computing unit, etc. The core is used to perform specific computing tasks. In addition, the computing unit also includes registers (for example Figure 1CThe register file and shared memory in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared memory in a computing unit is used to share data between the cores of that computing unit.

[0058] like Figure 1C As shown, each streaming processor cluster also provides a buffer for caching data from the N computing units within the streaming processor cluster.

[0059] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 1C (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0060] In each computing unit, the thread beam scheduling / distribution module ( Figure 1C (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple cores of the computing unit can run thread bundles. Depending on the number of cores in the computing unit, multiple thread bundles in a thread block can be executed concurrently or in a time-sharing manner. Multiple threads in each thread bundle will execute the same instructions. Memory execution instructions are issued to shared memory in the computing unit or further issued to intermediate-level cache, global cache, or global memory (e.g., High Bandwidth Memory, HBM) for read and write operations, etc.

[0061] In the fields of artificial intelligence and high-performance computing, accelerator cards based on multi-die architecture (also known as multi-grid or multi-die accelerator cards) are gradually becoming widely used. Multi-die technology integrates two or more dies (also known as chips or dies) with independent computing capabilities onto the same substrate or interposer through advanced packaging (such as 2.5D / 3D packaging or silicon interposers), enabling them to work together efficiently and present a single, logically unified, high-performance chip. When the package contains two dies, this architecture can also be called a dual-die architecture.

[0062] The multi-chip accelerator card (also known as a multi-chip device) provided in at least one embodiment of this disclosure may include a multi-chip graphics processor, a multi-chip general-purpose graphics processor, a multi-chip tensor processor, a multi-chip deep learning processor, a multi-chip accelerator processor, a multi-chip neural network processor, etc., and the embodiments of this disclosure do not limit this.

[0063] The above text Figure 1C The general-purpose graphics processor shown is a single-die architecture, meaning that all streaming processor clusters (including compute units and caches), global cache, and high-bandwidth memory are integrated onto a single die. The following uses... Figure 1C Based on the single-core general-purpose graphics processor shown, an example of a dual-core general-purpose graphics processor is introduced.

[0064] Figure 1D This is a schematic structural diagram of a dual-die general-purpose graphics processor provided for at least one embodiment of the present disclosure. The dual-die general-purpose graphics processor packages two independent GPGPU dies into a single board, presenting itself externally as a single logically unified high-performance chip.

[0065] For example, such as Figure 1D As shown, a dual-core general-purpose graphics processor may include a first core, a second core, and global memory. Each of the first and second cores may include a global cache and multiple streaming processor clusters, and the streaming processor cluster structure within each core may be similar to the aforementioned... Figure 1C The single-core embodiment shown has the same structure, where each streaming processor cluster includes multiple computing units, and each computing unit includes multiple cores, a register file, and shared memory. The global cache within the core can support data caching for all computing units within that core. The first core and the second core can be coupled via a high-speed interconnect structure. Global memory can be shared by the first and second cores, supporting concurrent access to global storage resources by both cores. Through the above dual-core architecture, the scale of computing resources can be expanded without exceeding the limitations of a single photomask size, thereby improving the computing throughput and energy efficiency of general-purpose graphics processors. Of course, the above dual-core general-purpose graphics processor is merely an example, and this disclosure does not limit its scope.

[0066] An operator is an executable unit representing a specific computational task (such as a convolution operator, matrix multiplication operator, or communication operator). Each operator ultimately needs to be mapped to a specific circuit on a chip for execution. Since each chip has independent computing resources and execution capabilities, in a multi-chip accelerator card, multiple chips can execute different operators in parallel; however, in a single-chip accelerator card, all operators must be executed serially or in time-division multiplexing on the same chip, making cross-chip operator-level parallelism impossible.

[0067] for Figure 1BThe shown two-path pipeline parallel training process is performed by single-die devices. Since there is a lack of physically independent computing resources in a single-die device, the forward propagation task in the first pipeline and the backward propagation task in the second pipeline cannot be truly parallel executed at the same time. In other words, on a single-die accelerator card, only one operator can be executed at any time, and the so-called "parallel" is achieved by time-sharing multiplexing, which can only achieve the overlap of communication and calculation (i.e., logical overlap), but cannot achieve the true simultaneous execution of two computationally intensive tasks (such as forward propagation and backward propagation) (i.e., physical parallelism). For example, in a single-die device Card 3, in the Stream A direction, all computing cores are used to perform forward propagation calculation of Stage 3; at this time, since the computing resources have been completely occupied, the backward propagation calculation of Stage 4 in the Stream B direction cannot be performed at the same time. Therefore, only logical parallelism can be achieved inside a single-die device, and physical parallelism cannot be achieved. Figure 1B

[0068] In addition, some pipeline schedulers usually regard all computing devices in a cluster (in a training system) as topologically equivalent nodes when performing stage allocation, and adopt linear or random mapping strategies. For example, Stage 0 is allocated to GPU0, Stage 1 is allocated to GPU 1, and so on. However, this mapping strategy ignores the existence of die-to-die (D2D) links in multi-die devices.

[0069] The two-path pipeline parallel training process introduced above has extremely frequent "forward activation value transmission" and "backward gradient return" in the steady state stage. If logically adjacent stages (for example, two consecutive stages) are deployed on two accelerator cards across nodes instead of being preferentially deployed on two dies in the same accelerator card, high-frequency communication will flow through the low-bandwidth cross-node network, causing serious communication congestion; while the high-bandwidth D2D link may be in an idle state, causing resource waste.

[0070] At least one embodiment of the present disclosure provides a task scheduling method, a task scheduling device, a training method, a training system, an electronic device, and a non-transitory computer-readable storage medium.

[0071] ​The task scheduling method provided in at least one embodiment of the present disclosure is applied to a training system including multiple multi-core particle accelerator cards, and each multi-core particle accelerator card includes multiple core particles integrated in a single package. The method includes: dividing a model to be trained into multiple stages, and constructing a first pipeline and a second pipeline based on the multiple stages respectively, the first pipeline and the second pipeline being opposite in data flow and used for processing different training batches, wherein each stage includes at least one layer of the model to be trained; and mapping the first pipeline and the second pipeline to the core particles of the multiple multi-core particle accelerator cards, so that each core particle in at least one multi-core particle accelerator card carries two stages, which are respectively used for corresponding training tasks of the first pipeline and the second pipeline.

[0072] The task scheduling method provided in at least one embodiment of the present disclosure aligns the logical double-pipeline with the physical multi-core particle architecture, and uses the multi-core particle hardware to naturally carry the bidirectional data flow structure of the double-pipeline. Since each core particle has independent computing resources and execution capabilities, multiple core particles in a multi-core particle accelerator card can execute different operators in parallel, so that two core particles in the same multi-core particle accelerator card can execute the forward propagation task and the backward propagation task at the same time, effectively realizing the conversion of the forward propagation task and the backward propagation task from "logical overlap" to "physical parallelism", and significantly improving the training efficiency and expansion capability of large-scale models on a device cluster.

[0073] Compared with time-sharing multiplexing of a single-core particle accelerator card, the theoretical calculation capability upper limit of the model training realized by using the above task scheduling method is doubled, and the operator switching overhead in the single-core particle accelerator card is effectively avoided.

[0074] The task scheduling method provided in at least one embodiment of the present disclosure is especially suitable for a steady-state training scenario with sufficient micro-batches (usually the number of micro-batches ≥ 2 × the number of pipeline stages). The task scheduling can realize the overlap of pipeline stages in the physical space and the deep overlap of calculation and D2D communication in the time dimension, thereby maximizing the hardware resource utilization and hiding the communication delay.

[0075] The task scheduling method provided in at least one embodiment of the present disclosure can be applied to a training system including multiple multi-core particle accelerator cards, and each multi-core particle accelerator card includes multiple core particles integrated in a single package, which can be referred to in the above description. An example of the multi-core particle accelerator card is a double-core particle accelerator card including two core particles integrated in a single package.

[0076] The multiple multi-core particle accelerator cards in the training system perform data transmission through direct links, and the multiple core particles in a single multi-core particle accelerator card perform data transmission through Die-to-Die (D2D) links.

[0077] For example, the accelerator card and the accelerator card can be connected by a point-to-point connection through a communication medium, thereby realizing direct connection. For example, the communication medium can include a direct attach copper (DAC); for another example, the communication medium can also include an active electrical cable (AEC). The disclosure does not specifically limit the type of communication medium.

[0078] On the hardware architecture, each multi-core accelerator card can integrate at least one direct link controller and a corresponding physical port supporting a high-speed direct link protocol, and each physical port can include a plurality of high-speed differential signal lines for connecting external communication media. When two multi-core accelerator cards are docked through communication media with their respective physical ports, a high-speed direct link is formed between the two, and the link constitutes a high-speed serial point-to-point physical channel.

[0079] For example, data can be transmitted in the form of data blocks on the link, and cross-card memory access can be achieved through an address mapping mechanism provided by the driver. For example, the accelerator cards interconnected through the high-speed direct link can have a shared memory pool; after one accelerator card writes data to a shared address, another accelerator card can directly read the data through a mapped address in a local process without the need for a host relay or explicit copying.

[0080] Further, the multi-core accelerator card can be configured to support load / store semantics. For example, a remote accelerator card pre-allocates a storage space and maps it to a uniform virtual address visible to a local device thread through a driver; a local accelerator card thread can then directly read and write the memory of the remote accelerator card through standard load or store instructions, realizing low-latency and high-bandwidth collaborative computing.

[0081] In the above communication process, the direct memory access (DMA) engine in the multi-core accelerator card can work with the direct link controller: when batch data transfer between cards needs to be performed, the DMA engine serves as the initiation and scheduling unit of data transmission, and generates memory read / write requests; these requests are routed to the direct link controller through the on-chip interconnection, and the direct link controller encapsulates the requests into data packets conforming to the high-speed direct link protocol, and sends them to the destination accelerator card through the high-speed direct link. Therefore, the DMA engine is located in the transmission logic layer and is responsible for task scheduling and data flow control, while the direct link controller and the high-speed direct link constitute the underlying hardware channel and provide high-bandwidth and low-latency physical connection capabilities.

[0082] Multiple cores in the same multi-core particle acceleration card can be interconnected through D2D links in a D2D bus architecture, and a bus protocol runs on the links to support multi-core concurrent access. The D2D link refers to a direct high-speed interconnection between multiple cores located in the same package through an intra-package interconnection structure. For example, through advanced packaging interconnection technologies such as Through-Silicon Vias (TSV), Redistribution Layer (RDL), or microbump, a short-distance, high-density electrical connection channel is physically constructed to support high-bandwidth, low-latency, and low-power data communication between cores.

[0083] The task scheduling method provided by at least one embodiment of the present disclosure can be executed by a scheduler, which can be the training framework itself or a component of the training framework. The training framework is used to build, train, and optimize machine learning or deep learning models, and can provide a high-level abstract interface to simplify model definition, data loading, loss calculation, gradient update, distributed training, device management, and other processes. The training framework can have a built-in task scheduling mechanism (such as data parallelism, model parallelism, pipeline scheduling, etc.), and can also support custom scheduling strategies through plugins or extensions to efficiently utilize computing resources and accelerate the training process.

[0084] Figure 2 An exemplary flowchart of a task scheduling method provided by at least one embodiment of the present disclosure.

[0085] For example, as shown in Figure 2 The task scheduling method provided by at least one embodiment of the present disclosure includes steps S101-S102.

[0086] Step S101: dividing a to-be-trained model into multiple stages, and constructing a first pipeline and a second pipeline based on the multiple stages, the first pipeline and the second pipeline having opposite data flow directions and being used for processing different training batches, wherein each stage includes at least one layer of the to-be-trained model.

[0087] Step S102: mapping the first pipeline and the second pipeline to cores of multiple multi-core particle acceleration cards, so that each core in at least one multi-core particle acceleration card carries two stages, the two stages being used for executing corresponding training tasks of the first pipeline and the second pipeline, respectively.

[0088] The to-be-trained model described above can be a machine learning model to be trained, such as a deep learning model. The deep learning model can be a neural network structure including multiple layers (3 layers, 4 layers, 8 layers, or more layers), such as a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), and the like. The structure of these neural networks usually includes an input layer, a hidden layer, and an output layer, the hidden layer refers to those layers in the neural network between the input layer and the output layer, and the hidden layer is also referred to as a processing layer. The input layer is used to receive data to be processed, such as an image to be processed, and the output layer is used to output a processing result, such as a processed image. The processing layer can include a convolutional layer, a pooling layer, a batch normalization layer, a fully connected layer, and the like. Depending on the structure of the neural network, the processing layer can include different contents and combinations. In some examples, the to-be-trained model can be an artificial intelligence model (also referred to as an artificial intelligence large-scale model, simply referred to as a “large-scale model”) with a large number of parameters constructed by an artificial intelligence network. The large-scale model can be a content generation model based on a prompt word to generate content, such as a large language model (LLM), a visual large-scale model, or a multi-modal large-scale model, and the like. For example, the to-be-trained model can be a model based on a Transformer architecture, a model constructed based on a recurrent neural network, a model constructed based on an attention mechanism, and the like. The to-be-trained model can also be a hybrid expert model including multiple expert sub-networks and a gating mechanism, only part of the experts are activated for each input sample. Such an architecture can significantly reduce the actual computational overhead while maintaining high model capacity, and is therefore widely used in large-scale model scenarios. The specific type, structure, or implementation of the to-be-trained model is not limited in the embodiments of the present disclosure.

[0089] For example, in step S101, model splitting can be achieved by uniform division or non-uniform division: uniform division refers to approximately equally allocating the to-be-trained model to each stage according to the number of layers or the amount of calculation, so as to balance the calculation load of each computing device; non-uniform division differentiates the division according to the actual calculation complexity, parameter size, activation value size, or communication overhead of each layer, so as to optimize the overall pipeline throughput or reduce pipeline bubbles. The division result can be explicitly configured and specified by the user, or can be dynamically generated by the training framework based on automatic analysis of the model structure and cluster topology information, so that the constructed pipeline maximizes hardware resource utilization and training efficiency while meeting parallel strategy constraints.

[0090] For example, the training framework can scan the cluster hardware to obtain cluster topology information. The training framework automatically detects the number, model, memory capacity, and intra-node interconnection topology of the accelerators available on each computing node by calling the underlying device driver interface; at the same time, in combination with the cluster management environment, it identifies all nodes participating in the training and their network connection capabilities (such as whether they support Remote Direct Memory Access (RDMA) or the like). Based on the above information, the training framework can dynamically build a physical topology graph containing the attributes of computing nodes, accelerator cards, and links, which accurately reflects the communication paths between hardware, as well as the number of computing nodes and accelerator cards in the cluster, and serves as the basis for subsequent determination of parallel strategies and task scheduling, thereby achieving efficient and adaptive distributed training without the need for manual configuration.

[0091] On the basis of completing the cluster hardware scanning and building the physical topology graph, the training framework can further combine the user-specified parallel strategy configuration (such as data parallelism, model parallelism, pipeline parallelism, and partitioning method, etc.) to split the entire model to be trained. Specifically, the training framework can divide the model into a plurality of continuous or non-continuous stages according to the number of stages defined in the user configuration, and the range of layers or operators contained in each stage.

[0092] A global training batch is a subset of input samples. In order to achieve finer-grained computation scheduling in pipeline parallelism, a global training batch can be divided into a plurality of micro-batches. Each micro-batch contains a part of the input samples in the global batch, and can be independently performed forward propagation and backward propagation.

[0093] In step S101, the model to be trained can be divided into N stages (Stage 0~Stage N-1), and based on the N stages, two pipelines for processing different micro-batches and having opposite data flow directions are logically constructed: a first pipeline and a second pipeline. Each pipeline contains all stages from Stage 0 to Stage N-1, but the scheduling direction of the micro-batches is opposite.

[0094] It should be noted that the "first pipeline" and the "second pipeline" are exemplary terms used to distinguish different pipelines, and do not refer to a specific pipeline, nor do they indicate a specific order, hierarchical relationship, or quantity limitation between the pipelines.

[0095] Through the construction of the above pipeline, in the training execution process, any micro-batch will be processed in order in the pipeline to which it belongs: the forward propagation starts from Stage 0 and is passed to Stage N-1 stage by stage; and the corresponding backward propagation starts from Stage N-1 and returns the gradient to Stage 0 stage by stage. The first pipeline and the second pipeline constructed in step S101 are respectively used to process different micro-batches. For example, for a global training batch containing 20 micro-batches (MB 0~MB 19), the first pipeline can process MB 0~MB 9, and the second pipeline can process MB 10~MB 19, so as to realize bidirectional and staggered pipeline execution on the same hardware resources.

[0096] In step S102, the first pipeline and the second pipeline constructed in step S101 are mapped to the core grains of the plurality of multi-core grain acceleration cards, so that each core grain in at least one multi-core grain acceleration card carries two stages for executing corresponding training tasks of the first pipeline and the second pipeline. For example, the training tasks can include forward propagation calculation tasks, backward propagation calculation tasks, gradient synchronization tasks, parameter update tasks, etc. After the mapping is completed, each core grain in at least one multi-core grain acceleration card carries a stage belonging to the first pipeline and a stage belonging to the second pipeline, that is, each core grain in at least one multi-core grain acceleration card needs to execute the training tasks of the first pipeline and the training tasks of the second pipeline in the training process.

[0097] Through the above mapping, the first pipeline and the second pipeline share the same set of hardware resources. Since each core grain has independent computing resources and execution capabilities, in the multi-core grain acceleration card, multiple core grains can execute different operators in parallel, so that two core grains in the same multi-core grain acceleration card can execute forward propagation tasks and backward propagation tasks at the same time, effectively realizing the physical overlap of forward propagation tasks and backward propagation tasks, and significantly improving the training efficiency and expansion capability of large-scale models on a device cluster.

[0098] An example of step S101 can include: dividing the model to be trained into N consecutive stages, and constructing the first pipeline and the second pipeline based on the N consecutive stages, where N is a positive integer, the first pipeline includes N consecutive stages and schedules data between execution units in a first direction, and the second pipeline includes N consecutive stages and schedules data between execution units in a second direction opposite to the first direction, and the execution units described above include core grains.

[0099] In the above example, the model to be trained is divided into N consecutive stages (Stage 0~Stage N-1), and both the first pipeline and the second pipeline constructed based on the division include the complete N consecutive stages Stage 0~Stage N-1.

[0100] For example, if there are N execution units (EU 0~EU N-1) corresponding to the N consecutive stages, the first pipeline schedules data among the execution units in the direction from EU 0 to EU N-1 when propagating in the forward direction, and in the direction from EU N-1 to EU 0 when propagating in the reverse direction. The second pipeline schedules data among the execution units in the direction from EU N-1 to EU 0 when propagating in the forward direction, and in the direction from EU 0 to EU N-1 when propagating in the reverse direction.

[0101] It should be noted that the above "first direction" and "second direction" are exemplary terms used to distinguish different data scheduling directions, and do not refer to a fixed direction, nor represent a specific order, hierarchical relationship, or quantity limitation between the directions.

[0102] In the task scheduling method provided in at least one embodiment of the present disclosure, the execution unit can be a corelet in a multi-corelet acceleration card. For example, the training framework can uniformly logically number all corelets participating in training in the training system, so as to facilitate task scheduling and data flow control.

[0103] One example of step S102 can include steps S1021~S1022.

[0104] Step S1021: For at least one multi-corelet acceleration card in the plurality of multi-corelet acceleration cards, perform the following operation: sequentially assign stages to each corelet of the multi-corelet acceleration card according to the stage arrangement order corresponding to the first pipeline and the second pipeline.

[0105] Step S1022: In response to all corelets of the multi-corelet acceleration card currently assigned with stages being assigned with stages, assign stages to corelets in the next multi-corelet acceleration card.

[0106] In step S1021, since the data flow directions of the first pipeline and the second pipeline are opposite, when assigning stages to each corelet, the stages are assigned in different orders for the first pipeline and the second pipeline.

[0107] One example of "sequentially assigning stages to each corelet of the multi-corelet acceleration card according to the stage arrangement order corresponding to the first pipeline and the second pipeline" in step S1021 can include steps S1021a~S1021b.

[0108] Step S1021a: For the first pipeline, stages are assigned to each die of the multi-die accelerator card in the forward order of stages.

[0109] Step S1021b: For the second pipeline, stages are assigned to each die of the multi-die accelerator card in the reverse order of stages.

[0110] For example, for the consecutive stages Stage 0~Stage N-1 obtained by dividing the model to be trained, the forward order of stages is from Stage 0 to Stage N-1, and the reverse order of stages is from Stage N-1 to Stage 0.

[0111] Suppose that there are M multi-die accelerator cards (Card 0~Card M-1) in the training system, and each multi-die accelerator card includes L dies, then the L dies in Card j (0≤j<M) can be marked as Die j_0~Die j_L-1.

[0112] Take Card 0 in the M multi-die accelerator cards as an example to start assigning stages. For the first pipeline, stages need to be assigned to Die 0_0~Die 0_L-1 in the forward order of stages, that is, Stage 0 in the first pipeline is assigned to Die 0_0, Stage 1 is assigned to Die 0_1, Stage 2 is assigned to Die 0_2, …, and Stage L-1 is assigned to Die 0_L-1. For the second pipeline, stages need to be assigned to Die 0_0~Die 0_L-1 in the reverse order of stages, that is, Stage N-1 in the first pipeline is assigned to Die 0_0, Stage N-2 is assigned to Die 0_1, Stage N-3 is assigned to Die 0_2, …, and Stage N-L is assigned to Die 0_L-1.

[0113] Each die participating in the execution of the training task is assigned two stages, corresponding to the first pipeline and the second pipeline respectively. For example, in the example above, Die 0_0 is assigned Stage 0 of the first pipeline and Stage N-1 of the second pipeline.

[0114] In step S1022, if all the corelets of the multi-corelet acceleration card that is currently being assigned stages (also referred to as the current multi-corelet acceleration card) have been assigned stages, then the corelets in the next multi-corelet acceleration card can be assigned stages. Here, the "next multi-corelet acceleration card" refers to the next available multi-corelet acceleration card following the current multi-corelet acceleration card in a predefined multi-corelet acceleration card traversal order. The traversal order can be determined based on factors such as a physical topology order, a logical ID order assigned by a training framework, an order specified in a user profile, etc.

[0115] In some examples, in response to all the corelets of the multi-corelet acceleration card that is currently being assigned stages having been assigned stages of the first pipeline, then the corelets in the next multi-corelet acceleration card can continue to be assigned stages of the first pipeline. In response to all the corelets of the multi-corelet acceleration card that is currently being assigned stages having been assigned stages of the second pipeline, then the corelets in the next multi-corelet acceleration card can continue to be assigned stages of the second pipeline. In other examples, in response to all the corelets of the multi-corelet acceleration card that is currently being assigned stages having been assigned stages of the first pipeline and the second pipeline, then the corelets in the next multi-corelet acceleration card can continue to be assigned stages of the first pipeline and the second pipeline.

[0116] Corresponding to the above example, that is, when Die 0_0~Die 0_L-1 in Card 0 have all been assigned stages, then the corelets Die 1_0~Die 1_L-1 in Card 1 can continue to be assigned stages. For example, for the first pipeline, Stage 0~Stage L-1 have been assigned to the corelets of Card 0, and currently Die 1_0~Die 1_L-1 need to be assigned stages in the forward stage order, that is, Stage L of the first pipeline is assigned to Die 1_0, Stage L+1 is assigned to Die 1_1, Stage L+2 is assigned to Die 1_2, …, and Stage 2L-1 is assigned to Die 1_L-1. For the second pipeline, Stage N-1~Stage N-L have been assigned to the corelets of Card 0, and currently Die 1_0~Die 1_L-1 need to be assigned stages in the reverse stage order, that is, Stage N-L-1 of the first pipeline is assigned to Die 1_0, Stage N-L-2 is assigned to Die 1_1, Stage N-L-3 is assigned to Die 1_2, …, and Stage N-2L is assigned to Die 1_L-1.

[0117] And so on, the stages can continue to be allocated to the dies in Card 2, Card 3, …, Card M-1 in a similar manner until the N stages of the first pipeline and the N stages of the second pipeline have been allocated to the respective dies of multiple multi-die acceleration cards, thereby achieving the mapping of the first pipeline and the second pipeline to the respective dies of the multi-die acceleration cards.

[0118] An example of step S102 may include: allocating S corresponding to the first pipeline i and S corresponding to the second pipeline N-i-1 to D j,k where , , where S i represents the (i + 1)-th stage in N consecutive stages, S N-i-1 represents the (N - i)-th stage in N consecutive stages, D j,k represents the (k + 1)-th die in the (j + 1)-th multi-die acceleration card, represents floor division, and % represents modulo operation.

[0119] In this example, the training system includes M multi-die acceleration cards, and each multi-die acceleration card includes L dies, , and 0 ≤ i < N, 0 ≤ j < M, 0 ≤ k < L, where i, j, and k are non-negative integers.

[0120] In this example, the N stages are the first stage to the N-th stage, corresponding to S0 to S N-1 , that is, Stage 0 to Stage N-1 as mentioned above. The M multi-die acceleration cards are the first multi-die acceleration card to the M-th multi-die acceleration card, corresponding to Card 0 to Card M-1 as mentioned above. The L dies in a single multi-die acceleration card are the first die to the L-th die. For example, the first die D 1,0 in the second multi-die acceleration card corresponds to Die 1_0 as mentioned above.

[0121] According to the above formula, when N = 8, M = 4, and L = 2, the following mapping relationship can be obtained: when i = 0, j = 0, k = 0, N - i - 1 = 7, that is, allocating the first stage S0 corresponding to the first pipeline and the eighth stage S7 corresponding to the second pipeline to the first die D 0,0 in the first multi-die acceleration card; when i = 1, j = 0, k = 1, N - i - 1 = 6, that is, allocating the second stage S1 corresponding to the first pipeline and the seventh stage S6 corresponding to the second pipeline to the second die D 0,1When i = 2, j = 1, k = 0, and N-i-1 = 5, that is, the third stage S2 corresponding to the first pipeline and the sixth stage S5 corresponding to the second pipeline are allocated to the first core D in the second multi-core acceleration card 1,0 When i = 7, j = 3, k = 1, and N-i-1 = 0, that is, the eighth stage S7 corresponding to the first pipeline and the first stage S0 corresponding to the second pipeline are allocated to the second core D in the fourth multi-core acceleration card 3,1 .

[0122] Through the above allocation mode, all core resources in a single multi-core acceleration card can be fully utilized, and logically adjacent stages are preferentially deployed on the cores of the same multi-core acceleration card, so that high-frequency communication for transmitting activation values and gradients in the process of forward propagation and backward propagation can be preferentially performed through the D2D link, idle high-bandwidth D2D links are avoided, and the core resource utilization rate is effectively improved.

[0123] The task scheduling method provided in at least one embodiment of the present disclosure can further include: during at least part of the training process, for at least one of the plurality of multi-core acceleration cards, determining that a forward propagation calculation task is performed by a first core in the multi-core acceleration card and a backward propagation calculation task is performed by a second core in the multi-core acceleration card, so that the forward propagation calculation task and the backward propagation calculation task performed by the first core and the second core, respectively, at least partially overlap in time.

[0124] For example, during at least part of the training process (for example, a training steady state stage), for the same multi-core acceleration card, the training framework can determine that a forward propagation calculation task is performed by a first core and a backward propagation calculation task is performed by a second core, and correspondingly sends a task scheduling instruction to the second core to make the second core perform the backward propagation calculation task. In some examples, the above forward propagation task and backward propagation task can belong to a first pipeline and a second pipeline, respectively.

[0125] The task scheduling method provided in at least one embodiment of the present disclosure can further include: during at least part of the training process, for at least one of the plurality of multi-core acceleration cards, determining that a forward propagation calculation task is performed by a first core in the multi-core acceleration card and a backward propagation calculation task is performed by a second core in the multi-core acceleration card, so that the forward propagation calculation task and the backward propagation calculation task performed by the first core and the second core, respectively, at least partially overlap in time.

[0126] For example, during at least part of the training process (e.g., a steady state phase of training), for the same multi-die accelerator card, the training framework can determine that, while a first die performs a backward propagation computation task, a second die performs a forward propagation computation task, and send corresponding task scheduling instructions to the second die to cause the second die to perform the forward propagation computation task. In some examples, the forward propagation task and the backward propagation task described above can belong to a first pipeline and a second pipeline, respectively.

[0127] It should be noted that "first die", "second die" here and below are exemplary terms used to distinguish different dies in the same multi-die accelerator card, and do not refer to a specific die, nor indicate a specific order, hierarchical relationship or quantity limitation between the dies.

[0128] Through the above task scheduling method, the two different dies in the same multi-die accelerator card simultaneously perform the forward propagation computation task and the backward propagation computation task, respectively, during at least part of the training process, thereby realizing the true parallelism in physics of the forward propagation computation task and the backward propagation computation task within the same multi-die accelerator card. Compared with time-sharing multiplexing of a single-die accelerator card, the theoretical calculation capacity upper limit of model training realized by the above task scheduling method is doubled, and the operator switching overhead within the single-die accelerator card is effectively avoided.

[0129] Taking Die 1_0 and Die 1_1 in Card 1 as an example, when Die 1_0 performs the forward propagation computation task, it can be determined that Die 1_1 performs the backward propagation computation task. When Die 1_1 performs the forward propagation computation task, it can be determined that Die 1_0 performs the backward propagation computation task.

[0130] The task scheduling method provided by at least one embodiment of the present disclosure can further include: for at least one multi-die accelerator card of a plurality of multi-die accelerator cards, determining that an activation value obtained by a first die of the multi-die accelerator card performing a forward propagation computation task is transmitted to a second die of the multi-die accelerator card through an inter-die interconnection link, wherein a stage corresponding to the forward propagation computation task of the second die is after a stage corresponding to the forward propagation computation task of the first die according to a stage forward arrangement order.

[0131] In the backward propagation algorithm, in order to calculate the gradient, it is necessary to access the activation value generated during the forward propagation in the backward propagation process.

[0132] In the above example, the training framework can control the data transmission and storage between the cores, so that for a certain multi-core accelerator card, in the forward propagation phase, the first core can directly transmit and store the calculated activation values to the second core through the high-bandwidth D2D link, and in the subsequent back propagation phase, when the second core needs to perform gradient calculation of the corresponding stage, the activation values can be directly read from the local without the need to recalculate or load from the external storage space, thereby significantly reducing the data loading delay and improving the training efficiency.

[0133] In the task scheduling method provided by at least one embodiment of the present disclosure, the scheduling of tasks follows the principle of first arrival first execution, and the execution order can be determined and fixed in advance in the training process arrangement stage. Through the preset scheduling strategy, it can effectively avoid scheduling the calculation tasks from different pipelines (i.e. the first pipeline and the second pipeline) to the same core at the same time, thereby preventing resource competition, ensuring execution certainty, and maintaining stable throughput of the pipeline.

[0134] Corresponding to the task scheduling method provided by at least one embodiment of the present disclosure, at least one embodiment of the present disclosure also provides a training method. The training method can be applied to a training system including a plurality of multi-core accelerator cards, and specific reference can be made to the description above.

[0135] Figure 3 A schematic flowchart of a training method provided by at least one embodiment of the present disclosure.

[0136] For example, as shown in Figure 3 The training method provided by at least one embodiment of the present disclosure includes the following steps S201 performed by each core of at least one multi-core accelerator card in a plurality of multi-core accelerator cards.

[0137] Step S201: According to the two stages carried by the core, the corresponding training tasks of the first pipeline and the second pipeline are respectively executed to train the to-be-trained model.

[0138] In step S201, the to-be-trained model is divided into a plurality of stages, each stage including at least one layer of the to-be-trained model, the first pipeline and the second pipeline are mapped to the cores of the plurality of multi-core accelerator cards, the first pipeline and the second pipeline are constructed based on the plurality of stages, and the first pipeline and the second pipeline have opposite data flow directions and are used to process different training batches.

[0139] In the training method, each corelet has independent computing resources and execution capabilities. Therefore, in the multi-corelet acceleration card, multiple corelets can perform different operators in parallel, so that two corelets in the same multi-corelet acceleration card can perform forward propagation tasks and backward propagation tasks at the same time, effectively realizing the physical overlap of forward propagation tasks and backward propagation tasks, and significantly improving the training efficiency and expansion capability of large-scale models on a device cluster.

[0140] An example of step S201 can include: during at least part of the training process, when a first corelet in the multi-corelet acceleration card performs a forward propagation calculation task, a second corelet in the multi-corelet acceleration card performs a backward propagation calculation task, so that the forward propagation calculation task and the backward propagation calculation task performed by the first corelet and the second corelet respectively at least partially overlap in time. For example, the forward propagation task and the backward propagation task can belong to the first pipeline and the second pipeline, respectively.

[0141] Another example of step S201 can include: during at least part of the training process, when a first corelet in the multi-corelet acceleration card performs a backward propagation calculation task, a second corelet in the multi-corelet acceleration card performs a forward propagation calculation task, so that the forward propagation calculation task and the backward propagation calculation task performed by the first corelet and the second corelet respectively at least partially overlap in time. For example, the forward propagation task and the backward propagation task can belong to the first pipeline and the second pipeline, respectively.

[0142] Through the training method, during at least part of the training process, two different corelets in the same multi-corelet acceleration card simultaneously perform forward propagation calculation tasks and backward propagation calculation tasks, respectively, thereby realizing the true parallelism of forward propagation calculation tasks and backward propagation calculation tasks within the same multi-corelet acceleration card. Compared with the time-sharing multiplexing of a single-corelet acceleration card, the theoretical calculation capacity of the training method is doubled, and the operator switching overhead within the single-corelet acceleration card is effectively avoided.

[0143] The training method provided by at least one embodiment of the present disclosure can further include: for at least one multi-corelet acceleration card in the plurality of multi-corelet acceleration cards, the first corelet in the multi-corelet acceleration card transmits the activation value obtained by performing the forward propagation calculation task to the second corelet in the multi-corelet acceleration card through the inter-corelet interconnection link, wherein the stage corresponding to the forward propagation calculation task of the second corelet is arranged in a forward order after the stage corresponding to the forward propagation calculation task of the first corelet.

[0144] Through the training method, for a certain multi-core particle acceleration card, in the forward propagation stage, the first core particle can directly transmit and store the calculated activation value to the second core particle through the high-bandwidth D2D link, and in the subsequent back propagation stage, when the second core particle needs to perform gradient calculation of the corresponding stage, the activation value can be directly read from the local without the need to recalculate or load from the external storage space, thereby significantly reducing the data loading delay and improving the training efficiency.

[0145] For other related content of the training method, please refer to the related description in the task scheduling method above, which will not be repeated here.

[0146] It should be further pointed out that in various embodiments of the present disclosure, the execution order of each step of the task scheduling method and the training method is not limited, although the execution process of each step is described in a specific order above, which does not constitute a limitation on the embodiments of the present disclosure. Each step in the task scheduling method and the training method can be executed in series or in parallel, which can be determined according to actual needs.

[0147] For example, compared with the above description, the task scheduling method and the training method provided by at least one embodiment of the present disclosure can further include more or fewer steps, and the embodiments of the present disclosure are not limited thereto.

[0148] Figure 4A A schematic diagram of a training system provided by at least one embodiment of the present disclosure.

[0149] For example, as shown in Figure 4A , the training system includes four double-core particle acceleration cards (Card 0~Card 3), each of which includes two core particles integrated in a single package.

[0150] The training framework can scan the cluster hardware to construct a physical topology map of the training system, define each double-core particle acceleration card as a "computing pair", and number the core particles in each double-core particle acceleration card. For example, Card 0 includes core particles Die 0_0 and Die 0_1, Card 1 includes core particles Die 1_0 and Die 1_1, Card 2 includes core particles Die 2_0 and Die 2_1, and Card 3 includes core particles Die 3_0 and Die 3_1.

[0151] Figure 4B A schematic diagram of a task scheduling method provided by at least one embodiment of the present disclosure. Figure 4B The task scheduling method of Figure 4A the training system shown.

[0152] As Figure 4A and Figure 4BAs shown, the training framework divides the model to be trained into 8 stages, and based on the 8 stages, respectively constructs a first stream (Stream A) and a second stream (Stream B) with opposite data flow directions and for processing different training batches, and maps Stream A and Stream B to the core dies of 4 double-core accelerator cards. The mapping strategy is: Stage 0~Stage 7 of Stream A are sequentially mapped to the core dies of Card 0~Card 3, and Stage 0~Stage 7 of Stream B are sequentially mapped to the core dies of Card 3~Card 0. After mapping, Stage 0 of Stream A and Stage 7 of Stream B are allocated to Die 0_0, Stage 1 of Stream A and Stage 6 of Stream B are allocated to Die 0_1, Stage 2 of Stream A and Stage 5 of Stream B are allocated to Die 1_0, Stage 3 of Stream A and Stage 4 of Stream B are allocated to Die 1_1, Stage 4 of Stream A and Stage 3 of Stream B are allocated to Die 2_0, Stage 5 of Stream A and Stage 2 of Stream B are allocated to Die 2_1, Stage 6 of Stream A and Stage 1 of Stream B are allocated to Die 3_0, and Stage 7 of Stream A and Stage 0 of Stream B are allocated to Die 3_1.

[0153] For a global training batch containing 20 micro-batches (MB 0~MB 19), Stream A can be used to process MB 0~MB 9, and Stream B is used to process MB 10~MB 19. In the initial stage of training, Die 0_0 can start processing the forward propagation calculation of MB 0, and Die 3_1 can simultaneously start processing the forward propagation calculation of MB 10, and the other core dies are in a waiting state at this time.

[0154] When Die 0_0 completes the forward propagation calculation of MB 0 in Stage 0, Die 0_1 can begin executing the forward propagation calculation of MB 0 in Stage 1, and Die 0_0 will simultaneously begin executing the forward propagation calculation of MB 1 in Stage 0. On the other hand, when Die 3_1 completes the forward propagation calculation of MB 10 in Stage 0, Die 3_0 can begin executing the forward propagation calculation of MB 10 in Stage 1, and Die 3_1 will simultaneously begin executing the forward propagation calculation of MB 11 in Stage 0. This continues, with all cores beginning their tasks. When Die 3_1 completes the forward propagation calculation of MB 0 in Stage 7, it will continue executing the backward propagation calculation of MB 0 in Stage 7, and so on, until Die 0_0 completes the backward propagation calculation of MB 0 in Stage 0. At this point, both the forward and backward propagation calculations of MB 0 are complete. During this process, training gradually enters a steady-state phase.

[0155] In the steady-state phase, taking Die 1_0 and Die 1_1 in Card 1 as examples, Die 1_0 receives the activation value data sent by Die 0_1 and performs the forward propagation calculation in Stage 2 of Stream A. Simultaneously, Die 1_1 receives the gradient data sent by Die 1_0 and performs the backpropagation calculation in Stage 4 of Stream B. At this time, the two particles in Card 1 simultaneously perform both forward and backpropagation calculation tasks, achieving a physical overlap between the forward and backpropagation calculations.

[0156] like Figure 4B As shown, the dual-core accelerator cards communicate with each other via a direct link, while the two cores within the card communicate with each other via a D2D link. Therefore, taking Die 1_0 as an example, during forward propagation, Die 1_0 can directly transmit and store the calculated activation values ​​to Die 1_1 via a high-bandwidth D2D link. In the subsequent backpropagation stage, when Die 1_1 needs to perform Stage 3 gradient calculations, it can directly read the activation values ​​locally without recalculating or loading them from external storage, thus significantly reducing data loading latency and improving training efficiency.

[0157] It should be noted that, Figure 4A and Figure 4B This is merely an example; the embodiments disclosed herein do not limit the number of stages, the number of multi-core accelerator cards in the training system, or the number of cores in the multi-core accelerator cards.

[0158] Figure 5A schematic block diagram of a task scheduling apparatus is provided for at least one embodiment of the present disclosure. The task scheduling apparatus can be applied to a training system comprising a plurality of multi-chiplet acceleration cards, each of which comprises a plurality of chiplets integrated in a single package. For example, the task scheduling apparatus can be disposed in a training framework.

[0159] For example, as shown in FIG. 5, Figure 5 The task scheduling apparatus 500 comprises a division module 501 and an allocation module 502.

[0160] For example, the division module 501 is configured to divide a model to be trained into a plurality of stages, and construct a first pipeline and a second pipeline based on the plurality of stages respectively, wherein each stage comprises at least one layer of the model to be trained, and the first pipeline and the second pipeline are opposite in data flow and used for processing different training batches.

[0161] For example, the allocation module 502 is configured to map the first pipeline and the second pipeline to chiplets of a plurality of multi-chiplet acceleration cards, so that each chiplet of at least one multi-chiplet acceleration card carries two stages, and the two stages are used for executing corresponding training tasks of the first pipeline and the second pipeline respectively.

[0162] In some examples, the division module is further configured to divide the model to be trained into N consecutive stages, and construct the first pipeline and the second pipeline based on the N consecutive stages respectively, wherein N is a positive integer, the first pipeline comprises the N consecutive stages and schedules data between execution units in a first direction, and the second pipeline comprises the N consecutive stages and schedules data between execution units in a second direction opposite to the first direction, and the execution units comprise chiplets.

[0163] In some examples, the allocation module is further configured to, for at least one multi-chiplet acceleration card of the plurality of multi-chiplet acceleration cards, perform the following operations: sequentially allocate stages to each chiplet of the multi-chiplet acceleration card according to a stage arrangement order corresponding to stages of the first pipeline and the second pipeline.

[0164] In some examples, the allocation module is further configured to, for the first pipeline, sequentially allocate stages to each chiplet of the multi-chiplet acceleration card according to a forward stage arrangement order, and for the second pipeline, sequentially allocate stages to each chiplet of the multi-chiplet acceleration card according to a reverse stage arrangement order.

[0165] In some examples, the allocation module is further configured to, in response to all chiplets of a multi-chiplet acceleration card currently allocated with stages being allocated with stages, allocate stages to chiplets in a next multi-chiplet acceleration card.

[0166] In some examples, the plurality of multi-chiplet acceleration cards comprises M multi-chiplet acceleration cards, and each multi-chiplet acceleration card comprises L chiplets, , and 0≤i i and the S N-i-1 corresponding to the second pipeline are allocated to D j,k , where , , where S i represents the i+1th stage in the N continuous stages, S N-i-1 represents the N-i th stage in the N continuous stages, D j,k represents the k+1th corelet in the j+1th multi-corelet acceleration card, represents the floor, and % represents the remainder.

[0167] In some examples, the training task includes a forward propagation computing task and a backward propagation computing task, and the task scheduling apparatus further includes a determination module. The determination module is configured to, during at least part of the training process, for at least one of the plurality of multi-corelet acceleration cards, determine that the backward propagation computing task is performed by a second corelet in the multi-corelet acceleration card when the forward propagation computing task is performed by a first corelet in the multi-corelet acceleration card, or determine that the forward propagation computing task is performed by the second corelet in the multi-corelet acceleration card when the backward propagation computing task is performed by the first corelet in the multi-corelet acceleration card, so that the forward propagation computing task and the backward propagation computing task performed by the first corelet and the second corelet, respectively, at least partially overlap in time.

[0168] In some examples, the determination module is further configured to, for at least one of the plurality of multi-corelet acceleration cards, determine that an activation value obtained by the forward propagation computing task performed by a first corelet in the multi-corelet acceleration card is transmitted to a second corelet in the multi-corelet acceleration card through an inter-corelet interconnection link, where the stage corresponding to the forward propagation computing task of the second corelet is arranged in a forward order after the stage corresponding to the forward propagation computing task of the first corelet.

[0169] In some examples, data transmission is performed between the plurality of multi-corelet acceleration cards through a direct link, and data transmission is performed between the plurality of corelets through an inter-corelet interconnection link.

[0170] In some examples, the multi-corelet acceleration card is a dual-corelet acceleration card, and the dual-corelet acceleration card includes two corelets integrated in a single package.

[0171] It should be noted that the various modules described above can be implemented by software, hardware, firmware, or any combination thereof, for example, the allocation module can be implemented as an allocation circuit, and the division module can be implemented as a division circuit, and the specific implementation of the embodiments of the present disclosure is not limited thereto.

[0172] It should be understood that the task scheduling apparatus provided by at least one embodiment of the present disclosure can be used to implement the foregoing task scheduling method, and can also achieve similar technical effects to the foregoing task scheduling method, which will not be described here.

[0173] It should be noted that, in the embodiments of the present disclosure, the task scheduling apparatus can include more or fewer modules or units, and the connection relationship between the various modules or units is not limited and can be determined according to actual needs. The specific construction manner of each module or unit is not limited and can be constituted by an analog device according to the circuit principle, or can be constituted by a digital chip, or constituted in other applicable manners.

[0174] Figure 6 A schematic block diagram of a training system provided by at least one embodiment of the present disclosure.

[0175] For example, as shown in Figure 6 The training system 600 provided by at least one embodiment of the present disclosure includes a plurality of multi-core particle acceleration cards 601, and each multi-core particle acceleration card 601 includes a plurality of core particles 602 integrated in a single package. Each core particle of at least one multi-core particle acceleration card in the plurality of multi-core particle acceleration cards is configured to perform a corresponding training task of a first pipeline and a second pipeline, respectively, according to two stages carried by the core particle, to train a to-be-trained model.

[0176] Wherein, the to-be-trained model is divided into a plurality of stages, each stage including at least one layer of the to-be-trained model, the first pipeline and the second pipeline are mapped to the core particles of the plurality of multi-core particle acceleration cards, the first pipeline and the second pipeline are constructed based on the plurality of stages, and the first pipeline and the second pipeline have opposite data flow directions and are used to process different training batches.

[0177] In some examples, during at least part of the training process, when a first core particle in the multi-core particle acceleration card performs a forward propagation computation task, a second core particle in the multi-core particle acceleration card is configured to perform a backward propagation computation task, so that the forward propagation computation task and the backward propagation computation task performed by the first core particle and the second core particle, respectively, at least partially overlap in time, wherein the forward propagation task and the backward propagation task belong to the first pipeline and the second pipeline, respectively.

[0178] In some examples, during at least part of the training process, when a first core particle in the multi-core particle acceleration card performs a backward propagation computation task, a second core particle in the multi-core particle acceleration card is configured to perform a forward propagation computation task, so that the forward propagation computation task and the backward propagation computation task performed by the first core particle and the second core particle, respectively, at least partially overlap in time, wherein the forward propagation task and the backward propagation task belong to the first pipeline and the second pipeline, respectively.

[0179] In some examples, for at least one of the plurality of multi-chiplet acceleration cards, a first chiplet in the multi-chiplet acceleration card is configured to transmit, via an inter-chiplet link, an activation value resulting from performing a forward propagation computation task to a second chiplet in the multi-chiplet acceleration card for storage, wherein a stage corresponding to the forward propagation computation task of the second chiplet is arranged in a forward order after a stage corresponding to the forward propagation computation task of the first chiplet.

[0180] Other relevant information about the training system can be referred to the relevant description in the task scheduling method above, which will not be repeated here.

[0181] Figure 7 A schematic block diagram of an electronic device provided by at least one embodiment of the present disclosure.

[0182] For example, as shown in Figure 7 The electronic device 700 includes at least one processor 701 and at least one memory 702. The at least one memory 702 includes one or more computer program modules. The one or more computer program modules are stored in the memory 702 and configured to be executed by the at least one processor 701, and the one or more computer program modules include instructions for executing the task scheduling method or the training method described above, which, when executed by the at least one processor 701, can perform one or more steps of the task scheduling method or the training method provided by at least one embodiment of the present disclosure. The memory 702 and the processor 701 can be interconnected by a bus system and / or other forms of connection mechanism (not shown).

[0183] For example, the processor 701 can be a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), an artificial intelligence (AI) accelerator, or other forms of processing units with data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA), etc. For example, the central processing unit (CPU) can be of X86, ARM, RISC-V architecture, etc. The processor 701 can be a general-purpose processor or a special-purpose processor, and can control other components in the electronic device 700 to perform desired functions.

[0184] For example, memory 702 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0185] Figure 8 This is a schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0186] The electronic devices in at least one embodiment of this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0187] The electronic device includes at least one processor and a memory. The processor may be referred to as processing device 801 as described below, and the memory may include at least one of ROM 802, RAM 803, and storage device 808 as described below. The memory is used to store programs for performing the methods described in the various method embodiments above; the processor is configured to execute the programs stored in the memory. The processor may include a central processing unit (CPU) or other forms of processing unit having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0188] like Figure 8 As shown, the electronic device 800 may include a processing unit 801 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in ROM 802 or a program loaded from storage device 808 into RAM 803. RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interfaces are also connected to bus 804.

[0189] In general, the following devices can be connected to the I / O interface 805: input devices 806, including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 807, including, for example, a display, a speaker, a vibrator, and the like; storage devices 808, including, for example, a magnetic tape, a hard disk, and the like; and communication devices 809. The communication devices 809 can allow the electronic device 800 to communicate wirelessly or via a wire with other devices to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it is understood that all of the shown devices are not required to be implemented or present. More or fewer devices can alternatively be implemented or present.

[0190] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to at least one embodiment of the present disclosure. For example, at least one embodiment of the present disclosure includes a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication devices 809, or installed from the storage devices 808, or installed from the ROM 802. When the computer program is executed by the processing devices 801, the above-mentioned functions defined in the methods of at least one embodiment of the present disclosure are performed.

[0191] It should be noted that the computer readable medium in the above disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In at least one embodiment of the disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. While in at least one embodiment of the disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, an optical cable, a radio frequency (RF), etc., or any suitable combination of the above.

[0192] The computer readable medium described above can be contained in the electronic device 800 described above; or can exist separately and not be assembled into the electronic device 800.

[0193] Figure 9 A schematic block diagram of a non-transitory computer readable storage medium provided for at least one embodiment of the disclosure.

[0194] For example, as shown in Figure 9 The non-transitory computer readable storage medium 900 stores computer readable instructions 901, which, when executed by at least one processor, perform one or more steps of the task scheduling method or training method described above.

[0195] For example, the storage medium can include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM), a compact disc read only memory (CD-ROM), a flash memory, or any combination of the above storage media, or other applicable storage medium. For example, the readable storage medium can also be the memory 702 in the computer 700 in FIG. 1, and the relevant description can refer to the foregoing, which will not be repeated here. Figure 7 The memory 702 in FIG. 1 can be the readable storage medium in the computer 700 in FIG. 1, and the relevant description can refer to the foregoing, which will not be repeated here.

[0196] Although the present disclosure has been described in detail with general description and specific embodiments, some modifications or improvements can be made to the present disclosure on the basis of the embodiments of the present disclosure, which is obvious to those skilled in the art. Therefore, these modifications or improvements made on the basis of the spirit of the present disclosure are within the scope of the present disclosure.

[0197] For the present disclosure, the following points need to be explained:

[0198] (1) The drawings of the embodiments of the present disclosure only involve the structures involved in the embodiments of the present disclosure, and other structures can refer to the general design.

[0199] (2) In order to be clear, the thickness of the layer or region is enlarged or reduced in the drawings used to describe the embodiments of the present disclosure, that is, these drawings are not drawn according to the actual proportion.

[0200] (3) In the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined to obtain new embodiments.

[0201] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited to this, and the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A task scheduling method, characterized in that, The method is applied to a training system comprising multiple multi-core accelerator cards, wherein the multi-core accelerator cards include multiple cores integrated in a single package, wherein the method includes: The model to be trained is divided into multiple stages, and a first pipeline and a second pipeline with opposite data flow directions are constructed based on the multiple stages to process different training batches, wherein each stage includes at least one layer of the model to be trained. The first pipeline and the second pipeline are mapped to the chips of the plurality of multi-chip accelerator cards, such that each chip in at least one multi-chip accelerator card carries two stages, the two stages being used to execute the corresponding training tasks of the first pipeline and the second pipeline, respectively.

2. The method according to claim 1, characterized in that, The process of dividing the model to be trained into multiple stages and constructing a first pipeline and a second pipeline with opposite data flow directions for processing different training batches based on the multiple stages includes: The model to be trained is divided into N consecutive stages, and the first pipeline and the second pipeline are constructed based on the N consecutive stages respectively. Wherein, N is a positive integer, the first pipeline includes the N consecutive stages and schedules data between execution units in a first direction, the second pipeline includes the N consecutive stages and schedules data between execution units in a second direction opposite to the first direction, and the execution unit includes a core.

3. The method according to claim 1, characterized in that, The step of mapping the first pipeline and the second pipeline to the dies of the plurality of multi-die accelerator cards includes: For at least one of the plurality of multi-core accelerator cards, perform the following operations: According to the stage arrangement order corresponding to the first and second pipelines, stages are sequentially assigned to each chip of the multi-chip accelerator card.

4. The method according to claim 3, characterized in that, The step of sequentially assigning stages to each chip of the multi-chip accelerator card according to the stage arrangement order corresponding to the first pipeline and the second pipeline includes: For the first pipeline, stages are sequentially assigned to each chip of the multi-chip accelerator card according to the forward arrangement of stages; For the second production line, stages are sequentially assigned to each chip of the multi-chip accelerator card according to the reverse stage arrangement order.

5. The method according to claim 4, characterized in that, The step of mapping the first pipeline and the second pipeline to the dies of the plurality of multi-die accelerator cards further includes: In response to the current allocation phase, all chips of the multi-chip accelerator card have been allocated, which marks the next chip allocation phase in the multi-chip accelerator card.

6. The method according to claim 2, characterized in that, The multiple multi-die acceleration cards include M multi-die acceleration cards, and each multi-die acceleration card includes L dies, where 0 ≤ i < N, 0 ≤ j < M, 0 ≤ k < L, and i, j, k are non-negative integers. The step of mapping the first pipeline and the second pipeline to the chips of the plurality of multi-chip accelerator cards includes: S corresponding to the first production line i S corresponding to the second production line N-i-1 Assigned to D j,k Among them , , Among them, S i S represents the (i+1)th stage in the N consecutive stages. N-i-1 D represents the Ni-th stage in the N consecutive stages. j,k This represents the (k+1)th core in the (j+1)th multi-core accelerator card. % indicates rounding down, and % indicates taking the remainder.

7. The method according to claim 1, characterized in that, The training task includes a forward propagation computation task and a backward propagation computation task, and the method further includes: During at least part of the training process, for at least one of the plurality of multi-core accelerator cards, the following operations are performed: When the forward propagation computation task is performed by the first core in the multi-core accelerator card, the backward propagation computation task is performed by the second core in the multi-core accelerator card; or When the backpropagation computation task is performed by the first core of the multi-core accelerator card, the forward propagation computation task is performed by the second core of the multi-core accelerator card. This ensures that the forward propagation computation task and the backward propagation computation task performed by the first core and the second core, respectively, overlap at least partially in time.

8. The method according to claim 1, characterized in that, The method further includes: For at least one of the plurality of multi-core accelerator cards, perform the following operations: The activation value obtained by the first core in the multi-core accelerator card from the forward propagation computation task is transmitted to the second core in the multi-core accelerator card for storage through the inter-core interconnect link. The stage corresponding to the forward propagation computation task of the second core is arranged in a forward order after the stage corresponding to the forward propagation computation task of the first core.

9. The method according to claim 1, characterized in that, The multiple multi-core accelerator cards transmit data via direct links, and the multiple cores transmit data via inter-core interconnect links.

10. The method according to claim 1, characterized in that, The multi-core accelerator card is a dual-core accelerator card, which includes two cores integrated in a single package.

11. A training method, characterized in that, The method is applied to a training system comprising multiple multi-core accelerator cards, the multi-core accelerator cards comprising multiple cores integrated in a single package, the method comprising: The following operations are performed by each core of at least one of the plurality of multi-core accelerator cards: The corresponding training tasks of the first and second pipelines are executed according to the two stages carried by the core particle to train the model to be trained. The model to be trained is divided into multiple stages, each stage including at least one layer of the model to be trained. The first pipeline and the second pipeline are mapped to the chips of the multiple multi-chip accelerator cards. The first pipeline and the second pipeline are constructed based on the multiple stages. The data flow of the first pipeline and the second pipeline is opposite and used to process different training batches.

12. The method according to claim 11, characterized in that, The step of executing the corresponding training tasks of the first pipeline and the second pipeline according to the two stages carried by the chip includes: During at least part of the training process, when the first core in the multi-core accelerator card performs the forward propagation computation task, the second core in the multi-core accelerator card performs the backward propagation computation task; or During at least part of the training process, when the first core in the multi-core accelerator card performs the backpropagation computation task, the second core in the multi-core accelerator card performs the forward propagation computation task. This ensures that the forward propagation computation task and the backward propagation computation task performed by the first core and the second core, respectively, overlap at least partially in time.

13. The method according to claim 11, characterized in that, The method further includes: For at least one of the plurality of multi-core accelerator cards The activation value obtained by the first core in the multi-core accelerator card from the forward propagation computation task is transmitted to the second core in the multi-core accelerator card for storage through the core interconnection link. The stage corresponding to the forward propagation computation task of the second core is arranged in a forward order after the stage corresponding to the forward propagation computation task of the first core.

14. The method according to claim 11, characterized in that, The multiple multi-core accelerator cards transmit data via direct links, and the multiple cores transmit data via inter-core interconnect links.

15. A task scheduling device, characterized in that, The device is used in a training system comprising multiple multi-core accelerator cards, the multi-core accelerator cards comprising multiple cores integrated in a single package, wherein the device includes: The partitioning module is configured to divide the model to be trained into multiple stages, and to construct a first pipeline and a second pipeline with opposite data flow directions and used to process different training batches based on the multiple stages, wherein each stage includes at least one layer of the model to be trained. The allocation module is configured to map the first pipeline and the second pipeline to the chips of the plurality of multi-chip accelerator cards, such that each chip in at least one multi-chip accelerator card carries two stages, the two stages being used to execute the corresponding training tasks of the first pipeline and the second pipeline, respectively.

16. A training system, characterized in that, The system includes multiple multi-core accelerator cards, each multi-core accelerator card including multiple cores integrated in a single package, wherein each core of at least one of the multiple multi-core accelerator cards is configured as follows: The corresponding training tasks of the first and second pipelines are executed according to the two stages carried by the core particle to train the model to be trained. The model to be trained is divided into multiple stages, each stage including at least one layer of the model to be trained. The first pipeline and the second pipeline are mapped to the chips of the multiple multi-chip accelerator cards. The first pipeline and the second pipeline are constructed based on the multiple stages. The data flow of the first pipeline and the second pipeline is opposite and used to process different training batches.

17. An electronic device, characterized in that, The electronic device includes: At least one processor; At least one memory, including one or more computer program modules; The one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the method according to any one of claims 1-14.

18. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-readable instructions, wherein the computer-readable instructions, when executed by at least one processor, perform the method according to any one of claims 1-14.

Images