Task scheduling method and device, training method and system, equipment and medium

Abstract

Embodiments of this disclosure provide a task scheduling method and apparatus, a training method and system, a device, and a medium. The task scheduling method is applied to a training system comprising multiple multi-core accelerator cards, each multi-core accelerator card including multiple cores integrated in a single package. The method includes: dividing the model to be trained into multiple stages, and constructing a first pipeline and a second pipeline with opposite data flows for processing different training batches based on the multiple stages, each stage including at least one layer of the model to be trained; mapping the first pipeline and the second pipeline to the cores of the multiple multi-core accelerator cards, such that each core in at least one multi-core accelerator card carries two stages, the two stages being used to execute corresponding training tasks of the first pipeline and the second pipeline, respectively. This method effectively achieves physical overlap between forward propagation and backward propagation tasks, significantly improving the training efficiency and scalability of large-scale models on device clusters.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of artificial intelligence, specifically to a task scheduling method and apparatus, a training method and system, a device, and a medium. Background Technology

[0002] With the development of large-scale artificial intelligence models, the number of model parameters and computational complexity continue to rise, posing severe challenges to training systems in terms of computing power, memory, and communication. The training process not only requires massive amounts of high-quality data and high-performance hardware clusters, but also faces problems such as slow convergence speed, low resource utilization, and high training costs. Therefore, how to effectively improve the training efficiency of large-scale models under limited computing resources and time budgets has become one of the most pressing technical challenges to be solved. Summary of the Invention

[0003] This disclosure provides at least one embodiment of a task scheduling method applied to a training system including multiple multi-chip accelerator cards, wherein the multi-chip accelerator cards include multiple chips integrated in a single package, wherein the method includes: dividing a model to be trained into multiple stages, and constructing a first pipeline and a second pipeline with opposite data flow directions and used to process different training batches based on the multiple stages, wherein each stage includes at least one layer of the model to be trained; mapping the first pipeline and the second pipeline to the chips of the multiple multi-chip accelerator cards, such that each chip in at least one multi-chip accelerator card carries two stages, wherein the two stages are respectively used to execute the corresponding training tasks of the first pipeline and the second pipeline.

[0004] In a task scheduling method provided in at least one embodiment of this disclosure, the model to be trained is divided into multiple stages, and a first pipeline and a second pipeline with opposite data flow directions and used to process different training batches are constructed based on the multiple stages, including: dividing the model to be trained into N consecutive stages, constructing the first pipeline and the second pipeline based on the N consecutive stages, wherein N is a positive integer, the first pipeline includes the N consecutive stages and schedules data between execution units in a first direction, the second pipeline includes the N consecutive stages and schedules data between execution units in a second direction opposite to the first direction, and the execution unit includes a core.

[0005] In a task scheduling method provided in at least one embodiment of this disclosure, mapping the first pipeline and the second pipeline to the chips of the plurality of multi-chip accelerator cards includes: for at least one of the plurality of multi-chip accelerator cards, performing the following operation: sequentially allocating stages to each chip of the multi-chip accelerator card according to the stage arrangement order corresponding to the first pipeline and the second pipeline.

[0006] In the task scheduling method provided by at least one embodiment of the present disclosure, arranging the stages corresponding to the first pipeline and the second pipeline in sequence and sequentially allocating stages to each die of the multi-die acceleration card includes: for the first pipeline, sequentially allocating stages to each die of the multi-die acceleration card in the forward stage arrangement order; for the second pipeline, sequentially allocating stages to each die of the multi-die acceleration card in the reverse stage arrangement order.

[0007] In the task scheduling method provided by at least one embodiment of the present disclosure, mapping the first pipeline and the second pipeline to the dies of the multiple multi-die acceleration cards further includes: in response to all the dies of the multi-die acceleration card currently allocated with stages having been allocated stages, allocating stages to the dies in the next multi-die acceleration card.

[0008] In the task scheduling method provided by at least one embodiment of the present disclosure, the multiple multi-die acceleration cards include M multi-die acceleration cards, and each multi-die acceleration card includes L dies. , and 0 ≤ i < N, 0 ≤ j < M, 0 ≤ k < L, where i, j, and k are non-negative integers. Among them, mapping the first pipeline and the second pipeline to the dies of the multiple multi-die acceleration cards includes: allocating the S i corresponding to the first pipeline and the S N-i-1 corresponding to the second pipeline to D j,k , where , , where S i represents the (i + 1)-th stage among the N consecutive stages, S N-i-1 represents the N - i-th stage among the N consecutive stages, D j,k represents the (k + 1)-th die in the (j + 1)-th multi-die acceleration card. represents rounding down, and % represents taking the remainder.

[0009] In the task scheduling method provided by at least one embodiment of the present disclosure, the training task includes a forward propagation calculation task and a backward propagation calculation task, and the method further includes: during at least part of the training process, for at least one multi-die acceleration card among the multiple multi-die acceleration cards, perform the following operations: determine that when the first die in the multi-die acceleration card performs the forward propagation calculation task, the second die in the multi-die acceleration card performs the backward propagation calculation task; or determine that when the first die in the multi-die acceleration card performs the backward propagation calculation task, the second die in the multi-die acceleration card performs the forward propagation calculation task, so that the forward propagation calculation task and the backward propagation calculation task respectively performed by the first die and the second die overlap at least partially in time.

[0010] The task scheduling method provided in at least one embodiment of this disclosure further includes: for at least one of the plurality of multi-core accelerator cards, performing the following operations: determining that the activation value obtained by the first core of the multi-core accelerator card from performing a forward propagation computation task is transmitted to the second core of the multi-core accelerator card for storage through an inter-core interconnection link, wherein the stage corresponding to the forward propagation computation task of the second core is arranged in a forward order after the stage corresponding to the forward propagation computation task of the first core.

[0011] In the task scheduling method provided in at least one embodiment of this disclosure, the plurality of multi-core accelerator cards transmit data through direct links, and the plurality of cores transmit data through inter-core interconnect links.

[0012] In the task scheduling method provided in at least one embodiment of this disclosure, the multi-core accelerator card is a dual-core accelerator card, which includes two cores integrated in a single package.

[0013] This disclosure provides at least one embodiment of a training method applied to a training system comprising multiple multi-chip accelerator cards, the multi-chip accelerator cards comprising multiple chips integrated in a single package, the training method comprising: each chip of at least one of the multiple multi-chip accelerator cards performing the following operations: executing corresponding training tasks of a first pipeline and a second pipeline according to two stages carried by the chip, respectively, to train a model to be trained, wherein the model to be trained is divided into multiple stages, each stage comprising at least one layer of the model to be trained, the first pipeline and the second pipeline are mapped to the chips of the multiple multi-chip accelerator cards, the first pipeline and the second pipeline are constructed based on the multiple stages, the first pipeline and the second pipeline have opposite data flows and are used to process different training batches.

[0014] In the training method provided in at least one embodiment of this disclosure, the corresponding training tasks of the first pipeline and the second pipeline are executed according to the two stages carried by the chip, including: during at least part of the training process, when the first chip in the multi-chip accelerator card performs a forward propagation computation task, the second chip in the multi-chip accelerator card performs a backward propagation computation task; or during at least part of the training process, when the first chip in the multi-chip accelerator card performs a backward propagation computation task, the second chip in the multi-chip accelerator card performs a forward propagation computation task, such that the forward propagation computation task and the backward propagation computation task performed by the first chip and the second chip respectively overlap at least partially in time.

[0015] The training method provided in at least one embodiment of this disclosure further includes: for at least one of the plurality of multi-core accelerator cards, the first core of the multi-core accelerator card transmits the activation value obtained by performing the forward propagation computation task to the second core of the multi-core accelerator card for storage through the inter-core interconnection link, wherein the stage corresponding to the forward propagation computation task of the second core is arranged in a forward order after the stage corresponding to the forward propagation computation task of the first core.

[0016] In the training method provided in at least one embodiment of this disclosure, the plurality of multi-chip accelerator cards transmit data through direct links, and the plurality of chips transmit data through inter-chip interconnect links.

[0017] At least one embodiment of this disclosure provides a task scheduling apparatus applied to a training system including multiple multi-core accelerator cards, wherein the multi-core accelerator cards include multiple cores integrated in a single package. The apparatus includes: a partitioning module configured to partition a model to be trained into multiple stages, and to construct a first pipeline and a second pipeline with opposite data flows for processing different training batches based on the multiple stages, wherein each stage includes at least one layer of the model to be trained; and an allocation module configured to map the first pipeline and the second pipeline to the cores of the multiple multi-core accelerator cards, such that each core in at least one multi-core accelerator card carries two stages, the two stages being used to execute corresponding training tasks of the first pipeline and the second pipeline, respectively.

[0018] At least one embodiment of this disclosure provides a training system including multiple multi-chip accelerator cards, each multi-chip accelerator card including multiple chips integrated in a single package, wherein each chip of at least one of the multiple multi-chip accelerator cards is configured to: execute corresponding training tasks of a first pipeline and a second pipeline according to two stages carried by the chip, respectively, to train a model to be trained, wherein the model to be trained is divided into multiple stages, each stage including at least one layer of the model to be trained, the first pipeline and the second pipeline are mapped to the chips of the multiple multi-chip accelerator cards, the first pipeline and the second pipeline are constructed based on the multiple stages, and the data flow of the first pipeline and the second pipeline is opposite and used to process different training batches.

[0019] At least one embodiment of this disclosure provides an electronic device, including: at least one processor; at least one memory including one or more computer program modules; wherein the one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the method provided by at least one embodiment of this disclosure.

[0020] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by at least one processor, perform the method provided in at least one embodiment of this disclosure.

[0021] The task scheduling method, task scheduling device, training method, training system, electronic device, and storage medium provided in at least one embodiment of this disclosure align the logical dual-pipeline with the physical multi-core architecture, utilizing the inherent multi-core physical structure of multi-core hardware to support the bidirectional data flow structure of the dual-pipeline. Since each core has independent computing resources and execution capabilities, multiple cores in a multi-core accelerator card can execute different operators in parallel. This allows two cores in the same multi-core accelerator card to execute forward propagation and backward propagation tasks simultaneously, effectively transforming the "logical overlap" of forward and backward propagation tasks into "physical parallelism," significantly improving the training efficiency and scalability of large-scale models on device clusters. Compared to time-sharing multiplexing of single-core accelerator cards, the theoretical computational capacity for model training achieved using the above strategy is doubled, and the operator switching overhead within single-core accelerator cards is effectively avoided. This solution is suitable for steady-state training scenarios with a sufficient number of micro-batches (typically ≥ 2 × the number of pipeline stages). Task scheduling can be used to achieve overlap of pipeline stages in physical space and deep overlap of computation and D2D communication in the time dimension, thereby maximizing hardware resource utilization and hiding communication latency. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0023] Figure 1A This is a schematic diagram of a pipelined parallel training process.

[0024] Figure 1B This is a schematic diagram of a dual-path pipeline parallel training process.

[0025] Figure 1C This is a schematic structural diagram of a single-chip general-purpose graphics processor provided for at least one embodiment of the present disclosure.

[0026] Figure 1D This is a schematic structural diagram of a dual-core general-purpose graphics processor provided for at least one embodiment of the present disclosure.

[0027] Figure 2 This is a schematic flowchart illustrating a task scheduling method provided in at least one embodiment of the present disclosure.

[0028] Figure 3 This is a schematic flowchart illustrating a training method provided in at least one embodiment of the present disclosure.

[0029] Figure 4A This is a schematic diagram of a training system provided in at least one embodiment of the present disclosure.

[0030] Figure 4B This is a schematic diagram of a task scheduling method provided in at least one embodiment of the present disclosure.

[0031] Figure 5 This is a schematic block diagram of a task scheduling device provided for at least one embodiment of the present disclosure.

[0032] Figure 6 This is a schematic block diagram of a training system provided for at least one embodiment of the present disclosure.

[0033] Figure 7 This is a schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0034] Figure 8 This is a schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0035] Figure 9 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0037] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” and “right” are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0038] The present disclosure will now be described through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and known components may be omitted. When any component of an embodiment of the present disclosure appears in more than one drawing, the component is represented by the same or similar reference numerals in each drawing.

[0039] Because a large-scale model (such as a Transformer-based model) can have hundreds of billions of parameters, a single machine or GPU is no longer sufficient to handle the training task. Therefore, distributed training is widely used for large-scale model training in artificial intelligence. Distributed training refers to using multiple machines to work collaboratively to accelerate the training process of large deep learning models. In distributed training, the original model, dataset, and training process are decomposed and distributed across multiple machines for simultaneous processing, thereby effectively utilizing more computing resources and shortening training time. During model training, various parallel strategies can be adopted to save GPU memory and improve computational efficiency, such as data parallelism (DP), pipeline parallelism (PP), and tensor parallelism (TP).

[0040] Data parallelism refers to dividing the training dataset of a model into multiple subsets, which are then distributed across multiple computing devices. Each computing device holds a complete copy of the model, enabling it to independently train the subset assigned to it. For example, each computing device can independently perform forward and backward propagation computations on its subset.

[0041] Tensor parallelism refers to breaking down a layer (or operator) of a model and distributing some of the operator's weights across different computing devices, thereby reducing the memory usage on each device (e.g., a GPU). A neural network model consists of multiple layers, each of which can be understood as a function that performs specific mathematical operations on the input tensor. In tensor parallelism, the tensor of a layer is divided into multiple parts, and these parts are distributed across multiple computing devices. Each device is responsible for only a portion of the model's computation and exchanges necessary intermediate results through communication.

[0042] Pipeline parallelism is a crucial technique for training large-scale models. In pipeline parallelism, the model's network layers are sequentially divided into multiple stages, each containing several consecutive layers and distributed across different computing devices, thus forming a cross-device computational pipeline. For example, during forward propagation, each computing device passes activation values ​​to the next pipeline stage, and during backward propagation, each computing device transmits gradients back to the previous pipeline stage.

[0043] To improve pipeline throughput, a global training batch is typically divided into multiple micro-batches. These micro-batches flow sequentially into the pipeline, progressing step-by-step between different stages, forming a pipeline-like execution pattern. As the number of micro-batches increases, the pipeline approaches full capacity, and pipeline bubbles (i.e., idle time for computing devices waiting for data) are compressed, significantly improving computational efficiency. Through this mechanism, multiple computing devices can concurrently process data from different micro-batches within the same training cycle, achieving computational overlap in the time dimension and effectively improving hardware resource utilization and overall training efficiency.

[0044] Figure 1A This is a schematic diagram of a pipelined parallel training process.

[0045] Figure 1A The pipelined parallel training process illustrated employs a One Forward pass followed by One Backward pass (1F1B) scheduling strategy. The core idea of ​​the 1F1B strategy is that within each pipeline stage, after completing the forward propagation computation of one micro-batch sequentially, its corresponding backward propagation computation immediately begins, without waiting for all micro-batches to complete their forward propagation, thereby improving training efficiency and throughput. Under this scheduling mechanism, the computing device in each stage immediately begins processing and generating output after receiving the output from the previous stage, and then passes it to the computing device in the next stage.

[0046] For example, such as Figure 1A As shown, the multiple layers of the model to be trained are divided into four stages (stages A to D, not shown in the figure), which are assigned to computing devices A through D for computation. Each stage includes, for example, eight micro-batches of data. Fi in the row corresponding to computing device A represents the forward propagation of the i-th micro-batch of data in stage A, Bx in the row corresponding to computing device D represents the backward propagation of the x-th micro-batch of data in stage D, and so on. During forward computation, the computation of the next stage depends on the computation of the previous stage. For example, the forward propagation F1 of the first micro-batch of data on computing device B (corresponding to stage B) can only begin after the F1 computation on computing device A (corresponding to stage A) is completed. Similarly, during backward computation, the computation of the previous stage depends on the computation of the next stage. For example, the backward propagation B1 of the first micro-batch of data on computing device C (corresponding to stage C) can only begin after the B1 computation on computing device D (corresponding to stage D) is completed.

[0047] Due to this dependency, each computing device will be idle while waiting for input data or gradients. For example, in the early stages of training, only computing device A starts processing F1, while computing devices B, C, and D are all waiting. This process results in a large number of wasted computation cycles, forming significant pipeline bubbles (e.g., ...). Figure 1A (As shown in the blank box in the middle), this phenomenon is particularly severe when there are many stages or a small number of micro-batches, which restricts the utilization of hardware resources and training throughput.

[0048] To reduce pipeline bubbles, a dual-pipeline parallel approach can be employed, logically constructing two pipelines with opposite data flows but sharing hardware resources. During training, these two pipelines run concurrently, processing different micro-batches of data and each performing its own complete forward and backward propagation. Thanks to this design, while the first pipeline is processing the forward computation of one micro-batch, the second pipeline can simultaneously process the forward computation of another micro-batch. Once a micro-batch completes its entire forward stage in its respective pipeline, its backward propagation can begin immediately without waiting for all forward tasks in other pipelines to complete. Because the two pipelines advance alternately, forward and backward propagation achieve closer overlap on the timeline, significantly reducing pipeline bubbles.

[0049] Figure 1B This is a schematic diagram of a dual-path pipeline parallel training process.

[0050] like Figure 1BAs shown, the model to be trained is divided into 8 consecutive stages (Stage 0 to Stage 7), each stage including one or more consecutive model layers. During training, any micro-batch of data is propagated sequentially from Stage 0 to Stage 7; the corresponding backpropagation starts from Stage 7 and propagates gradients back to Stage 0. For example, assuming the model to be trained includes 40 model layers (Layer 0 to Layer 39), each stage can include five consecutive model layers. The correspondence between stages and model layers can be seen in the following example: Stage 0 includes Layer 0 to Layer 4; Stage 1 includes Layer 5 to Layer 9; Stage 2 includes Layer 10 to Layer 14, and so on.

[0051] like Figure 1B As shown, the eight stages of the model to be trained can be logically organized into two independent pipelines with opposite data flow directions (Stream A and Stream B). Each pipeline contains all eight stages (Stage 0 to Stage 7), and corresponding stages in these two pipelines are paired and assigned to the same computing device. The arrows in the figure indicate the forward propagation directions of Stream A and Stream B, respectively. Figure 1B Eight computing devices (Card0~Card7) are shown. Specifically, Stage i of the first pipeline and Stage 7-i of the second pipeline can be assigned to computing device Card i (where 0≤i<8, and i is a non-negative integer).

[0052] For example, stage 0 of the first pipeline and stage 7 of the second pipeline are assigned to Card 0, stage 1 of the first pipeline and stage 6 of the second pipeline are assigned to Card 1, and so on. During training, the first and second pipelines run simultaneously, processing different micro-batches of data and performing complete forward and backward propagation respectively. When the first pipeline performs forward propagation on the first micro-batch, its computation proceeds in a forward order from Stage 0 to Stage 7, corresponding to Card 0 to Card 7 on the physical device. The second pipeline can simultaneously perform forward propagation on the second micro-batch, and its computation also proceeds in a forward order from Stage 0 to Stage 7, but corresponding to Card 7 to Card 0 on the physical device. Similarly, when the first pipeline performs backpropagation on the first microbatch, its computation is performed in reverse order from Stage 7 to Stage 0, corresponding to Card 7 to Card 0 on the physical device. When the second pipeline performs backpropagation on the second microbatch, its computation is also performed in reverse order from Stage 7 to Stage 0, corresponding to Card 0 to Card 7 on the physical device. In this way, when training reaches a steady state, the logical overlap of forward and backward propagation in the two pipelines can be achieved.

[0053] The training system may include multiple computing nodes, each computing node may include one or more hosts, and each host may connect to one or more computing devices. The hosts and computing devices in a computing node can work together to complete computing tasks. This disclosure does not limit the number of computing nodes included in the training system, or the number of hosts and computing devices included in each computing node. Within a computing node, data transmission between the host and computing devices can be achieved in various ways. A computing device (also referred to as a "slave device") can either be directly connected to the host via a hardware interface (e.g., via a PCI Express, PCIe bus) to achieve low-latency data transmission, or it can work collaboratively with the host via a network in a software-defined manner (such as virtualization technology).

[0054] For example, the host described above may include a central processing unit (CPU), and the computing device may include a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), an accelerated processing unit (APU), a neural network processing unit (NPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA), etc.

[0055] Figure 1C This is a schematic structural diagram of a single-chip general-purpose graphics processor provided for at least one embodiment of the present disclosure.

[0056] like Figure 1C As shown, a general-purpose graphics processor is actually an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), such as including... Figure 1C The diagram shows streaming processor clusters 1, ..., M, where M is a positive integer greater than 1. In a general-purpose graphics processing unit (GPU), one streaming processor cluster handles one computational task, or multiple streaming processor clusters handle one computational task. Multiple streaming processor clusters share data through a global cache or global memory. A streaming processor cluster can also be referred to as a computational core.

[0057] like Figure 1C As shown, taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple computing units, such as... Figure 1C The system is structured as Computation Unit 1, Computation Unit 2, ..., Computation Unit N, where N is a positive integer. Each Computation Unit (CU) performs arithmetic and logical operations, such as accumulation, reduction, and standard addition, subtraction, multiplication, and division. A Computation Unit includes multiple Cores, each containing an Arithmetic Logic Unit (ALU), a floating-point unit, etc., and is used to execute specific computational tasks. Furthermore, the Computation Unit also includes registers (e.g., ...). Figure 1CThe register file and shared memory in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared memory in a computing unit is used to share data between the cores of that computing unit.

[0058] like Figure 1C As shown, each streaming processor cluster also provides a buffer for caching data from the N computing units within the streaming processor cluster.

[0059] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 1C (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0060] In each computing unit, the thread beam scheduling / distribution module ( Figure 1C (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple cores of the computing unit can run thread bundles. Depending on the number of cores in the computing unit, multiple thread bundles in a thread block can be executed concurrently or in a time-sharing manner. Multiple threads in each thread bundle will execute the same instructions. Memory execution instructions are issued to shared memory in the computing unit or further issued to intermediate-level cache, global cache, or global memory (e.g., High Bandwidth Memory, HBM) for read and write operations, etc.

[0061] In the fields of artificial intelligence and high-performance computing, accelerator cards based on multi-die architecture (also known as multi-grid or multi-die accelerator cards) are gradually becoming widely used. Multi-die technology integrates two or more dies (also known as chips or dies) with independent computing capabilities onto the same substrate or interposer through advanced packaging (such as 2.5D / 3D packaging or silicon interposers), enabling them to work together efficiently and present a single, logically unified, high-performance chip. When the package contains two dies, this architecture can also be called a dual-die architecture.

[0062] The multi-chip accelerator card (also known as a multi-chip device) provided in at least one embodiment of this disclosure may include a multi-chip graphics processor, a multi-chip general-purpose graphics processor, a multi-chip tensor processor, a multi-chip deep learning processor, a multi-chip accelerator processor, a multi-chip neural network processor, etc., and the embodiments of this disclosure do not limit this.

[0063] The above text Figure 1C The general-purpose graphics processor shown is a single-die architecture, meaning that all streaming processor clusters (including compute units and caches), global cache, and high-bandwidth memory are integrated onto a single die. The following uses... Figure 1C Based on the single-core general-purpose graphics processor shown, an example of a dual-core general-purpose graphics processor is introduced.

[0064] Figure 1D This is a schematic structural diagram of a dual-die general-purpose graphics processor provided for at least one embodiment of the present disclosure. The dual-die general-purpose graphics processor packages two independent GPGPU dies into a single board, presenting itself externally as a single logically unified high-performance chip.

[0065] For example, such as Figure 1D As shown, a dual-core general-purpose graphics processor may include a first core, a second core, and global memory. Each of the first and second cores may include a global cache and multiple streaming processor clusters, and the streaming processor cluster structure within each core may be similar to the aforementioned... Figure 1C The single-core embodiment shown has the same structure, where each streaming processor cluster includes multiple computing units, and each computing unit includes multiple cores, a register file, and shared memory. The global cache within the core can support data caching for all computing units within that core. The first core and the second core can be coupled via a high-speed interconnect structure. Global memory can be shared by the first and second cores, supporting concurrent access to global storage resources by both cores. Through the above dual-core architecture, the scale of computing resources can be expanded without exceeding the limitations of a single photomask size, thereby improving the computing throughput and energy efficiency of general-purpose graphics processors. Of course, the above dual-core general-purpose graphics processor is merely an example, and this disclosure does not limit its scope.

[0066] An operator is an executable unit representing a specific computational task (such as a convolution operator, matrix multiplication operator, or communication operator). Each operator ultimately needs to be mapped to a specific circuit on a chip for execution. Since each chip has independent computing resources and execution capabilities, in a multi-chip accelerator card, multiple chips can execute different operators in parallel; however, in a single-chip accelerator card, all operators must be executed serially or in time-division multiplexing on the same chip, making cross-chip operator-level parallelism impossible.

[0067] for Figure 1BThe dual-pipeline parallel training process shown employs single-core computing devices. Because single-core devices lack physically independent computing resources, they cannot truly execute the forward propagation task in the first pipeline and the backward propagation task in the second pipeline in parallel at the same time. In other words, on a single-core accelerator card, only one operator can be executed at any given time. The so-called "parallelism" is achieved through time-sharing multiplexing, only achieving overlap in communication and computation (i.e., logical overlap), but not the true simultaneous execution (i.e., physical parallelism) of two computationally intensive tasks (such as forward and backward propagation). Figure 1B Taking a single-core device, Card 3, as an example, in Stream A, all computational cores are used to execute the forward propagation computation of Stage 3. At this time, since the computational resources are fully occupied, the backward propagation computation of Stage 4 cannot be executed simultaneously in Stream B. Therefore, only logical parallelism can be achieved within a single-core device, but not physical parallelism.

[0068] Furthermore, some pipeline schedulers, during the execution phase allocation, typically treat all computing devices within the cluster (training system) as topologically equivalent nodes and employ linear or random mapping strategies. For example, Stage 0 might be allocated to GPU0, Stage 1 to GPU1, and so on. However, this mapping strategy ignores the existence of die-to-die (D2D) interconnects in multi-die devices.

[0069] The dual-pipeline parallel training process described above involves extremely frequent "positive activation propagation" and "backward gradient propagation" during the steady-state phase. If logically adjacent stages (e.g., two consecutive stages) are deployed on two accelerator cards across a node, instead of prioritizing deployment on two chips within the same accelerator card, high-frequency communication will flow through the low-bandwidth cross-node network, causing severe communication congestion; while the high-bandwidth D2D link may remain idle, resulting in resource waste.

[0070] This disclosure provides at least one embodiment of a task scheduling method, a task scheduling apparatus, a training method, a training system, an electronic device, and a non-transitory computer-readable storage medium.

[0071] The task scheduling method provided in at least one embodiment of this disclosure is applied to a training system including multiple multi-chip accelerator cards, wherein the multi-chip accelerator cards include multiple chips integrated in a single package. The method includes: dividing a model to be trained into multiple stages, and constructing a first pipeline and a second pipeline with opposite data flows for processing different training batches based on the multiple stages, wherein each stage includes at least one layer of the model to be trained; mapping the first pipeline and the second pipeline to the chips of the multiple multi-chip accelerator cards, such that each chip in at least one multi-chip accelerator card carries two stages, the two stages being used for corresponding training tasks of the first pipeline and the second pipeline, respectively.

[0072] The task scheduling method provided in at least one embodiment of this disclosure aligns the logical dual-pipeline with the physical multi-core architecture, utilizing the inherent multi-core physical structure of multi-core hardware to support the bidirectional data flow structure of the dual-pipeline. Since each core possesses independent computing resources and execution capabilities, multiple cores in a multi-core accelerator card can execute different operators in parallel. This allows two cores in the same multi-core accelerator card to execute forward propagation and backward propagation tasks simultaneously, effectively transforming the "logical overlap" of forward and backward propagation tasks into "physical parallelism," significantly improving the training efficiency and scalability of large-scale models on device clusters.

[0073] Compared with time-division multiplexing of single-core accelerator cards, the theoretical upper limit of model training achieved by the above task scheduling method is doubled, and the operator switching overhead inside the single-core accelerator card is effectively avoided.

[0074] The task scheduling method provided in at least one embodiment of this disclosure is particularly suitable for steady-state training scenarios with a sufficient number of micro-batches (typically ≥ 2 × the number of pipeline stages). Task scheduling can achieve overlap of pipeline stages in physical space and deep overlap of computation and D2D communication in the time dimension, thereby maximizing hardware resource utilization and hiding communication latency.

[0075] The task scheduling method provided in at least one embodiment of this disclosure can be applied to a training system including multiple multi-core accelerator cards. A multi-core accelerator card includes multiple cores integrated in a single package, as described above. One example of a multi-core accelerator card is a dual-core accelerator card, which includes two cores integrated in a single package.

[0076] In this training system, multiple multi-core accelerator cards transmit data via direct links, while multiple cores within a single multi-core accelerator card transmit data via die-to-die (D2D) links.

[0077] For example, accelerator cards can achieve point-to-point connection via a communication medium, thus realizing a direct connection. For example, the communication medium may include a direct-attach copper cable (DAC); or, for example, an active electrical cable (AEC). This disclosure does not specifically limit the type of communication medium.

[0078] In terms of hardware architecture, each multi-core accelerator card can integrate at least one direct link controller and a corresponding physical port supporting high-speed direct link protocols. Each physical port can contain multiple high-speed differential signal lines for connecting to external communication media. When two multi-core accelerator cards connect their respective physical ports through a communication medium, a high-speed direct link is formed between them, which constitutes a high-speed serial point-to-point physical channel.

[0079] For example, data can be transmitted in blocks over the link, and cross-card memory access can be achieved through address mapping mechanisms provided by the driver. For instance, accelerator cards interconnected through this high-speed direct link can have a shared memory pool; after one accelerator card writes data to a shared address, another accelerator card can directly read the data through the mapped address in its local process, without the need for host relay or explicit copying.

[0080] Furthermore, multi-core accelerator cards can be configured to support load / store semantics. For example, the remote accelerator card pre-allocates a storage space and maps it to a unified virtual address visible to the local device thread through the driver; the local accelerator card thread can then directly read and write the memory of the remote accelerator card through standard load or store instructions to achieve low-latency, high-bandwidth collaborative computing.

[0081] During the aforementioned communication process, the Direct Memory Access (DMA) engine in the multi-chip accelerator card can work in conjunction with the direct link controller: when inter-card batch data transfer is required, the DMA engine, as the initiator and scheduling unit for data transfer, generates memory read / write requests; these requests are routed to the direct link controller via the on-chip interconnect, where the direct link controller encapsulates the requests into data packets conforming to the high-speed direct link protocol and sends them to the destination accelerator card via the high-speed direct link. Therefore, the DMA engine resides at the transport logic layer, responsible for task scheduling and data flow control, while the direct link controller and the high-speed direct link constitute the underlying hardware path, providing high-bandwidth, low-latency physical connectivity.

[0082] Multiple chips within the same multi-chip accelerator card can be interconnected via D2D links using a D2D bus architecture. The bus protocol runs on this link, supporting concurrent access from multiple chips. A D2D link refers to a direct, high-speed interconnect between multiple chips located within the same package through an intra-package interconnect structure. For example, advanced packaging interconnect technologies such as through-silicon vias (TSVs), redistribution layers (RDLs), or microbumps can physically construct short-distance, high-density electrical connection channels, thereby supporting high-bandwidth, low-latency, and low-power data communication between chips.

[0083] The task scheduling method provided in at least one embodiment of this disclosure can be executed by a scheduler, which can be the training framework itself or a component of the training framework. The training framework is used to build, train, and optimize machine learning or deep learning models, and can provide a high-level abstraction interface to simplify processes such as model definition, data loading, loss calculation, gradient update, distributed training, and device management. The training framework can have a built-in task scheduling mechanism (e.g., data parallelism, model parallelism, pipeline scheduling, etc.) and can also support custom scheduling strategies through plugins or extensions to efficiently utilize computing resources and accelerate the training process.

[0084] Figure 2 This is a schematic flowchart illustrating a task scheduling method provided in at least one embodiment of the present disclosure.

[0085] For example, such as Figure 2 As shown, at least one embodiment of the present disclosure provides a task scheduling method including steps S101 to S102.

[0086] Step S101: Divide the model to be trained into multiple stages, and construct a first pipeline and a second pipeline with opposite data flow directions for processing different training batches based on the multiple stages, wherein each stage includes at least one layer of the model to be trained.

[0087] Step S102: Map the first pipeline and the second pipeline to the chips of multiple multi-chip accelerator cards, so that each chip in at least one multi-chip accelerator card carries two stages, which are used to execute the corresponding training tasks of the first pipeline and the second pipeline, respectively.

[0088] The model to be trained mentioned above can be a machine learning model to be trained, such as a deep learning model. Deep learning models can be neural network structures with multiple layers (3, 4, 8, or more layers), such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Long Short-Term Memory (LSTM) networks. These neural network structures typically include input layers, hidden layers, and output layers. Hidden layers are those layers located between the input and output layers; they are also called processing layers. The input layer receives the data to be processed, such as the image to be processed, and the output layer outputs the processing result, such as the processed image. Processing layers can include convolutional layers, pooling layers, batch normalization layers, fully connected layers, etc. Depending on the structure of the neural network, the processing layer can include different content and combinations. In some examples, the model to be trained can be an artificial intelligence model with a large number of parameters built from an artificial intelligence network (also known as a large-scale artificial intelligence model, or simply a "large model"). Large-scale models can be content generation models based on prompt words, such as Large Language Models (LLMs), large-scale visual models, or multimodal large-scale models. Examples include models based on Transformer architectures, recurrent neural networks, or attention mechanisms. The model to be trained can also be a hybrid expert model, comprising multiple expert subnetworks and a gating mechanism. Each input sample activates only some of the experts. This architecture can significantly reduce actual computational overhead while maintaining high model capacity, and is therefore widely used in large-scale model scenarios. The specific type, structure, or implementation of the model to be trained is not limited in the embodiments disclosed herein.

[0089] For example, in step S101, model partitioning can be achieved through uniform or non-uniform partitioning: uniform partitioning involves distributing the model to be trained approximately equally across stages based on the number of layers or computational cost to balance the computational load of each computing device; non-uniform partitioning involves differentiated partitioning based on the actual computational complexity, parameter size, activation value, or communication overhead of each layer to optimize overall pipeline throughput or reduce pipeline bubbles. The partitioning result can be explicitly configured by the user or dynamically generated by the training framework based on automatic analysis of the model structure and cluster topology information, ensuring that the constructed pipeline maximizes hardware resource utilization and training efficiency while satisfying parallel strategy constraints.

[0090] For example, the training framework can scan the cluster hardware to obtain cluster topology information. By calling the underlying device driver interface, the training framework automatically detects the number, model, memory capacity, and interconnect topology of available accelerators on each compute node. Simultaneously, combined with the cluster management environment, it identifies all nodes participating in training and their network connectivity capabilities (e.g., whether they support Remote Direct Memory Access (RDMA)). Based on this information, the training framework can dynamically construct a physical topology map containing compute nodes, accelerator cards, and link attributes. This topology map accurately reflects the communication paths between hardware components, as well as the number of compute nodes and accelerator cards in the cluster, serving as the basis for subsequent determination of parallel strategies and task scheduling. This enables efficient and adaptive distributed training without manual configuration.

[0091] After completing the cluster hardware scan and constructing the physical topology map, the training framework can further divide the entire model to be trained by combining the user-specified parallel strategy configuration (such as data parallelism, model parallelism, pipeline parallelism, and partitioning method). Specifically, the training framework can divide the model into several continuous or non-continuous stages according to the number of stages defined in the user configuration and the range of layers or operators contained in each stage.

[0092] A global training batch is a subset of the input samples. To achieve finer-grained computation scheduling in pipelined parallelism, a global training batch can be divided into multiple micro-batches. Each micro-batch contains a portion of the input samples from the global batch and can undergo forward and backward propagation independently.

[0093] In step S101, the model to be trained can be divided into N stages (Stage 0 to Stage N-1), and based on these N stages, two pipelines are logically constructed to process different micro-batches with opposite data flow directions: the first pipeline and the second pipeline. Each pipeline completely includes all stages from Stage 0 to Stage N-1, but the scheduling direction of its micro-batches is opposite.

[0094] It should be noted that "first production line" and "second production line" are exemplary terms used to distinguish different production lines, and do not refer to a specific production line, nor do they indicate a specific order, hierarchical relationship or quantity limit between production lines.

[0095] Through the construction of the pipeline described above, during training execution, any micro-batch will be processed sequentially within its respective pipeline: forward propagation starts from Stage 0 and propagates step by step to Stage N-1; the corresponding backward propagation starts from Stage N-1 and propagates gradients back to Stage 0 step by step. The first and second pipelines constructed in step S101 are used to process different micro-batches. For example, for a global training batch containing 20 micro-batches (MB 0~MB 19), the first pipeline can process MB 0~MB 9, and the second pipeline can process MB 10~MB 19, thereby achieving bidirectional, interleaved pipeline execution on the same hardware resources.

[0096] In step S102, the first and second pipelines constructed in step S101 are mapped to the chips of multiple multi-chip accelerator cards. Each chip in at least one multi-chip accelerator card carries two stages, which are used to execute the corresponding training tasks of the first and second pipelines, respectively. For example, the training tasks may include forward propagation computation, backpropagation computation, gradient synchronization, parameter update, etc. After the mapping is completed, each chip in at least one multi-chip accelerator card carries both a stage belonging to the first pipeline and a stage belonging to the second pipeline. That is, each chip in at least one multi-chip accelerator card needs to execute the training tasks of the first and second pipelines during training.

[0097] Through the above mapping, the first and second pipelines share the same set of hardware resources. Since each chip has independent computing resources and execution capabilities, multiple chips in a multi-chip accelerator card can execute different operators in parallel. This allows two chips in the same multi-chip accelerator card to execute forward propagation and backward propagation tasks at the same time, effectively achieving physical overlap between the forward and backward propagation tasks and significantly improving the training efficiency and scalability of large-scale models on device clusters.

[0098] An example of step S101 may include: dividing the model to be trained into N consecutive stages, and constructing a first pipeline and a second pipeline based on the N consecutive stages, wherein N is a positive integer, the first pipeline includes N consecutive stages and schedules data between execution units in a first direction, the second pipeline includes N consecutive stages and schedules data between execution units in a second direction opposite to the first direction, and the execution units mentioned above include kernels.

[0099] In the example above, the model to be trained is divided into N consecutive stages (Stage 0 to Stage N-1), and the first pipeline and the second pipeline built based on this division both include the complete N consecutive stages Stage 0 to Stage N-1.

[0100] For example, if there are N execution units (EU 0 to EU N-1) corresponding one-to-one with the N consecutive stages, then during forward propagation of the first pipeline, data is scheduled between execution units in the direction from EU 0 to EU N-1; during backward propagation, data is scheduled between execution units in the direction from EU N-1 to EU 0. During forward propagation of the second pipeline, data is scheduled between execution units in the direction from EU N-1 to EU 0; during backward propagation, data is scheduled between execution units in the direction from EU 0 to EU N-1.

[0101] It should be noted that the terms "first direction" and "second direction" mentioned above are exemplary terms used to distinguish different data scheduling directions, and do not refer to a fixed direction, nor do they indicate a specific order, hierarchical relationship or quantity limit between directions.

[0102] In the task scheduling method provided in at least one embodiment of this disclosure, the execution unit can be a chip in a multi-chip accelerator card. For example, the training framework can assign a unified logical number to all chips participating in training in the training system for task scheduling and data flow control.

[0103] An example of step S102 may include the following steps S1021 to S1022.

[0104] Step S1021: For at least one of the multiple multi-core accelerator cards, perform the following operation: sequentially assign stages to each core of the multi-core accelerator card according to the stage arrangement order corresponding to the first pipeline and the second pipeline.

[0105] Step S1022: In response to the current allocation phase, all chips of the multi-chip accelerator card have been allocated, the next chip allocation phase in the multi-chip accelerator card begins.

[0106] In step S1021, since the data flow of the first pipeline and the second pipeline is opposite, the allocation of each chip is carried out in a different order for the first pipeline and the second pipeline.

[0107] An example of step S1021, "assigning stages to each chip of the multi-chip accelerator card in sequence according to the stage arrangement order corresponding to the first and second pipelines," may include the following steps S1021a to S1021b.

[0108] Step S1021a: For the first production line, assign stages to each chip of the multi-chip accelerator card in sequence according to the forward arrangement of stages.

[0109] Step S1021b: For the second pipeline, assign stages to each chip of the multi-chip accelerator card in reverse order of stage arrangement.

[0110] For example, for the continuous stages Stage 0 to Stage N-1 obtained after dividing the model to be trained, the forward order of the stages is from Stage 0 to Stage N-1, and the reverse order of the stages is from Stage N-1 to Stage 0.

[0111] Assuming the training system includes M multi-core accelerator cards (Card 0 ~ Card M-1), and each multi-core accelerator card includes L cores, then the L cores in Card j (0≤j<M) can be labeled as Die j_0 ~ Die j_L-1.

[0112] Taking Card 0 of M multi-core accelerator cards as an example, the allocation process begins. For the first pipeline, stages are allocated sequentially in a forward order: Die 0_0 to Die 0_L-1. That is, Stage 0 in the first pipeline is allocated to Die 0_0, Stage 1 to Die 0_1, Stage 2 to Die 0_2, ..., and Stage L-1 to Die 0_L-1. For the second pipeline, stages are allocated sequentially in a reverse order: Die 0_0 to Die 0_L-1. That is, Stage N-1 in the first pipeline is allocated to Die 0_0, Stage N-2 to Die 0_1, Stage N-3 to Die 0_2, ..., and Stage NL to Die 0_L-1.

[0113] Each die participating in the training task is assigned two stages, corresponding to the first pipeline and the second pipeline, respectively. For example, in the example above, Die 0_0 is assigned to Stage 0 of the first pipeline and Stage N-1 of the second pipeline.

[0114] In step S1022, if all chips of the multi-chip accelerator card currently being allocated a stage (also referred to as the current multi-chip accelerator card) have already been allocated a stage, then a stage can be allocated to the chips of the next multi-chip accelerator card. Here, "next multi-chip accelerator card" refers to the next available multi-chip accelerator card that immediately follows the current multi-chip accelerator card in the predefined multi-chip accelerator card traversal order. This traversal order can be determined based on factors such as the physical topology order, the logical ID order assigned by the training framework, or the order specified in the user configuration file.

[0115] In some examples, in response to the stage where all chips of the multi-chip accelerator card currently being allocated have been allocated to the first pipeline, the stage of allocating the first pipeline for chips in the next multi-chip accelerator card can continue. In response to the stage where all chips of the multi-chip accelerator card currently being allocated have been allocated to the second pipeline, the stage of allocating the second pipeline for chips in the next multi-chip accelerator card can continue. In other examples, in response to the stage where all chips of the multi-chip accelerator card currently being allocated have been allocated to both the first and second pipelines, the stage of allocating both the first and second pipelines for chips in the next multi-chip accelerator card can continue.

[0116] Corresponding to the example above, that is, when Die 0_0 to Die 0_L-1 in Card 0 have all been assigned stages, stages can then be assigned to Die 1_0 to Die 1_L-1 in Card 1. For example, for the first pipeline, Stages 0 to L-1 have been assigned to the various Dies in Card 0. Now, stages need to be assigned to Die 1_0 to Die 1_L-1 in the forward order of stages. That is, Stage L in the first pipeline is assigned to Die 1_0, Stage L+1 is assigned to Die 1_1, Stage L+2 is assigned to Die 1_2, ..., and Stage 2L-1 is assigned to Die 1_L-1. For the second production line, Stage N-1 to Stage NL have been assigned to the various chips in Card 0. Now, the stages need to be assigned sequentially in reverse order from Die 1_0 to Die 1_L-1. That is, Stage NL-1 from the first production line is assigned to Die 1_0, Stage NL-2 is assigned to Die 1_1, Stage NL-3 is assigned to Die 1_2, ..., and Stage N-2L is assigned to Die 1_L-1.

[0117] And so on, the cores in Card 2, Card 3, …, Card M-1 can continue to be assigned stages in a similar manner until the N stages of the first pipeline and the N stages of the second pipeline have been assigned to the respective cores of multiple multi-core accelerator cards, thus achieving the mapping of the first pipeline and the second pipeline to the respective cores of the multi-core accelerator cards.

[0118] An example of step S102 may include: Assigning the S corresponding to the first pipeline i and the S corresponding to the second pipeline N-i-1 to D j,k where , , where S i represents the (i + 1)-th stage among N consecutive stages, S N-i-1 represents the (N - i)-th stage among N consecutive stages, and D j,k represents the (k + 1)-th core in the (j + 1)-th multi-core accelerator card. represents floor function, and % represents modulo operation.

[0119] In this example, the training system includes M multi-core accelerator cards, and each multi-core accelerator card includes L cores. , and 0 ≤ i < N, 0 ≤ j < M, 0 ≤ k < L, where i, j, and k are non-negative integers.

[0120] In this example, the N stages are the first stage to the N-th stage, corresponding to S0 to S N-1 , that is, Stage 0 to Stage N - 1 as mentioned above. The M multi-core accelerator cards are the first multi-core accelerator card to the M-th multi-core accelerator card, corresponding to Card 0 to Card M - 1 as mentioned above. The L cores in a single multi-core accelerator card are the first core to the L-th core. For example, the first core D 1,0 in the second multi-core accelerator card corresponds to Die 1_0 as mentioned above.

[0121] According to the above formula, when N = 8, M = 4, and L = 2, the following mapping relationship can be obtained: When i = 0, j = 0, k = 0, N - i - 1 = 7, that is, the first stage S0 corresponding to the first pipeline and the eighth stage S7 corresponding to the second pipeline are assigned to the first core D 0,0 in the first multi-core accelerator card; when i = 1, j = 0, k = 1, N - i - 1 = 6, that is, the second stage S1 corresponding to the first pipeline and the seventh stage S6 corresponding to the second pipeline are assigned to the second core D 0,1In the middle; when i=2, j=1, k=0, Ni-1=5, that is, the third stage S2 corresponding to the first pipeline and the sixth stage S5 corresponding to the second pipeline are allocated to the first core D in the second multi-core accelerator card. 1,0 In the middle; and so on, when i=7, j=3, k=1, Ni-1=0, that is, the eighth stage S7 corresponding to the first pipeline and the first stage S0 corresponding to the second pipeline are allocated to the second core D in the fourth multi-core accelerator card. 3,1 middle.

[0122] The above allocation method can make full use of all the core resources in a single multi-core accelerator card, and prioritize the deployment of logically adjacent stages on the cores of the same multi-core accelerator card. Thus, during forward and backward propagation, high-frequency communication for transmitting activation values ​​and gradients can be prioritized through the D2D link, avoiding idle high-bandwidth D2D links and effectively improving the utilization rate of core resources.

[0123] The task scheduling method provided in at least one embodiment of this disclosure may further include: during at least part of the training process, for at least one of a plurality of multi-core accelerator cards, determining that when a forward propagation computation task is performed by a first core of the multi-core accelerator card, a backward propagation computation task is performed by a second core of the multi-core accelerator card, such that the forward propagation computation task and the backward propagation computation task performed by the first core and the second core, respectively, overlap at least partially in time.

[0124] For example, during at least part of the training process (e.g., the steady-state training phase), for the same multi-core accelerator card, the training framework can determine that while the first core performs the forward propagation computation task, the second core performs the backpropagation computation task, and send a corresponding task scheduling instruction to the second core to cause it to perform the backpropagation computation task. In some examples, the aforementioned forward propagation task and backpropagation task can belong to the first pipeline and the second pipeline, respectively.

[0125] The task scheduling method provided in at least one embodiment of this disclosure may further include: during at least part of the training process, for at least one of a plurality of multi-core accelerator cards, determining that when a backpropagation computation task is performed by a first core of the multi-core accelerator card, a forward propagation computation task is performed by a second core of the multi-core accelerator card, such that the forward propagation computation task and the backpropagation computation task performed by the first core and the second core, respectively, overlap at least partially in time.

[0126] For example, during at least part of the training process (e.g., the steady-state training phase), for the same multi-core accelerator card, the training framework can determine that while the first core is performing the backpropagation computation task, the second core is performing the forward propagation computation task, and send corresponding task scheduling instructions to the second core to enable it to perform the forward propagation computation task. In some examples, the aforementioned forward propagation task and backpropagation task can belong to the first pipeline and the second pipeline, respectively.

[0127] It should be noted that the terms "first chip" and "second chip" used here and below are exemplary terms used to distinguish different chips in the same multi-chip accelerator card, and do not refer to a specific chip, nor do they indicate a specific order, hierarchical relationship or quantity limit between chips.

[0128] The task scheduling method described above effectively enables two different cores within the same multi-core accelerator card to simultaneously execute forward and backward propagation computation tasks, respectively, during at least part of the training process. This achieves true physical parallelism between the forward and backward propagation computation tasks within the same multi-core accelerator card. Compared to time-division multiplexing with a single-core accelerator card, the theoretical computational capability for model training achieved using this task scheduling method is doubled, while effectively avoiding the operator switching overhead within a single-core accelerator card.

[0129] Taking Die 1_0 and Die 1_1 in Card 1 as examples: When Die 1_0 is performing the forward propagation computation task, it can be determined that Die 1_1 will perform the backward propagation computation task. When Die 1_1 is performing the forward propagation computation task, it can be determined that Die 1_0 will perform the backward propagation computation task.

[0130] The task scheduling method provided in at least one embodiment of this disclosure may further include: for at least one of a plurality of multi-core accelerator cards, determining that the activation value obtained by the first core of the multi-core accelerator card from performing a forward propagation computation task is transmitted to the second core of the multi-core accelerator card for storage through an inter-core interconnection link, wherein the stage corresponding to the forward propagation computation task of the second core is arranged in a forward order after the stage corresponding to the forward propagation computation task of the first core.

[0131] In the backpropagation algorithm, in order to calculate the gradient, it is necessary to access the activation values ​​generated during forward propagation.

[0132] In the example above, the training framework can control the data transmission and storage between cores, so that for a certain multi-core accelerator card, during the forward propagation phase, the first core can directly transmit and store the calculated activation value to the second core through a high-bandwidth D2D link. In the subsequent backpropagation phase, when the second core needs to perform gradient calculation for the corresponding phase, it can directly read the activation value from the local memory without recalculating or loading it from external storage space, thereby significantly reducing data loading latency and improving training efficiency.

[0133] In the task scheduling method provided in at least one embodiment of this disclosure, task scheduling follows the principle of first-come, first-served, and the execution order can be predetermined and fixed during the training process orchestration stage. Through the preset scheduling strategy, it is possible to effectively avoid scheduling computing tasks from different pipelines (i.e., the first pipeline and the second pipeline) to the same chip simultaneously, thereby preventing resource contention, ensuring execution determinism, and maintaining stable pipeline throughput.

[0134] Corresponding to the task scheduling method provided in at least one embodiment of this disclosure, at least one embodiment of this disclosure also provides a training method. This training method can be applied to a training system including multiple multi-core accelerator cards, as described above.

[0135] Figure 3 This is a schematic flowchart illustrating a training method provided in at least one embodiment of the present disclosure.

[0136] For example, such as Figure 3 As shown, the training method provided in at least one embodiment of this disclosure includes: performing the following step S201 by each chip of at least one multi-chip accelerator card from a plurality of multi-chip accelerator cards.

[0137] Step S201: Execute the corresponding training tasks of the first pipeline and the second pipeline according to the two stages carried by the core to train the model to be trained.

[0138] In step S201, the model to be trained is divided into multiple stages, each stage including at least one layer of the model to be trained. The first pipeline and the second pipeline are mapped to the chips of multiple multi-chip accelerator cards. The first pipeline and the second pipeline are constructed based on the above multiple stages. The data flow of the first pipeline and the second pipeline is opposite and used to process different training batches.

[0139] In the above training method, since each core has independent computing resources and execution capabilities, multiple cores in a multi-core accelerator card can execute different operators in parallel. This allows two cores in the same multi-core accelerator card to execute forward propagation and backward propagation tasks at the same time, effectively achieving physical overlap between forward propagation and backward propagation tasks, and significantly improving the training efficiency and scalability of large-scale models on device clusters.

[0140] An example of step S201 may include: during at least part of the training process, when the first chip in the multi-chip accelerator card performs a forward propagation computation task, the second chip in the multi-chip accelerator card performs a backward propagation computation task, such that the forward propagation computation task and the backward propagation computation task performed by the first chip and the second chip, respectively, overlap at least partially in time. For example, the aforementioned forward propagation task and backward propagation task may belong to a first pipeline and a second pipeline, respectively.

[0141] Another example of step S201 may include: during at least part of the training process, when the first chip in the multi-chip accelerator card performs a backpropagation computation task, the second chip in the multi-chip accelerator card performs a forward propagation computation task, such that the forward propagation computation task and the backpropagation computation task performed by the first chip and the second chip, respectively, overlap at least partially in time. For example, the aforementioned forward propagation task and backpropagation task may belong to the first pipeline and the second pipeline, respectively.

[0142] The training method described above effectively enables two different cores within the same multi-core accelerator card to simultaneously execute forward and backward propagation computation tasks, respectively, during at least part of the training process. This achieves true physical parallelism between the forward and backward propagation computation tasks within the same multi-core accelerator card. Compared to time-division multiplexing of a single-core accelerator card, the theoretical computational capability of this training method is doubled, and it effectively avoids the operator switching overhead within a single-core accelerator card.

[0143] The training method provided in at least one embodiment of this disclosure may further include: for at least one multi-core accelerator card among a plurality of multi-core accelerator cards, the first core of the multi-core accelerator card transmits the activation value obtained by performing a forward propagation computation task to a second core of the multi-core accelerator card for storage through an inter-core interconnection link, wherein the stage corresponding to the forward propagation computation task of the second core is arranged in a forward order after the stage corresponding to the forward propagation computation task of the first core.

[0144] Using the training method described above, for a multi-core accelerator card, during the forward propagation phase, the first core can directly transmit and store the calculated activation value to the second core via a high-bandwidth D2D link. In the subsequent backward propagation phase, when the second core needs to perform gradient calculations for the corresponding phase, it can directly read the activation value from the local memory without recalculating or loading it from external storage, thereby significantly reducing data loading latency and improving training efficiency.

[0145] For other related information about this training method, please refer to the relevant introduction in the task scheduling method section above, which will not be repeated here.

[0146] It should also be noted that the execution order of the steps in the task scheduling method and training method is not limited in the various embodiments of this disclosure. Although the execution process of each step has been described in a specific order above, this does not constitute a limitation on the embodiments of this disclosure. The steps in the task scheduling method and training method can be executed serially or in parallel, which can be determined according to actual needs.

[0147] For example, compared to the above description, the task scheduling method and training method provided in at least one embodiment of this disclosure may include more or fewer steps, and the embodiments of this disclosure do not limit this.

[0148] Figure 4A This is a schematic diagram of a training system provided in at least one embodiment of the present disclosure.

[0149] For example, such as Figure 4A As shown, the training system includes four dual-core accelerator cards (Card 0~Card 3), each of which includes two cores integrated in a single package.

[0150] The training framework can scan the cluster hardware to construct a physical topology map of the training system, defining each dual-core accelerator card as a "computation pair" and numbering the cores in each dual-core accelerator card. For example, Card 0 includes cores Die 0_0 and Die 0_1, Card 1 includes cores Die 1_0 and Die 1_1, Card 2 includes cores Die 2_0 and Die 2_1, and Card 3 includes cores Die 3_0 and Die 3_1.

[0151] Figure 4B This is a schematic diagram of a task scheduling method provided in at least one embodiment of the present disclosure. Figure 4B Task scheduling methods are applied to Figure 4A The training system shown.

[0152] like Figure 4A and Figure 4BAs shown, the training framework divides the model to be trained into 8 stages, and constructs a first pipeline (Stream A) and a second pipeline (Stream B) with opposite data flows to process different training batches based on these 8 stages. Stream A and Stream B are then mapped to the chips of four dual-chip accelerator cards. The mapping strategy is as follows: Stages 0 to 7 of Stream A are mapped sequentially to the chips of Card 0 to Card 3, and Stages 0 to 7 of Stream B are mapped sequentially to the chips of Card 3 to Card 0. After mapping, Stream A's Stage 0 and Stream B's Stage 7 are assigned to Die 0_0, Stream A's Stage 1 and Stream B's Stage 6 are assigned to Die 0_1, Stream A's Stage 2 and Stream B's Stage 5 are assigned to Die 1_0, Stream A's Stage 3 and Stream B's Stage 4 are assigned to Die 1_1, Stream A's Stage 4 and Stream B's Stage 3 are assigned to Die 2_0, Stream A's Stage 5 and Stream B's Stage 2 are assigned to Die 2_1, Stream A's Stage 6 and Stream B's Stage 1 are assigned to Die 3_0, and Stream A's Stage 7 and Stream B's Stage 0 are assigned to Die 3_1.

[0153] For a global training batch containing 20 micro-batches (MB 0~MB 19), Stream A can be used to process MB 0~MB 9, and Stream B can be used to process MB 10~MB 19. At the beginning of training, Die 0_0 can start processing the forward propagation calculation of MB 0, and Die 3_1 can start processing the forward propagation calculation of MB 10 at the same time, while other cores are in a waiting state at this time.

[0154] When Die 0_0 completes the forward propagation calculation of MB 0 in Stage 0, Die 0_1 can begin executing the forward propagation calculation of MB 0 in Stage 1, and Die 0_0 will simultaneously begin executing the forward propagation calculation of MB 1 in Stage 0. On the other hand, when Die 3_1 completes the forward propagation calculation of MB 10 in Stage 0, Die 3_0 can begin executing the forward propagation calculation of MB 10 in Stage 1, and Die 3_1 will simultaneously begin executing the forward propagation calculation of MB 11 in Stage 0. This continues, with all cores beginning their tasks. When Die 3_1 completes the forward propagation calculation of MB 0 in Stage 7, it will continue executing the backward propagation calculation of MB 0 in Stage 7, and so on, until Die 0_0 completes the backward propagation calculation of MB 0 in Stage 0. At this point, both the forward and backward propagation calculations of MB 0 are complete. During this process, training gradually enters a steady-state phase.

[0155] In the steady-state phase, taking Die 1_0 and Die 1_1 in Card 1 as examples, Die 1_0 receives the activation value data sent by Die 0_1 and performs the forward propagation calculation in Stage 2 of Stream A. Simultaneously, Die 1_1 receives the gradient data sent by Die 1_0 and performs the backpropagation calculation in Stage 4 of Stream B. At this time, the two particles in Card 1 simultaneously perform both forward and backpropagation calculation tasks, achieving a physical overlap between the forward and backpropagation calculations.

[0156] like Figure 4B As shown, the dual-core accelerator cards communicate with each other via a direct link, while the two cores within the card communicate with each other via a D2D link. Therefore, taking Die 1_0 as an example, during forward propagation, Die 1_0 can directly transmit and store the calculated activation values ​​to Die 1_1 via a high-bandwidth D2D link. In the subsequent backpropagation stage, when Die 1_1 needs to perform Stage 3 gradient calculations, it can directly read the activation values ​​locally without recalculating or loading them from external storage, thus significantly reducing data loading latency and improving training efficiency.

[0157] It should be noted that, Figure 4A and Figure 4B This is merely an example; the embodiments disclosed herein do not limit the number of stages, the number of multi-core accelerator cards in the training system, or the number of cores in the multi-core accelerator cards.

[0158] Figure 5This is a schematic block diagram of a task scheduling device provided for at least one embodiment of the present disclosure. The task scheduling device can be applied to a training system including multiple multi-chip accelerator cards, wherein the multi-chip accelerator cards include multiple chips integrated in a single package. For example, the task scheduling device can be disposed within a training frame.

[0159] For example, such as Figure 5 As shown, the task scheduling device 500 provided in at least one embodiment of this disclosure includes a partitioning module 501 and an allocation module 502.

[0160] For example, the partitioning module 501 is configured to partition the model to be trained into multiple stages, and to construct a first pipeline and a second pipeline with opposite data flow directions for processing different training batches based on the multiple stages, wherein each stage includes at least one layer of the model to be trained.

[0161] For example, the allocation module 502 is configured to map the first pipeline and the second pipeline to the chips of a plurality of multi-chip accelerator cards, such that each chip in at least one multi-chip accelerator card carries two stages, the two stages being used to execute the corresponding training tasks of the first pipeline and the second pipeline, respectively.

[0162] In some examples, the partitioning module is further configured to divide the model to be trained into N consecutive stages, and to construct a first pipeline and a second pipeline based on the N consecutive stages, where N is a positive integer. The first pipeline includes N consecutive stages and schedules data between execution units in a first direction, and the second pipeline includes N consecutive stages and schedules data between execution units in a second direction opposite to the first direction. The execution units include kernels.

[0163] In some examples, the allocation module is further configured to perform the following operation for at least one of a plurality of multi-core accelerator cards: sequentially allocate stages to each core of the multi-core accelerator card according to the stage arrangement order corresponding to the first pipeline and the second pipeline.

[0164] In some examples, the allocation module is further configured to allocate stages sequentially to each chip of the multi-chip accelerator card in a forward-order sequence for the first pipeline; and to allocate stages sequentially to each chip of the multi-chip accelerator card in a reverse-order sequence for the second pipeline.

[0165] In some examples, the allocation module is further configured to, in response to the current allocation phase where all chips of the multi-chip accelerator card have been allocated, proceed to the next chip allocation phase in the multi-chip accelerator card.

[0166] In some examples, multiple multi-core accelerator cards include M multi-core accelerator cards, and each multi-core accelerator card includes L cores. , and \(0\leq i < N\), \(0\leq j < M\), \(0\leq k < L\), where \(i\), \(j\), and \(k\) are non-negative integers, and the allocation module is further configured to allocate the \(S\) corresponding to the first pipeline i and the \(S\) corresponding to the second pipeline N-i-1 to \(D\) j,k where , , where \(S\) i represents the \((i + 1)\)-th stage in \(N\) consecutive stages, \(S\) N-i-1 represents the \((N - i)\)-th stage in \(N\) consecutive stages, \(D\) j,k represents the \((k + 1)\)-th die in the \((j + 1)\)-th multi-die acceleration card, \(\lfloor\)\( \rfloor\) represents floor division, and % represents modulo operation.

[0167] In some examples, the training task includes a forward propagation calculation task and a backward propagation calculation task, and the task scheduling device further includes a determination module. The determination module is configured to, during at least part of the training process, for at least one of the multiple multi-die acceleration cards, perform the following operations: determine that when the first die in the multi-die acceleration card performs the forward propagation calculation task, the second die in the multi-die acceleration card performs the backward propagation calculation task; or determine that when the first die in the multi-die acceleration card performs the backward propagation calculation task, the second die in the multi-die acceleration card performs the forward propagation calculation task, so that the forward propagation calculation task and the backward propagation calculation task respectively performed by the first die and the second die at least partially overlap in time.

[0168] In some examples, the determination module is further configured to, for at least one of the multiple multi-die acceleration cards, perform the following operations: determine that the activation value obtained by the first die in the multi-die acceleration card performing the forward propagation calculation task is transmitted through the inter-die interconnection link and stored in the second die in the multi-die acceleration card, where the stage corresponding to the forward propagation calculation task of the second die is after the stage corresponding to the forward propagation calculation task of the first die in the forward stage arrangement order.

[0169] In some examples, data is transmitted between multiple multi-die acceleration cards through a direct connection link, and data is transmitted between multiple dies through an inter-die interconnection link.

[0170] In some examples, the multi-die acceleration card is a dual-die acceleration card, and the dual-die acceleration card includes two dies integrated in a single package.

[0171] It should be noted that the above various modules can be implemented by software, hardware, firmware, or any combination thereof. For example, the allocation module can be implemented as an allocation circuit, and the division module can be implemented as a division circuit. The embodiments of the present disclosure do not limit their specific implementation manners.

[0172] It should be understood that the task scheduling apparatus provided in at least one embodiment of this disclosure can be used to implement the aforementioned task scheduling method and can also achieve similar technical effects as the aforementioned task scheduling method, which will not be elaborated here.

[0173] It should be noted that, in the embodiments of this disclosure, the task scheduling device may include more or fewer modules or units, and the connection relationship between the various modules or units is not limited and can be determined according to actual needs. The specific configuration of each module or unit is not limited; it can be constructed from analog devices according to circuit principles, or from digital chips, or in other suitable ways.

[0174] Figure 6 This is a schematic block diagram of a training system provided for at least one embodiment of the present disclosure.

[0175] For example, such as Figure 6 As shown, the training system 600 provided in at least one embodiment of this disclosure includes a plurality of multi-chip accelerator cards 601, each multi-chip accelerator card 601 including a plurality of chips 602 integrated in a single package. Each chip of at least one of the plurality of multi-chip accelerator cards is configured to: execute corresponding training tasks of a first pipeline and a second pipeline according to the two stages carried by the chip, respectively, to train the model to be trained.

[0176] The model to be trained is divided into multiple stages, each stage including at least one layer of the model to be trained. The first pipeline and the second pipeline are mapped to the cores of multiple multi-core accelerator cards. The first pipeline and the second pipeline are built based on multiple stages. The data flow of the first pipeline and the second pipeline is opposite and used to process different training batches.

[0177] In some examples, during at least part of the training process, when the first chip in the multi-chip accelerator card performs a forward propagation computation task, the second chip in the multi-chip accelerator card is configured to perform a backward propagation computation task, such that the forward propagation computation task and the backward propagation computation task performed by the first chip and the second chip, respectively, overlap at least partially in time, wherein the forward propagation task and the backward propagation task belong to the first pipeline and the second pipeline, respectively.

[0178] In some examples, during at least part of the training process, when the first chip in the multi-chip accelerator card performs a backpropagation computation task, the second chip in the multi-chip accelerator card is configured to perform a forward propagation computation task, such that the forward propagation computation task and the backpropagation computation task performed by the first chip and the second chip, respectively, overlap at least partially in time, wherein the forward propagation task and the backpropagation task belong to the first pipeline and the second pipeline, respectively.

[0179] In some examples, for at least one of a plurality of multi-core accelerator cards, the first core of the multi-core accelerator card is configured to transmit the activation value obtained from performing a forward propagation computation task to a second core of the multi-core accelerator card for storage via an inter-core interconnect link, wherein the stage corresponding to the forward propagation computation task of the second core is arranged in a forward order after the stage corresponding to the forward propagation computation task of the first core.

[0180] For other related information about this training system, please refer to the relevant introduction in the task scheduling method section above, which will not be repeated here.

[0181] Figure 7 This is a schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0182] For example, such as Figure 7 As shown, the electronic device 700 includes at least one processor 701 and at least one memory 702. The at least one memory 702 includes one or more computer program modules. These computer program modules are stored in the memory 702 and configured to be executed by the at least one processor 701. The one or more computer program modules include instructions for performing the task scheduling method or training method described above. When executed by the at least one processor 701, they can perform one or more steps of the task scheduling method or training method provided in at least one embodiment of this disclosure. The memory 702 and the processor 701 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0183] For example, processor 701 can be a central processing unit (CPU), digital signal processor (DSP), graphics processing unit (GPU), general-purpose graphics processing unit (GPGPU), artificial intelligence (AI) accelerator, or other form of processing unit with data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA); for example, the central processing unit (CPU) can be an x86, ARM, or RISC-V architecture. Processor 701 can be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 700 to perform desired functions.

[0184] For example, memory 702 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0185] Figure 8 This is a schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0186] The electronic devices in at least one embodiment of this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0187] The electronic device includes at least one processor and a memory. The processor may be referred to as processing device 801 as described below, and the memory may include at least one of ROM 802, RAM 803, and storage device 808 as described below. The memory is used to store programs for performing the methods described in the various method embodiments above; the processor is configured to execute the programs stored in the memory. The processor may include a central processing unit (CPU) or other forms of processing unit having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0188] like Figure 8 As shown, the electronic device 800 may include a processing unit 801 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in ROM 802 or a program loaded from storage device 808 into RAM 803. RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interfaces are also connected to bus 804.

[0189] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, displays, speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0190] In particular, according to at least one embodiment of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, at least one embodiment of this disclosure includes a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of at least one embodiment of this disclosure.

[0191] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In at least one embodiment of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In at least one embodiment of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, radio frequency (RF), etc., or any suitable combination thereof.

[0192] The aforementioned computer-readable medium may be included in the aforementioned electronic device 800; or it may exist independently and not assembled into the electronic device 800.

[0193] Figure 9 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure.

[0194] For example, such as Figure 9 As shown, a non-transitory computer-readable storage medium 900 stores computer-readable instructions 901, which, when executed by at least one processor, perform one or more steps of the task scheduling method or training method described above.

[0195] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet computer, a hard drive for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media. For example, the readable storage medium may also be... Figure 7 The memory 702 in the memory is described in the foregoing content and will not be repeated here.

[0196] Although the present disclosure has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to the embodiments of the present disclosure, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present disclosure are within the scope of protection claimed by the present disclosure.

[0197] The following points should be noted regarding this disclosure:

[0198] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0199] (2) For clarity, the thickness of layers or regions in the drawings used to describe embodiments of the present disclosure is enlarged or reduced, i.e., these drawings are not drawn to actual scale.

[0200] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0201] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.

Claims

1. A task scheduling method, characterized by, The method is applied to a training system comprising multiple multi-core accelerator cards, wherein the multi-core accelerator cards include multiple cores integrated in a single package, wherein the method includes: The model to be trained is divided into multiple stages, and a first pipeline and a second pipeline with opposite data flow directions are constructed based on the multiple stages to process different training batches, wherein each stage includes at least one layer of the model to be trained. The first pipeline and the second pipeline are mapped to the chips of the plurality of multi-chip accelerator cards, such that each chip in at least one multi-chip accelerator card carries two stages, the two stages being used to execute the training tasks of the first pipeline and the second pipeline, respectively, wherein the training tasks include forward propagation computation tasks and backward propagation computation tasks.

2. The method of claim 1, wherein, The process of dividing the model to be trained into multiple stages and constructing a first pipeline and a second pipeline with opposite data flow directions for processing different training batches based on the multiple stages includes: The model to be trained is divided into N consecutive stages, and the first pipeline and the second pipeline are constructed based on the N consecutive stages respectively. Wherein, N is a positive integer, the first pipeline includes the N consecutive stages and schedules data between execution units in a first direction, the second pipeline includes the N consecutive stages and schedules data between execution units in a second direction opposite to the first direction, and the execution unit includes a core.

3. The method of claim 1, wherein, The step of mapping the first pipeline and the second pipeline to the dies of the plurality of multi-die accelerator cards includes: For at least one of the plurality of multi-core accelerator cards, perform the following operations: According to the stage arrangement order corresponding to the first and second pipelines, stages are sequentially assigned to each chip of the multi-chip accelerator card.

4. The method of claim 3, wherein, The step of sequentially assigning stages to each chip of the multi-chip accelerator card according to the stage arrangement order corresponding to the first pipeline and the second pipeline includes: For the first pipeline, stages are sequentially assigned to each chip of the multi-chip accelerator card according to the forward arrangement of stages; For the second production line, stages are sequentially assigned to each chip of the multi-chip accelerator card according to the reverse stage arrangement order.

5. The method according to claim 4, characterized in that, The step of mapping the first pipeline and the second pipeline to the dies of the plurality of multi-die accelerator cards further includes: In response to the current allocation phase, all chips of the multi-chip accelerator card have been allocated, which marks the next chip allocation phase in the multi-chip accelerator card.

6. The method according to claim 2, characterized in that, The multiple multi-die acceleration cards include M multi-die acceleration cards, and each multi-die acceleration card includes L dies. , and 0 ≤ i < N, 0 ≤ j < M, 0 ≤ k < L, where i, j, and k are non-negative integers. The step of mapping the first pipeline and the second pipeline to the chips of the plurality of multi-chip accelerator cards includes: S corresponding to the first production line i S corresponding to the second production line N-i-1 Assigned to D j,k Among them , , Among them, S i S represents the (i+1)th stage in the N consecutive stages. N-i-1 D represents the Ni-th stage in the N consecutive stages. j,k This represents the (k+1)th core in the (j+1)th multi-core accelerator card. % indicates rounding down, and % indicates taking the remainder.

7. The method according to claim 1, characterized in that, The method further includes: During at least part of the training process, for at least one of the plurality of multi-core accelerator cards, the following operations are performed: When the forward propagation computation task is performed by the first core in the multi-core accelerator card, the backward propagation computation task is performed by the second core in the multi-core accelerator card; or When the backpropagation computation task is performed by the first core of the multi-core accelerator card, the forward propagation computation task is performed by the second core of the multi-core accelerator card. This ensures that the forward propagation computation task and the backward propagation computation task performed by the first core and the second core, respectively, overlap at least partially in time.

8. The method according to claim 1, characterized in that, The method further includes: For at least one of the plurality of multi-core accelerator cards, perform the following operations: The activation value obtained by the first core in the multi-core accelerator card from the forward propagation computation task is transmitted to the second core in the multi-core accelerator card for storage through the inter-core interconnect link. The stage corresponding to the forward propagation computation task of the second core is arranged in a forward order after the stage corresponding to the forward propagation computation task of the first core.

9. The method according to claim 1, characterized in that, The multiple multi-core accelerator cards transmit data via direct links, and the multiple cores transmit data via inter-core interconnect links.

10. The method according to claim 1, characterized in that, The multi-core accelerator card is a dual-core accelerator card, which includes two cores integrated in a single package.

11. A training method, characterized in that, The method is applied to a training system comprising multiple multi-core accelerator cards, the multi-core accelerator cards comprising multiple cores integrated in a single package, the method comprising: The following operations are performed by each core of at least one of the plurality of multi-core accelerator cards: The training tasks of the first pipeline and the second pipeline are executed according to the two stages carried by the core to train the model to be trained. The training tasks include forward propagation computation tasks and backpropagation computation tasks. The model to be trained is divided into multiple stages, each stage including at least one layer of the model to be trained. The first pipeline and the second pipeline are mapped to the chips of the multiple multi-chip accelerator cards. The first pipeline and the second pipeline are constructed based on the multiple stages. The data flow of the first pipeline and the second pipeline is opposite and used to process different training batches.

12. The method according to claim 11, characterized in that, The step of executing the corresponding training tasks of the first pipeline and the second pipeline according to the two stages carried by the chip includes: During at least part of the training process, when the first core in the multi-core accelerator card performs the forward propagation computation task, the second core in the multi-core accelerator card performs the backward propagation computation task; or During at least part of the training process, when the first core in the multi-core accelerator card performs the backpropagation computation task, the second core in the multi-core accelerator card performs the forward propagation computation task. This ensures that the forward propagation computation task and the backward propagation computation task performed by the first core and the second core, respectively, overlap at least partially in time.

13. The method according to claim 11, characterized in that, The method further includes: For at least one of the plurality of multi-core accelerator cards The activation value obtained by the first core in the multi-core accelerator card from the forward propagation computation task is transmitted to the second core in the multi-core accelerator card for storage through the core interconnection link. The stage corresponding to the forward propagation computation task of the second core is arranged in a forward order after the stage corresponding to the forward propagation computation task of the first core.

14. The method according to claim 11, characterized in that, The multiple multi-core accelerator cards transmit data via direct links, and the multiple cores transmit data via inter-core interconnect links.

15. A task scheduling device, characterized in that, The device is used in a training system comprising multiple multi-core accelerator cards, the multi-core accelerator cards comprising multiple cores integrated in a single package, wherein the device includes: The partitioning module is configured to divide the model to be trained into multiple stages, and to construct a first pipeline and a second pipeline with opposite data flow directions and used to process different training batches based on the multiple stages, wherein each stage includes at least one layer of the model to be trained. The allocation module is configured to map the first pipeline and the second pipeline to the chips of the plurality of multi-chip accelerator cards, such that each chip in at least one multi-chip accelerator card carries two stages, the two stages being used to execute the training tasks of the first pipeline and the second pipeline, respectively, wherein the training tasks include forward propagation computation tasks and backward propagation computation tasks.

16. A training system, characterized in that, The system includes multiple multi-core accelerator cards, each comprising multiple cores integrated in a single package. Wherein, each core of at least one of the plurality of multi-core accelerator cards is configured as follows: The training tasks of the first pipeline and the second pipeline are executed according to the two stages carried by the core to train the model to be trained. The training tasks include forward propagation computation tasks and backpropagation computation tasks. The model to be trained is divided into multiple stages, each stage including at least one layer of the model to be trained. The first pipeline and the second pipeline are mapped to the chips of the multiple multi-chip accelerator cards. The first pipeline and the second pipeline are constructed based on the multiple stages. The data flow of the first pipeline and the second pipeline is opposite and used to process different training batches.

17. An electronic device, characterized in that, The electronic device includes: At least one processor; At least one memory, including one or more computer program modules; The one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the method according to any one of claims 1-14.

18. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-readable instructions, wherein the computer-readable instructions, when executed by at least one processor, perform the method according to any one of claims 1-14.

Images