Model parallel method and device for multi-machine multi-card scene, and electronic equipment

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a model parallelization method and device for a multi-machine multi-card scene, and electronic equipment, and the method comprises the steps: obtaining basic performance data of pipeline operation for model training; wherein the basic performance data comprises performance load information of a plurality of pipeline stages; obtaining a scheduling sequence of each assembly line stage according to the basic performance data of the assembly line operation; and according to the scheduling sequence of each assembly line stage, sequentially simulating the execution process of assembly line operation, and obtaining a simulation result of model parallel training. According to the method, a user can obtain performance evaluation highly close to real execution in a strategy selection stage, the problems that a traditional experiment is large in computing power requirement and high in cost are effectively solved, and the problem that small-scale deduction cannot accurately pre-judge large-scale scene performance is avoided.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, such as a model parallel method and apparatus, and electronic equipment for multi-machine, multi-card scenarios. Background Technology

[0002] With the rapid development of artificial intelligence technology, the number of parameters in deep learning models, especially large-scale language models based on the Transformer architecture, has grown exponentially, rapidly moving from billions to trillions. This dramatic expansion in scale has brought about enormous demands on computing power and GPU memory, making the single-machine, single-GPU training mode no longer sufficient to meet the training tasks of existing large models.

[0003] To address the computational bottleneck, related technologies have disclosed parallel training techniques, including data parallelism, tensor parallelism, and pipelined parallelism. Among them, pipelined parallelism divides the massive model network layers into multiple stages and distributes these stages across different computing devices (such as GPUs), with intermediate activation values ​​and gradients being passed between devices through point-to-point communication.

[0004] In the process of implementing the embodiments of this disclosure, at least the following problems were found in the related art: In hybrid parallel architectures, different model structures and hardware environments (such as communication bandwidth and computing power) correspond to different optimal hybrid parallel strategies. Currently, there is a lack of efficient and low-cost mechanisms to evaluate and select these strategies. Researchers and engineers often have to rely on rules of thumb or conduct small-scale simulations, making it difficult to accurately predict real-world performance on large-scale clusters.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0007] This disclosure provides a model parallelism method, apparatus, and electronic device for multi-machine, multi-card scenarios, which can improve the authenticity of performance evaluations obtained during the strategy selection phase.

[0008] In some embodiments, the model parallelism method for multi-machine, multi-GPU scenarios includes: acquiring basic performance data of pipeline operations for model training; wherein the basic performance data includes performance load information of multiple pipeline stages; acquiring a scheduling sequence for each pipeline stage based on the basic performance data of the pipeline operations; and simulating the execution process of the pipeline operations sequentially based on the scheduling sequence of each pipeline stage to obtain simulation results of model parallel training.

[0009] Optionally, basic performance data of pipeline operations used for model training is obtained, including: dividing the model to be trained into multiple pipeline stages according to the pipeline parallelism; obtaining performance load information of typical subgraphs constituting each pipeline stage; and mapping multiple pipeline stages and typical subgraphs of multiple pipeline stages to obtain performance load information of each pipeline stage and using it as basic performance data of pipeline operations.

[0010] Optionally, based on the basic performance data of the pipeline operation, the scheduling sequence for each pipeline stage is obtained, including: generating a spatiotemporal diagram based on the pipeline layout algorithm and the basic performance data; and obtaining the corresponding scheduling sequence based on the spatiotemporal diagram.

[0011] Optionally, a spatiotemporal graph is generated based on the pipeline layout algorithm and basic performance data, including: generating the spatiotemporal graph using the pipeline layout algorithm based on the number of micro-batches and pipeline parallelism obtained from the basic performance data.

[0012] Optionally, based on the spatiotemporal diagram, the corresponding scheduling sequence is obtained, including: setting communication timing during the calculation of the spatiotemporal diagram; and generating a complete scheduling scheme based on the spatiotemporal diagram and communication timing as the scheduling sequence for each pipeline stage.

[0013] Optionally, the principles for setting the communication timing during the calculation of the spatiotemporal graph include: sending and receiving tensors during the interval between the completion of the calculation.

[0014] Optionally, the principle for setting the timing of communication during the calculation of the spatiotemporal diagram also includes: when conducting data communication, data is first transmitted from the previous stage to the next stage, and then transmitted in reverse.

[0015] Optionally, based on the scheduling sequence of each pipeline stage, the execution process of the pipeline operation is simulated sequentially to obtain the simulation results of parallel model training, including: putting the initial event into a global queue to perform event queue initialization operation; retrieving the event with the earliest timestamp from the queue for processing, and checking whether to unlock new dependent operations after processing; encapsulating the new dependent operation into a new event and inserting it into the event queue for processing; and obtaining the simulation results of parallel model training after all events in the event queue have been processed.

[0016] In some embodiments, the model parallelization apparatus for multi-machine, multi-GPU scenarios includes a processor and a memory storing program instructions, the processor being configured to execute the model parallelization method for multi-machine, multi-GPU scenarios as described above when running the program instructions.

[0017] In some embodiments, the electronic device includes: an electronic device body; and a model parallelization device for multi-machine, multi-card scenarios, as described above, installed on the electronic device body.

[0018] The model parallelization method, apparatus, and electronic device for multi-machine, multi-card scenarios provided in this disclosure can achieve the following technical effects: In this embodiment, a closed-loop process is implemented to obtain simulation results by acquiring basic performance data, generating scheduling sequences, and finally simulating the execution to achieve accurate simulation of parallel training of multi-machine, multi-GPU models. First, basic performance data containing performance loads across multiple pipeline stages is extracted to provide a quantitative basis for simulation. Next, combined with pipeline parallelism strategies and parameters, the performance data is transformed into executable scheduling sequences for each stage, clarifying the execution logic of computation and communication. Finally, the entire training iteration process is simulated according to the scheduling sequences, outputting performance results that closely resemble real-world scenarios. This method does not rely on a large-scale cluster of GPUs; users can obtain performance evaluations highly similar to real-world execution during the strategy selection phase. It effectively solves the problems of high computational power requirements and costs associated with traditional experiments, avoids the inability of small-scale simulations to accurately predict the performance of large-scale scenarios, significantly reduces physical experiment costs, and shortens the development and optimization cycle of large-scale model training systems, providing efficient support for the scientific selection of hybrid parallelism strategies.

[0019] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0020] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a flowchart of the pipeline parallel simulation system provided in the embodiments of this disclosure; Figure 2 This is a schematic diagram of a model parallelization method for multi-machine, multi-card scenarios provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of another model parallel method for multi-machine, multi-card scenarios provided in this disclosure embodiment; Figure 4 This is a schematic diagram of another model parallel method for multi-machine, multi-card scenarios provided in this disclosure embodiment; Figure 5 This is a schematic diagram of another model parallel method for multi-machine, multi-card scenarios provided in this disclosure embodiment; Figure 6 This is a partial sample image of the simulation-generated visualization of the spatiotemporal diagram provided in the embodiments of this disclosure; Figure 7 This is a schematic diagram of a model parallel device for multi-machine, multi-card scenarios provided in an embodiment of this disclosure. Detailed Implementation

[0021] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0022] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0023] Unless otherwise stated, the term "multiple" means two or more.

[0024] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0025] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0026] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0027] With the rapid development of artificial intelligence technology, the number of parameters in deep learning models, especially large-scale language models based on the Transformer architecture, has grown exponentially, rapidly moving from billions to trillions. This dramatic expansion in scale has brought about enormous demands on computing power and GPU memory, making the single-machine, single-GPU training mode no longer sufficient to meet the training tasks of existing large models.

[0028] To address the bottleneck of computing power, parallel training techniques have emerged and become the standard paradigm for training large models. Current parallel training techniques mainly include data parallelism, tensor parallelism, and pipeline parallelism.

[0029] Pipeline parallelism divides the massive model network layers into multiple stages and distributes these stages across different computing devices (such as GPUs). These devices communicate point-to-point to exchange intermediate activation values ​​and gradients. This approach effectively addresses the problem of insufficient GPU memory on a single device to hold all model parameters and improves device utilization.

[0030] In practical industrial-grade large-scale model training scenarios, a hybrid parallel strategy is typically adopted, which combines data parallelism, tensor parallelism, and pipeline parallelism (i.e., 3D parallelism). However, in a hybrid parallel architecture, the selection of appropriate parallel configuration strategies (e.g., the number of pipeline stages, the size of micro-batches, the number of gradient accumulation steps, and the communication topology between different pipelines) has a decisive impact on training efficiency.

[0031] Different model architectures and hardware environments (such as communication bandwidth and computing power) correspond to different optimal hybrid parallelism strategies. Currently, there is a lack of an efficient and low-cost mechanism in existing technologies to evaluate and select these strategies. Researchers and engineers often have to rely on rules of thumb or conduct small-scale simulations, making it difficult to accurately predict real-world performance on large-scale clusters.

[0032] The embodiments disclosed herein can solve the following core problems existing in the prior art during the training and strategy verification stages of multi-machine, multi-GPU large model training experiments: Multi-machine, multi-GPU computing resources are required in large quantities, and the experimental threshold is high. Full training or performance testing of large models in real-world environments typically requires clusters of hundreds or thousands of high-performance accelerator cards (such as GPUs). This massive hardware requirement not only incurs high financial costs but also makes it difficult for ordinary research institutions or development teams to acquire sufficient physical resources to verify the effectiveness of their parallel strategies.

[0033] Complete testing and performance prediction are difficult to conduct during the experimental phase. Due to the scarcity of hardware resources, developers often can only conduct scaling experiments on small-scale clusters in an attempt to infer the performance of large-scale clusters. However, due to the complex inter-device communication, pipeline bubbles, and resource contention issues involved in pipeline parallelism, data from small-scale experiments often cannot be linearly extrapolated to large-scale scenarios, leading to policy selection bias and potentially causing resource waste or inefficiency during real-world training.

[0034] There is a lack of accurate simulation methods for the pipeline dimension. Existing simulation tools mostly focus on operator-level computation time simulation or simple network topology simulation, lacking system-level simulation for the parallel characteristics of pipelines. Specifically, existing technologies struggle to accurately simulate the interaction behavior, dependencies, and communication latency of multiple pipelines in a multi-machine, multi-GPU environment on limited physical resources (such as a single machine or a small number of machines).

[0035] like Figure 1 As shown, Figure 1 This is a flowchart of a pipelined parallel simulation system. This embodiment of the disclosure predicts the performance of a single iteration of parallel model training in a multi-machine, multi-GPU scenario through pipelined simulation. First, based on the model structure and parallel configuration, a pipelined operation load is generated from existing performance data within each stage. Then, a pipelined scheduling sequence is generated based on the selected pipelined parallel strategy and parameters. Finally, the pipelined load is simulated and executed according to the scheduling sequence to obtain performance simulation results such as the pipeline spatiotemporal graph and iteration time.

[0036] Through the embodiments disclosed herein, users can obtain performance evaluations that are close to actual execution during the strategy selection phase, thereby significantly reducing the cost of physical experiments and accelerating the development and optimization cycle of large model training systems.

[0037] Combination Figure 2 As shown, this disclosure provides a model parallelism method for multi-machine, multi-card scenarios, including: S201, Obtain basic performance data for pipeline operations used for model training; wherein, the basic performance data includes performance load information for multiple pipeline stages.

[0038] S202, based on the basic performance data of the pipeline operation, obtain the scheduling sequence for each pipeline stage.

[0039] S203, based on the scheduling sequence of each pipeline stage, simulates the execution process of the pipeline operation in sequence to obtain the simulation results of parallel model training.

[0040] This disclosure presents a model parallelism method for multi-machine, multi-GPU scenarios. This method achieves accurate simulation of multi-machine, multi-GPU model parallel training by acquiring basic performance data, generating scheduling sequences, and finally simulating execution to obtain simulation results—a closed-loop process. First, basic performance data containing performance loads across multiple pipeline stages is extracted to provide a quantitative basis for simulation. Next, combined with pipeline parallelism strategies and parameters, the performance data is transformed into executable scheduling sequences for each stage, clarifying the execution logic of computation and communication. Finally, the entire training iteration process is simulated according to the scheduling sequences, outputting performance results closely resembling real-world scenarios. This method does not rely on large-scale clusters of GPUs. Users can obtain performance evaluations highly similar to real-world execution during the strategy selection phase, effectively solving the problems of high computational power requirements and costs associated with traditional experiments. It avoids the inability of small-scale simulations to accurately predict the performance of large-scale scenarios, significantly reducing physical experiment costs and shortening the development and optimization cycle of large-scale model training systems, providing efficient support for the scientific selection of hybrid parallelism strategies.

[0041] Optionally, basic performance data of pipeline operations used for model training is obtained, including: dividing the model to be trained into multiple pipeline stages according to the pipeline parallelism; obtaining performance load information of typical subgraphs constituting each pipeline stage; and mapping multiple pipeline stages and typical subgraphs of multiple pipeline stages to obtain performance load information of each pipeline stage and using it as basic performance data of pipeline operations.

[0042] Combination Figure 3 As shown, this disclosure provides another model parallelism method for multi-machine, multi-card scenarios, including: S301 divides the model to be trained into multiple pipeline stages based on pipeline parallelism.

[0043] S302, obtain performance load information of a typical subgraph that constitutes each pipeline stage.

[0044] S303 maps multiple pipeline stages and typical subgraphs of multiple pipeline stages to obtain the performance load information of each pipeline stage and uses it as the basic performance data of the pipeline operation.

[0045] S304: Based on the basic performance data of the pipeline operation, obtain the scheduling sequence for each pipeline stage.

[0046] S305, based on the scheduling sequence of each pipeline stage, simulates the execution process of pipeline operations sequentially to obtain the simulation results of parallel model training.

[0047] In this embodiment, the model to be trained is first vertically divided according to the pipeline parallelism to form multiple pipeline stages, including regular stages (isomorphic models containing the same number of layers) and special stages (including embedding layers, output layers, or dense layers in hybrid expert models), to ensure that the model division fits the actual training scenario. Then, through small-scale experimental sampling or in-stage simulation systems, the performance load information of typical subgraphs (such as one or more Transformer Blocks) constituting each stage is obtained. This information contains key data such as single-card single-micro-batch forward and backward computation time, activation value size, and communication time between adjacent stages in the form of feature tuples. Finally, each pipeline stage is associated and matched with the corresponding typical subgraph to obtain standardized performance load information of each stage, which serves as the basic performance data of the pipeline operation. This approach transforms abstract model structures into concrete, quantifiable load data, ensuring that load information accurately reflects computational and communication overhead at different stages. This provides reliable data support for subsequent simulations, making simulation results closer to real training conditions. Consequently, it helps users obtain accurate performance references during the strategy selection phase, eliminating the need for real large-scale cluster testing. This significantly reduces the financial and resource costs of physical experiments and accelerates the development and optimization of large-scale model training systems.

[0048] Optionally, based on the basic performance data of the pipeline operation, the scheduling sequence for each pipeline stage is obtained, including: generating a spatiotemporal diagram based on the pipeline layout algorithm and the basic performance data; and obtaining the corresponding scheduling sequence based on the spatiotemporal diagram.

[0049] Combination Figure 4 As shown, this disclosure provides another model parallelism method for multi-machine, multi-card scenarios, including: S401, Obtain basic performance data for pipeline operations used for model training; wherein, the basic performance data includes performance load information for multiple pipeline stages.

[0050] S402 generates a spatiotemporal diagram based on the pipeline layout algorithm and basic performance data.

[0051] S403, based on the spatiotemporal diagram, obtain the corresponding scheduling sequence.

[0052] S404 simulates the execution process of pipeline operations sequentially according to the scheduling sequence of each pipeline stage, and obtains the simulation results of parallel model training.

[0053] In this embodiment, a spatiotemporal diagram containing only computational logic is first generated by combining specific pipeline deployment algorithms (such as classic algorithms or custom algorithms like 1F1B, Interleaved 1F1B, DualPipe, etc.) with the computational and communication load characteristics of each stage contained in the basic performance data, thus clarifying the allocation of computational tasks for each device at different times. Then, based on the spatiotemporal diagram, the communication logic design is supplemented to reasonably integrate computation and communication operations, forming a complete scheduling sequence, clarifying when each stage performs computation and when it transmits data with other stages.

[0054] This disclosure realizes the transformation of parallel strategies into executable logic. By constructing standardized scheduling sequences, it ensures that the simulation process can accurately reproduce the core interaction logic of pipeline parallelism. It effectively solves the problem that the existing technology lacks system-level simulation for pipeline dimensions, allowing users to predict the training performance under different strategies through the simulation execution of scheduling sequences during the strategy verification stage. This avoids resource waste or inefficiency caused by inappropriate strategies after real deployment, significantly reduces the cost of physical experiments, and accelerates the development and optimization cycle of large model training systems.

[0055] Optionally, a spatiotemporal graph is generated based on the pipeline layout algorithm and basic performance data, including: generating the spatiotemporal graph using the pipeline layout algorithm based on the number of micro-batches and pipeline parallelism obtained from the basic performance data.

[0056] In this embodiment, when generating the spatiotemporal graph, the number of micro-batches and pipeline parallelism, explicitly defined in the basic performance data, are used as core parameters, and a selected pipeline layout algorithm is used for targeted construction. The number of micro-batches determines the granularity of the training data split, and the pipeline parallelism defines the number of stages in the model segmentation; together, they constitute the key constraints for generating the spatiotemporal graph. The algorithm allocates the computational tasks of the micro-batches sequentially according to stages, clarifying the micro-batch computational tasks corresponding to each stage within different time windows, thus forming a spatiotemporal graph that intuitively reflects the computational timing and stage allocation.

[0057] The embodiments disclosed herein ensure that the spatiotemporal graph can accurately match specific parallel configuration parameters, accurately depict the computational distribution characteristics under different parameter combinations, and provide a scientific basis for the generation of subsequent scheduling sequences. The scheduling sequences generated based on this spatiotemporal graph can more accurately simulate the training process under different parameter configurations, allowing users to clearly predict the impact of key parameters such as micro-batch size and parallelism on training performance during the strategy selection stage, without the need for repeated testing through real large-scale experiments.

[0058] Optionally, based on the spatiotemporal diagram, the corresponding scheduling sequence is obtained, including: setting communication timing during the calculation of the spatiotemporal diagram; and generating a complete scheduling scheme based on the spatiotemporal diagram and communication timing as the scheduling sequence for each pipeline stage.

[0059] In this embodiment, the spatiotemporal graph only defines the execution sequence of computation tasks. To avoid communication and computation conflicts or deadlocks, communication operations need to be reasonably inserted during the computation process: first, the compatibility window between communication and computation is defined, and then the communication timing is precisely embedded into the computation logic of the spatiotemporal graph to ensure that the communication operation does not affect the continuity of the computation process; then, combined with the computation sequence of the spatiotemporal graph and the set communication timing, a complete scheduling scheme including computation operations (forward propagation, backward propagation) and communication operations (sending tensors, receiving tensors) is constructed to form the scheduling sequence of each pipeline stage.

[0060] This disclosure addresses the issues of communication and computation conflicts and deadlocks in traditional scheduling, ensuring the feasibility and efficiency of the scheduling sequence. It enables the simulation process to accurately reproduce the pipelined parallel computation and communication interaction behavior in a multi-machine, multi-GPU environment. Based on the simulation results of this scheduling sequence, users can accurately evaluate the impact of different communication strategies on training efficiency, obtaining near-real-world performance feedback during the strategy selection phase without the need for testing on a real cluster.

[0061] Optionally, the principles for setting the communication timing during the calculation of the spatiotemporal graph include: sending and receiving tensors during the interval between the completion of the calculation.

[0062] In this embodiment, when setting communication timing during spatiotemporal graph calculation, the principle of sending and receiving data during the intervals between calculation executions is followed. This avoids the execution periods of computational operations, scheduling data transmission operations within the idle windows of adjacent computational tasks. This approach ensures that communication operations do not consume computational resources, avoiding resource contention between computation and communication. It guarantees continuous execution of computational tasks while fully utilizing the device's idle time to complete data transmission, thus improving overall device utilization. Simultaneously, this principle effectively reduces pipeline idle time (bubbles), making the simulation process closer to the actual operating state of the device during training, and making the performance data of the simulation results (such as iteration time and device utilization) more valuable. Users can accurately predict the impact of communication overhead on training performance in different scenarios by using the scheduling sequence generated based on this communication timing setting. The optimal communication efficiency can be selected during the strategy selection phase without requiring large-scale real-world experimental verification, significantly reducing physical experiment costs and accelerating the development and optimization cycle of large-scale model training systems.

[0063] Optionally, the principle for setting the timing of communication during the calculation of the spatiotemporal diagram also includes: when conducting data communication, data is first transmitted from the previous stage to the next stage, and then transmitted in reverse.

[0064] In this embodiment of the present disclosure, during the communication process between each computation interval, the activation values ​​of the forward propagation are first transmitted from the previous stage to the next stage to ensure that the downstream stage can obtain the intermediate data required for computation in a timely manner and ensure the smooth progress of the forward computation process; after the forward transmission is completed, the gradient data of the backward propagation is transmitted to ensure that the upstream stage can receive the gradient information in a timely manner for parameter updates.

[0065] This design principle avoids path conflicts between forward and reverse transmissions, eliminates the risk of communication deadlocks caused by circular waiting, and optimizes the timing of data transmission, thereby improving the efficiency of the communication process. Communication timing based on this principle allows the scheduling sequence to more accurately simulate the communication interaction logic in a real multi-machine, multi-GPU environment, ensuring that simulation results accurately reflect training performance under different communication topologies. Users can clearly understand the impact of communication timing on training efficiency through simulation during the strategy selection phase, eliminating the need for costly testing on real clusters, significantly reducing physical experiment costs, and accelerating the development and optimization cycle of large-scale model training systems.

[0066] Optionally, based on the scheduling sequence of each pipeline stage, the execution process of the pipeline operation is simulated sequentially to obtain the simulation results of parallel model training, including: putting the initial event into a global queue to perform event queue initialization operation; retrieving the event with the earliest timestamp from the queue for processing, and checking whether to unlock new dependent operations after processing; encapsulating the new dependent operation into a new event and inserting it into the event queue for processing; and obtaining the simulation results of parallel model training after all events in the event queue have been processed.

[0067] Combination Figure 5 As shown, this disclosure provides another model parallelism method for multi-machine, multi-card scenarios, including: S501, Obtain basic performance data for pipeline operations used for model training; wherein, the basic performance data includes performance load information for multiple pipeline stages.

[0068] S502 obtains the scheduling sequence for each pipeline stage based on the basic performance data of the pipeline operation.

[0069] S503, the initial event is placed into the global queue to perform the event queue initialization operation.

[0070] S504 retrieves the event with the earliest timestamp from the queue for processing, and checks whether to unlock new dependency operations after processing is complete.

[0071] S505 encapsulates new dependency operations as new events and inserts them into the event queue for processing.

[0072] S506: After all events in the event queue have been processed, the simulation results of parallel model training are obtained.

[0073] In this embodiment of the disclosure, when simulating the execution process of the scheduling sequence, the initial operations without prior dependencies (usually the forward computation of the first micro-batch in the first pipeline stage) are first placed into a global queue, and the event queue is initialized by sorting them according to the expected start time. Then, the event with the earliest timestamp is taken out from the queue and executed. During the processing, the status (computation, communication, idle) of the corresponding computing device (such as GPU) is updated in real time, and it is checked whether new dependent operations are unlocked after the current event is completed. Then, the unlocked dependent operations are encapsulated as new events, their expected start and end times are calculated, and they are inserted into the corresponding positions in the queue to maintain the time order of the queue. When all events in the queue are processed and the computation and parameter updates of all micro-batches are completed, the simulation results containing data such as the total iteration time, pipeline spatiotemporal graph, bubble ratio, and GPU utilization are output.

[0074] Specifically, such as Figure 6 As shown, during the simulation, the state (computing, communication, idle) of each GPU at each point in time is accurately recorded and visualized to form a simulation pipeline spatiotemporal diagram, so as to observe the pipeline execution and performance bottlenecks. It can also be used to calculate performance statistics such as bubble ratio and GPU utilization.

[0075] This event-driven simulation method accurately records the state of each device at every point in time, fully reproducing the entire training iteration process, resulting in simulation results that are highly consistent with real execution. Based on these simulation results, users can comprehensively understand the performance of different parallel strategies during the strategy selection phase, without relying on real large-scale clusters for experiments. This significantly reduces the financial and time costs of physical experiments, while quickly selecting the optimal hybrid parallel strategy, accelerating the development and optimization cycle of large model training systems.

[0076] Combination Figure 7 As shown, this disclosure provides a model parallel processing device 70 for multi-machine, multi-card scenarios, including a processor 700 and a memory 701. Optionally, the device 70 may further include a communication interface 702 and a bus 703. The processor 700, communication interface 702, and memory 701 can communicate with each other via the bus 703. The communication interface 702 can be used for information transmission. The processor 700 can call logical instructions in the memory 701 to execute the model parallel processing method for multi-machine, multi-card scenarios described in the above embodiment.

[0077] Furthermore, the logic instructions in the aforementioned memory 701 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0078] The memory 701, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 700 executes functional applications and data processing by running the program instructions / modules stored in the memory 701, thereby implementing the model parallel method for multi-machine, multi-card scenarios in the above embodiments.

[0079] The memory 701 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 701 may include high-speed random access memory and may also include non-volatile memory.

[0080] This disclosure provides an electronic device, including: an electronic device body, and the aforementioned model parallelization device for multi-machine, multi-card scenarios. The model parallelization device for multi-machine, multi-card scenarios is mounted on the electronic device body. The mounting relationship described herein is not limited to placement inside the electronic device body, but also includes mounting connections with other components of the electronic device, including but not limited to physical connections, electrical connections, or signal transmission connections. Those skilled in the art will understand that the model parallelization device for multi-machine, multi-card scenarios can be adapted to feasible electronic device bodies to achieve other feasible embodiments.

[0081] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., and other media capable of storing program code.

[0082] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0083] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0084] The methods and products disclosed in the embodiments herein (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0085] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A model parallelism method for multi-machine, multi-card scenarios, characterized in that, include: Obtain basic performance data for pipeline operations used for model training; the basic performance data includes performance load information for multiple pipeline stages. Based on the basic performance data of the pipeline operation, obtain the scheduling sequence for each pipeline stage; Based on the scheduling sequence of each pipeline stage, the execution process of the pipeline operation is simulated sequentially to obtain the simulation results of parallel model training.

2. The method according to claim 1, characterized in that, Obtain basic performance data for the pipeline operations used for model training, including: Based on the pipeline parallelism, the model to be trained is divided into multiple pipeline stages; Obtain performance load information for a typical subgraph that constitutes each pipeline stage; By mapping multiple pipeline stages and typical subgraphs of multiple pipeline stages to obtain the performance load information of each pipeline stage, it can be used as the basic performance data of pipeline operation.

3. The method according to claim 1, characterized in that, Based on the basic performance data of the pipeline operation, obtain the scheduling sequence for each pipeline stage, including: Based on the pipeline layout algorithm and basic performance data, a spatiotemporal diagram is generated. Based on the spatiotemporal diagram, obtain the corresponding scheduling sequence.

4. The method according to claim 3, characterized in that, Based on the pipeline layout algorithm and basic performance data, a spatiotemporal diagram is generated, including: Based on the number of micro-batches and pipeline parallelism obtained from the basic performance data, a spatiotemporal graph is generated using a pipeline layout algorithm.

5. The method according to claim 3, characterized in that, Based on the spatiotemporal diagram, obtain the corresponding scheduling sequence, including: Establish communication opportunities during the calculation of the spatiotemporal diagram; A complete scheduling scheme is generated based on the spatiotemporal diagram and communication timing, serving as the scheduling sequence for each pipeline stage.

6. The method according to claim 5, characterized in that, The principles for setting communication timing during the calculation of the spatiotemporal diagram include: Tensors are sent and received during the interval between computation execution.

7. The method according to claim 5, characterized in that, The principles for setting communication timing during the calculation of the spatiotemporal diagram also include: In data communication, data is first transmitted from the previous stage to the next stage, and then transmitted in reverse.

8. The method according to any one of claims 1 to 7, characterized in that, Based on the scheduling sequence of each pipeline stage, the execution process of the pipeline operations is simulated sequentially to obtain the simulation results of parallel model training, including: The initial event is placed into the global queue to perform the event queue initialization operation; Retrieve the event with the earliest timestamp from the queue for processing, and check whether to unlock new dependency operations after processing is complete; The new dependency operation is encapsulated as a new event and inserted into the event queue for processing; After all events in the event queue have been processed, the simulation results of parallel model training are obtained.

9. A model parallelization device for multi-machine, multi-card scenarios, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to execute, when running the program instructions, the model parallel method for multi-machine, multi-card scenarios as described in any one of claims 1 to 8.

10. An electronic device, characterized in that, include: The electronic device itself; The model parallel device for multi-machine, multi-card scenarios as described in claim 9 is installed on the electronic device body.

Images