Parallel training method and apparatus for mixture-of-experts model

By generating and selecting suitable parallel strategies, combining the configuration and hardware information of the hybrid expert model, the load balancing problem of the hybrid expert model is solved, the training efficiency is improved, and the optimization of parallel training is achieved.

WO2025001026A1PCT designated stage expired Publication Date: 2025-01-02TSINGHUA UNIVERSITY

Patent Information

Application Number
PCT/CN2023/142843
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-30
Filing Date
2023-12-28
Publication Date
2025-01-02

AI Technical Summary

Technical Problem

The existing automated parallel training methods are mainly aimed at traditional models with dense architectures, and fail to effectively consider expert parallelism strategies and possible load balancing problems, resulting in inefficient training of hybrid expert models.

Method used

By generating multiple sets of candidate parallel policies, based on the configuration information and hardware information of the hybrid expert model, combined with the current load situation, the target parallel policy is determined, and compared with the historical parallel policy, select the better one for training to switch the parallel policy, Solve load balancing issues.

Benefits of technology

It effectively solves the load balancing problem, improves the parallel training efficiency of the hybrid expert model, ensures the load balancing of expert networks on each computing device, and avoids overload and idle situations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023142843_02012025_PF_FP_ABST
    Figure CN2023142843_02012025_PF_FP_ABST
Patent Text Reader

Abstract

A parallel training method and apparatus for a mixture-of-experts model, which method and apparatus belong to the technical field of deep learning. The parallel training method in one embodiment comprises: generating a plurality of candidate parallel policies according to configuration information and hardware information of a mixture-of-experts model; in the current training period, determining a target parallel policy from among the plurality of candidate parallel policies according to the current load condition of the mixture-of-experts model; determining a historical parallel policy which is used in the previous training period of the current training period; and training the mixture-of-experts model on the basis of the better one of the historical parallel policy and the target parallel policy. Provided in the technical solution of the present application is an automatic parallel training method for a mixture-of-experts model. Firstly, a parallel policy set is determined, and parallel policies are switched according to a load condition, such that the problem of load balancing can be effectively solved on the basis of introducing an expert policy, thereby increasing the parallel training efficiency of a mixture-of-experts model.
Need to check novelty before this filing date? Find Prior Art

Description

A parallel training method and device for hybrid expert model

[0001] Related applications

[0002] This application claims priority to the Chinese invention patent application with application number 202310789205.1 filed on June 30, 2023, and cites the entire contents disclosed in the above patent application as part of this application. Technical Field

[0003] The present application relates to the field of deep learning technology, and in particular to a parallel training method and apparatus for a hybrid expert model. Background Art

[0004] The Mixture of Experts (MoE) model is a type of neural network, but unlike typical neural networks, it can train multiple models based on the data separately. Each model is called an expert network. The idea behind the MoE is to train multiple expert networks, each applied to a different part of the dataset. As an emerging sparse activation deep learning model architecture, the MoE enables ultra-large-scale model training.

[0005] Existing automated parallel training mainly targets traditional models with dense architectures. On the one hand, it does not take into account expert parallel strategies, and on the other hand, it does not take into account the load balancing issues that expert parallel strategies may bring.

[0006] Summary of the Invention

[0007] In order to solve the above problems, the present application provides a parallel training method and device for a hybrid expert model, which can effectively solve the load balancing problem based on the introduction of expert strategies and improve the training efficiency of the hybrid expert model.

[0008] An embodiment of the present application provides a parallel training method for a hybrid expert model, the method comprising:

[0009] Generate multiple sets of candidate parallel strategies based on the configuration information and hardware information of the hybrid expert model;

[0010] In a current training cycle, determining a target parallel strategy from the multiple groups of candidate parallel strategies according to a current load condition of the hybrid expert model;

[0011] Determine the historical parallel strategy used in the previous training cycle of the current training cycle;

[0012] The hybrid expert model is trained based on a better one of the historical parallel strategy and the target parallel strategy.

[0013] Another embodiment of the present application provides a parallel training device for a hybrid expert model, the device comprising:

[0014] A candidate parallel strategy generation module is configured to generate multiple sets of candidate parallel strategies based on the configuration information of the hybrid expert model and the hardware information;

[0015] a target parallel strategy generating module configured to determine a target parallel strategy from the plurality of sets of candidate parallel strategies according to a current load of the hybrid expert model in a current training cycle;

[0016] a historical parallel strategy determination module configured to use the historical parallel strategy used in a training cycle previous to the current training cycle;

[0017] The parallel strategy switching module is configured to train the hybrid expert model based on a better one of the historical parallel strategy and the target parallel strategy.

[0018] Another embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the parallel training method for a hybrid expert model are implemented.

[0019] Yet another embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the parallel training method for a hybrid expert model.

[0020] Yet another embodiment of the present application provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the parallel training method for a hybrid expert model.

[0021] The beneficial effects of the present application are: generating multiple groups of candidate parallel strategies through the configuration information and hardware information of the hybrid expert model; and determining the target parallel strategy from the multiple groups of candidate parallel strategies according to the current load situation of the hybrid expert model in the current training cycle; further determining the historical parallel strategy used in the previous training cycle of the current training cycle; training the hybrid expert model based on the better one of the historical parallel strategy and the target parallel strategy, being able to switch the parallel strategy according to the load situation, effectively solving the load balancing problem on the basis of introducing the expert strategy, and improving the parallel training efficiency of the hybrid expert model.

[0022] Other features and advantages of the present application will become apparent from the following detailed description of exemplary embodiments of the present application with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0024] FIG1 is a flow chart of a parallel training method for a hybrid expert model of the present application.

[0025] FIG2 is a schematic diagram of a dense large model and an MOE model in the prior art.

[0026] FIG3 is a flow chart of the method for determining multiple groups of candidate parallel strategies of the present application.

[0027] FIG4 is a flow chart of the method for determining the target parallel strategy of the present application.

[0028] FIG5 is a schematic diagram of the load condition of the hybrid expert model of the present application.

[0029] FIG6 is a flow chart of the method for parallel strategy switching of the present application.

[0030] FIG7 is a schematic diagram of a parallel training device for a hybrid expert model according to the present application.

[0031] FIG8 is a block diagram of an electronic device for implementing a parallel training method for a hybrid expert model of the present application. DETAILED DESCRIPTION

[0032] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application.

[0033] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0034] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0035] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0036] This application proposes a parallel training method for a hybrid expert model, as shown in Figure 1, which includes:

[0037] Step 101: Generate multiple sets of candidate parallel strategies based on the configuration information and hardware information of the hybrid expert model;

[0038] Step 102: In the current training cycle, according to the current load of the hybrid expert model, a target parallel strategy is determined from multiple sets of candidate parallel strategies;

[0039] Step 103: Determine the historical parallel strategy used in the previous training cycle of the current training cycle;

[0040] Step 104: Train the hybrid expert model based on the better one of the historical parallel strategy and the target parallel strategy.

[0041] The present application provides a parallel training method for a hybrid expert model, which generates multiple groups of candidate parallel strategies through the configuration information and hardware information of the hybrid expert model; and in the current training cycle, determines the target parallel strategy from the multiple groups of candidate parallel strategies based on the current load of the hybrid expert model; further determines the historical parallel strategy used in the training cycle before the current training cycle; trains the hybrid expert model based on the better one of the historical parallel strategy and the target parallel strategy, and can switch the parallel strategy according to the load situation, effectively solves the load balancing problem based on the introduction of the expert strategy, and improves the parallel training efficiency of the hybrid expert model.

[0042] Each step in Figure 1 is described in detail below:

[0043] Step 101: Generate multiple groups of candidate parallel strategies based on the configuration information and hardware information of the hybrid expert model.

[0044] It's understood that a promising direction in deep neural network design in recent years has been to increase model size. From a model design perspective, sparse architectures have been proposed to break the proportional relationship between model size and computational cost in dense neural network models. For example, unlike Figure 2, which directly scales up a small model to a dense large one, a Mixture of Experts (MoE) model consists of several small models, each of which is called an expert network. For a training example, a gating network module selectively activates a small number of experts to perform computations on that example.

[0045] In this embodiment, multiple sets of candidate parallel strategies can be generated based on the configuration information and hardware information of the hybrid expert model. The configuration information of the hybrid expert model can include the number and size of expert networks. The hybrid expert model is typically deployed on computing devices on a distributed computing platform, and the hardware information can include the number of computing devices on the computing platform and the bandwidth of the computing devices. The conversion overhead within the candidate parallel strategy set is low, allowing for rapid conversion at runtime.

[0046] In some optional manners of this embodiment, as shown in FIG3 , multiple groups of candidate parallel strategies are generated based on the configuration information and hardware information of the hybrid expert model, including:

[0047] Step 1011: Based on the number of expert networks and the number of computing devices, tensor parallelism, data parallelism, pipeline parallelism, and expert parallelism are arranged and combined to generate multiple groups of initial parallel strategies.

[0048] In this embodiment, tensor parallelism, data parallelism, pipeline parallelism, and expert parallelism are all methods for parallel training of neural networks. Tensor parallelism divides the model between different computing units, allowing each unit to independently compute subtasks during training; data parallelism replicates and stores the model between different computing units, using different training data for training; pipeline parallelism trains across different computing units in a multi-stage pipeline format; and expert parallelism is a training method unique to hybrid expert models.

[0049] In this embodiment, based on the number of expert networks and computing devices, and using a search algorithm, arbitrary permutations and combinations of tensor parallelism, data parallelism, pipeline parallelism, and expert parallelism can be performed to generate multiple sets of initial parallelization strategies. For example, 50% tensor parallelism + 50% expert parallelism, 50% data parallelism + 50% expert parallelism, etc.

[0050] Step 1012: Perform performance prediction on the multiple groups of initial parallel strategies, and select multiple groups of candidate parallel strategies from the multiple groups of initial parallel strategies based on the performance prediction results.

[0051] In this embodiment, after obtaining multiple sets of initial parallel strategies, performance prediction is required for these sets to further narrow the search space and improve the efficiency of parallel training of the hybrid expert model. For example, for a given MoE model's routing algorithm, an upper limit on the computational load for each expert sub-model can be determined. This upper limit and performance prediction formula are then used to estimate the training performance of each initial parallel strategy. Ultimately, the highest-performing parallel strategy is selected as a candidate parallel strategy for training the hybrid expert model.

[0052] In some optional methods of this embodiment, in multiple groups of candidate parallel strategies, the number of expert networks of the hybrid expert model placed in each computing device is pre-fixed, that is, its segmentation and repeated placement scheme is fixed; the placement position of the expert network is not fixed, and has a mapping relationship with the candidate parallel strategy, that is, different candidate parallel strategies correspond to different placement positions of the expert network.

[0053] Step 102: In the current training cycle, a target parallel strategy is determined from multiple groups of candidate parallel strategies according to the current load of the hybrid expert model.

[0054] In this embodiment, in the current training cycle, based on the load condition of the expert network in this round of training, a target parallel strategy that best suits the current load condition can be determined from multiple groups of candidate parallel strategies.

[0055] In some optional manners of this embodiment, as shown in FIG4 , a target parallel strategy is determined from multiple groups of candidate parallel strategies based on the current load of the hybrid expert model, including:

[0056] Step 1021: Determine the current load of each expert network in the hybrid expert model.

[0057] In this embodiment, the load of each expert network can be determined according to the number of inputs received by the expert network. As shown in FIG5 , expert network 0 is in an overloaded state, and expert network 2 is in an idle state.

[0058] Step 1022: Sort the expert networks in the hybrid expert model in descending order according to the current load of each expert network to obtain the placement order of each expert network.

[0059] In this embodiment, the expert networks in the hybrid expert model are further sorted in descending order according to the current load of each expert network to obtain the placement order of each expert network, that is, the computing device to be placed is selected for each expert network in order from large to small load.

[0060] Step 1023: Select computing devices to be placed for each expert network in turn according to the placement order until the computing devices to be placed for the last expert network are selected. Determine a set of candidate parallel strategies that match the current placement method from multiple sets of candidate parallel strategies, and use this set of candidate parallel strategies as the target parallel strategy.

[0061] In some optional aspects of this embodiment, the following operations are performed on each expert network in sequence:

[0062] First, a set of computing devices is identified in which the number of placed expert networks is less than a preset threshold. For example, assuming there are N expert networks and M different computing devices, the preset threshold for the number of expert networks that can be placed on each computing device can be set to N / M. Thus, the computing devices in which the number of placed expert networks is less than the preset threshold are considered a computing device set, where the number of expert networks on this computing device set is not saturated. Furthermore, within this set of computing devices, a computing device with the lowest load of placed expert networks is identified and selected as the computing device to be placed. This placement method overcomes the load balancing issues caused by expert parallelism, avoids overload and idle conditions, and achieves a balanced configuration of the number of placed expert networks and the expert network load on each computing device.

[0063] Taking Figure 5 as an example, it is assumed that two expert networks can be placed on each computing device, and the hybrid expert model is deployed on three computing devices. The hybrid expert model includes four expert networks, and the placement order can be determined as expert network 0, expert network 1, expert network 2 and expert network 3 (not shown in Figure 5) according to the load situation. The computing devices to be placed are expert network 0, expert network 1, expert network 2 and expert network 3, respectively.

[0064] Among them, for expert network 0, it can select any computing device from computing device 0-computing device 2, for example, expert network 0 is placed on computing device 0; for expert network 1, the computing devices with the number of expert networks placed less than 2 include computing device 0, computing device 1 and computing device 2 as a computing device set, among which the load of computing device 1 and computing device 2 is the smallest, and it can select any computing device from computing device 1 and computing device 2, for example, expert network 1 is placed on computing device 1; for expert network 2, the computing devices with the number of expert networks placed less than 2 include computing device 0, computing device 1 and computing device 2 as a computing device set, among which the load of computing device 2 is the smallest, and expert network 2 is placed on computing device 2; for expert network 3, the computing devices with the number of expert networks placed less than 2 include computing device 0, computing device 1 and computing device 2 as a computing device set. Obviously, the load of expert network 2 is the smallest, and expert network 3 can be placed on computing device 2.

[0065] In this embodiment, after selecting the placed devices for each expert network in turn, a group of candidate parallel strategies that matches the current placement method is further determined from multiple groups of candidate parallel strategies, and this group of candidate parallel strategies is used as the target parallel strategy. It should be understood that the target parallel strategy is the preferred parallel strategy for the current load situation.

[0066] Step 103: Determine the historical parallel strategy used in the previous training cycle of the current training cycle.

[0067] In this embodiment, the historical parallel strategy used in the training cycle before the current training cycle is determined. In the first training cycle, a parallel strategy can be arbitrarily selected from multiple sets of candidate parallel strategies for training. In the training cycle after the first training cycle, the arbitrarily selected parallel strategy is the historical parallel strategy. Similarly, the historical parallel strategy used in the training cycle before the current training cycle can be arbitrarily determined. It should be noted that the historical parallel strategy is one of the multiple sets of candidate parallel strategies.

[0068] Step 104: Train the hybrid expert model based on the better one of the historical parallel strategy and the target parallel strategy.

[0069] It should be noted that although the target parallel strategy is the preferred parallel strategy for the current load situation, the time required for strategy switching also needs to be considered. To this end, in some optional embodiments of this embodiment, as shown in FIG6 , it is assumed that the training cycle includes K rounds of training. It should be understood that K is an integer greater than 0. Based on the better one of the historical parallel strategy and the target parallel strategy, the hybrid expert model is trained, including:

[0070] Step 1041: Determine, based on the current load situation, a first estimated execution time T1 required for one round of training of the hybrid expert model based on the historical parallel strategy and a second estimated execution time T2 required for one round of training of the hybrid expert model based on the target parallel strategy.

[0071] Step 1042: Determine the switching time TC required to switch the historical parallel strategy to the target parallel strategy.

[0072] Step 1043A: In response to the existence of T1×K>T2×K+TC, determine that the target parallel strategy is a better one, switch the historical parallel strategy to the target parallel strategy, and train the hybrid expert model based on the target parallel strategy.

[0073] In this implementation, that is, if the time it takes to execute K rounds of training using the historical parallel strategy is greater than the sum of the time it takes to execute K rounds of training using the target parallel strategy and the switching time, then strategy switching can be performed, switching the historical parallel strategy to the target parallel strategy, and training the hybrid expert model based on the target parallel strategy.

[0074] Step 1043B: In response to the existence of T1×K≤T2×K+TC, determine that the historical parallel strategy is a better one, do not switch the historical parallel strategy to the target parallel strategy, and continue to train the hybrid expert model based on the historical parallel strategy.

[0075] In this embodiment, if the time for executing K rounds of training using the historical parallel strategy is not greater than the sum of the time for executing K rounds of training using the target parallel strategy and the switching time, strategy switching cannot be performed, and the hybrid expert model continues to be trained based on the historical parallel strategy.

[0076] In this embodiment, the historical parallel strategy is further compared with the target parallel strategy considering the switching time, so that the hybrid expert model is trained based on the better one of the historical parallel strategy and the target parallel strategy, thereby further improving the accuracy and training efficiency of parallel training.

[0077] The present application provides a parallel training method for a hybrid expert model, which generates multiple groups of candidate parallel strategies through the configuration information and hardware information of the hybrid expert model; and in the current training cycle, determines the target parallel strategy from the multiple groups of candidate parallel strategies based on the current load of the hybrid expert model; further determines the historical parallel strategy used in the training cycle before the current training cycle; trains the hybrid expert model based on the better one of the historical parallel strategy and the target parallel strategy, and can switch the parallel strategy according to the load situation, effectively solves the load balancing problem based on the introduction of the expert strategy, and improves the parallel training efficiency of the hybrid expert model.

[0078] Based on the same inventive concept, the embodiments of the present application also provide a parallel training device for a hybrid expert model, which can be used to implement the method described in the above embodiments, as described in the following embodiments. Since the principle of the problem solved by the parallel training device for a hybrid expert model is similar to that of the parallel training method for a hybrid expert model, the implementation of the parallel training device for a hybrid expert model can refer to the implementation of the parallel training method for a hybrid expert model, and the repetitions will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0079] As shown in FIG7 , the parallel training device for the hybrid expert model includes:

[0080] The candidate parallel strategy generation module 701 is configured to generate multiple sets of candidate parallel strategies based on the configuration information of the hybrid expert model and the hardware information;

[0081] The target parallel strategy generating module 702 is configured to determine a target parallel strategy from the plurality of candidate parallel strategies according to the current load of the hybrid expert model in the current training cycle;

[0082] A historical parallel strategy determination module 703 is configured to determine the historical parallel strategy used in the previous training cycle of the current training cycle;

[0083] The parallel strategy switching module 704 is configured to train the hybrid expert model based on the better one of the historical parallel strategy and the target parallel strategy.

[0084] In some optional aspects of this embodiment, the configuration information includes the number of expert networks, the hardware information includes the number of computing devices, and the candidate parallel strategy generation module is further configured to:

[0085] According to the number of the expert networks and the number of the computing devices, tensor parallelism, data parallelism, pipeline parallelism and expert parallelism are arranged and combined to generate multiple groups of initial parallel strategies;

[0086] Performance prediction is performed on the multiple groups of initial parallel strategies, and based on the performance prediction results, multiple groups of candidate parallel strategies are screened out from the multiple groups of initial parallel strategies.

[0087] In some optional embodiments of this embodiment, in the multiple groups of candidate parallel strategies, the number of expert networks of the hybrid expert model placed in each computing device is fixed in advance; the placement position of the expert network is not fixed and has a mapping relationship with the candidate parallel strategy.

[0088] In some optional aspects of this embodiment, the target parallel strategy generation module includes:

[0089] an expert network load determination unit, configured to determine a current load condition of each expert network in the hybrid expert model;

[0090] a placement order determining unit configured to sort the expert networks in the hybrid expert model in descending order according to a current load condition of each expert network to obtain a placement order for each expert network;

[0091] The target parallel strategy generation unit is configured to select computing devices to be placed for each expert network in turn according to the placement order until the computing devices to be placed for the last expert network are selected, determine a group of candidate parallel strategies that match the current placement method from the multiple groups of candidate parallel strategies, and use this group of candidate parallel strategies as the target parallel strategy.

[0092] In some optional aspects of this embodiment, the target parallel strategy generation unit is further configured to:

[0093] For each expert network, perform the following operations:

[0094] Determining a set of computing devices having a number of placed expert networks less than a preset number threshold;

[0095] From the set of computing devices, a computing device with the smallest load of the placed expert network is determined, and the computing device is used as the computing device to be placed.

[0096] In some optional aspects of this embodiment, the training cycle includes K rounds of training, and the parallel strategy switching module is further configured to:

[0097] Determine, according to the current load condition, a first estimated execution time T1 required for performing a round of training on the hybrid expert model based on the historical parallel strategy and a second estimated execution time T2 required for performing a round of training on the hybrid expert model based on the target parallel strategy;

[0098] Determine a switching time TC required to switch the historical parallel strategy to the target parallel strategy;

[0099] In response to the existence of T1×K>T2×K+TC, determining that the target parallel strategy is a better one, switching the historical parallel strategy to the target parallel strategy, and training the hybrid expert model based on the target parallel strategy;

[0100] In response to the existence of T1×K≤T2×K+TC, the historical parallel strategy is determined to be a better one, the historical parallel strategy is not switched to the target parallel strategy, and the hybrid expert model continues to be trained based on the historical parallel strategy.

[0101] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0102] An electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the parallel training method for a hybrid expert model of the aforementioned embodiment.

[0103] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the steps of the parallel training method for a hybrid expert model of the aforementioned embodiment.

[0104] A computer program product includes a computer program / instruction, which implements the steps of the parallel training method for a hybrid expert model of the above embodiment when the computer program / instruction is executed by a processor.

[0105] FIG8 shows a schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0106] As shown in Figure 8, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0107] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0108] The computing unit 801 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as a parallel training method for a hybrid expert model.

[0109] For example, in some embodiments, a parallel training method for a hybrid expert model may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the parallel training method for a hybrid expert model described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform a parallel training method for a hybrid expert model in any other appropriate manner (e.g., by means of firmware).

[0110] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0112] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0113] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0114] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0115] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0116] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of this disclosure can be achieved. This is not a limitation herein.

[0117] Although some specific embodiments of the present application have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present application. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A parallel training method for a hybrid expert model, characterized in that: include: Generate multiple groups of candidate parallel strategies according to the configuration information and hardware information of the hybrid expert model; In a current training cycle, determining a target parallel strategy from the plurality of groups of candidate parallel strategies according to a current load condition of the hybrid expert model; Determine the historical parallel strategy used in the previous training cycle of the current training cycle; The hybrid expert model is trained based on a better one of the historical parallel strategy and the target parallel strategy.

2. The method according to claim 1, characterized in that The configuration information includes the number of expert networks, the hardware information includes the number of computing devices, and the generating of multiple groups of candidate parallel strategies according to the configuration information and hardware information of the hybrid expert model includes: According to the number of the expert networks and the number of the computing devices, tensor parallelism, data parallelism, pipeline parallelism and expert parallelism are arranged and combined to generate multiple groups of initial parallel strategies; Performance prediction is performed on the multiple groups of initial parallel strategies, and based on the performance prediction results, multiple groups of candidate parallel strategies are screened out from the multiple groups of initial parallel strategies.

3. The method according to claim 2, characterized in that In the plurality of groups of candidate parallel strategies, the number of expert networks of the hybrid expert model placed in each computing device is fixed in advance; the placement positions of the expert networks are not fixed and have a mapping relationship with the candidate parallel strategies.

4. The method according to claim 3, characterized in that Determining a target parallel strategy from the multiple groups of candidate parallel strategies according to the current load condition of the hybrid expert model includes: Determining the current load of each expert network in the hybrid expert model; According to the current load of each expert network, the expert networks in the hybrid expert model are sorted in descending order to obtain the placement order of each expert network; According to the placement order, computing devices to be placed are selected for each expert network in turn until the computing devices to be placed of the last expert network are selected, and a group of candidate parallel strategies matching the current placement method is determined from the multiple groups of candidate parallel strategies, and the group of candidate parallel strategies is used as the target parallel strategy.

5. The method according to claim 4, characterized in that The step of selecting a computing device to be placed for each expert network in turn according to the placement order includes: For each expert network, perform the following operations: Determine a set of computing devices having a number of placed expert networks less than a preset number threshold; From the computing device set, a computing device with the smallest load of the placed expert network is determined, and the computing device is used as the computing device to be placed.

6. The method according to claim 1, characterized in that The training cycle includes K rounds of training, and the training of the hybrid expert model based on the better one of the historical parallel strategy and the target parallel strategy includes: Determine, according to the current load condition, a first estimated execution time T1 required for performing a round of training on the hybrid expert model based on the historical parallel strategy and a second estimated execution time T2 required for performing a round of training on the hybrid expert model based on the target parallel strategy; Determine a switching time TC required to switch the historical parallel strategy to the target parallel strategy; In response to the existence of T1×K>T2×K+TC, determining that the target parallel strategy is a better one, switching the historical parallel strategy to the target parallel strategy, and training the hybrid expert model based on the target parallel strategy; In response to the existence of T1×K≤T2×K+TC, it is determined that the historical parallel strategy is a better one, the historical parallel strategy is not switched to the target parallel strategy, and the hybrid expert model continues to be trained based on the historical parallel strategy.

7. A parallel training device for a hybrid expert model, characterized in that: include: A candidate parallel strategy generation module is configured to generate multiple sets of candidate parallel strategies according to the configuration information of the hybrid expert model and the hardware information; A target parallel strategy generation module is configured to determine a target parallel strategy from the plurality of groups of candidate parallel strategies according to a current load condition of the hybrid expert model in a current training cycle; A historical parallel strategy determination module, configured as a historical parallel strategy used in a previous training cycle of a current training cycle; The parallel strategy switching module is configured to train the hybrid expert model based on a better one of the historical parallel strategy and the target parallel strategy.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the parallel training method for a hybrid expert model described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the parallel training method for a hybrid expert model described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the parallel training method for a hybrid expert model described in any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Hybrid expert model implementation method and system, electronic equipment and storage medium

    CN114841315A

  • Object selection model training method, object selection method and device

    CN115330058A

  • Neural network model training method and device, electronic equipment and storage medium

    CN115688917A

  • Parallel training method and device for hybrid expert model

    CN116502680A

  • Dynamic hardware selection for experts in mixture-of-experts model

    US20190188577A1

Cited By

  • Task processing method and system based on multiple expert layers, terminal and medium

    CN119416822A

  • Bus carbon emission factor prediction method and device for unit passengers, and medium

    CN120013301A

  • Heterogeneous hardware cluster distributed training method and device, electronic equipment and medium

    CN120066731A

  • Data processing method

    CN120234286A

  • Fatigue damage analysis method and system for industrial speed reducer based on finite element analysis

    CN120654501A