Hybrid expert model implementation method, system, electronic device, and storage medium

Abstract

The disclosure provides a hybrid expert model implementation method, system, electronic device and storage medium, relating to the fields of artificial intelligence such as deep learning and distributed storage, wherein the method can comprise: constructing a communication group, the communication group comprising a tensor parallel communication group, the tensor parallel communication group comprising at least two computing devices, and the sparse parameters of each computing device in the same tensor parallel communication group being split in a tensor parallel manner; and training a hybrid expert model based on the communication group. The scheme disclosed in the disclosure can ensure the normal progress of model training.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to hybrid expert model implementation methods, systems, electronic devices and storage media in the fields of deep learning and distributed storage. Background Technology

[0002] Mixed-of-Experts (MoE) models are a type of neural network, but unlike typical neural networks, they can be trained in multiple ways based on the data. Each model can be called an expert network. The idea behind MoE is to train multiple expert networks, each applied to a different part of the dataset. As an emerging sparse activation deep learning model architecture, MoE enables training on extremely large scales. Summary of the Invention

[0003] This disclosure provides methods, systems, electronic devices, and storage media for implementing hybrid expert models.

[0004] A hybrid expert model implementation method includes:

[0005] A communication group is constructed, which includes a tensor parallel communication group. The tensor parallel communication group includes at least two computing devices. The sparse parameters of each computing device in the same tensor parallel communication group are adopted by tensor parallel partitioning.

[0006] A hybrid expert model is trained based on the communication group.

[0007] A hybrid expert model implementation system includes a communication group for training a hybrid expert model based on the communication group;

[0008] The communication group includes a tensor parallel communication group, which includes at least two computing devices. The sparse parameters of each computing device in the same tensor parallel communication group are partitioned using a tensor parallel method.

[0009] An electronic device, comprising:

[0010] At least one processor; and

[0011] A memory communicatively connected to the at least one processor; wherein,

[0012] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0013] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0014] A computer program product includes a computer program / instructions that, when executed by a processor, implement the method described above.

[0015] One embodiment disclosed above has the following advantages or beneficial effects: by using tensor parallel partitioning for sparse parameters, the excessive size of sparse parameters (such as expert network parameters) can be avoided, thereby avoiding problems such as memory overflow caused by the inability of computing devices to support them, and ensuring the normal progress of model training.

[0016] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0017] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0018] Figure 1 This is a schematic diagram illustrating the data processing methods of existing hybrid expert models;

[0019] Figure 2 This is a flowchart of an embodiment of the hybrid expert model implementation method described in this disclosure;

[0020] Figure 3 This diagram illustrates the data processing method of a hybrid expert model after introducing data parallelism.

[0021] Figure 4 This is a schematic diagram illustrating the data processing method of the hybrid expert model after introducing tensor parallel partitioning and data parallelism as described in this disclosure;

[0022] Figure 5 This is a schematic diagram of the composition structure of the first embodiment 500 of the hybrid expert model implementation system described in this disclosure;

[0023] Figure 6 This is a schematic diagram of the composition structure of the second embodiment 600 of the hybrid expert model implementation system described in this disclosure;

[0024] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0027] Figure 1 This is a schematic diagram illustrating the data processing methods of existing hybrid expert models. For example... Figure 1 As shown, after the input data is processed (i.e. calculated) by the backbone network, the processing result H0 is obtained. The backbone network is usually composed of fully connected layers. Then, the gating network can select expert networks as routing networks for the processing result H0. Usually, the k expert networks with the highest scores are selected, where k is a positive integer. That is, the number of expert networks selected can be one or more. As shown in the figure, assuming that expert network 0 is selected, expert network 0 can further process the processing result H0 to obtain the processing result H0', which is the final output result.

[0028] for Figure 1 The hybrid expert model shown may encounter problems in practical applications where the expert network parameters are too large, leading to unsupported computing devices, such as memory overflow, which in turn affects the normal training of the model.

[0029] Therefore, this disclosure proposes a hybrid expert model implementation method. Accordingly, Figure 2 This is a flowchart illustrating an embodiment of the hybrid expert model implementation method described in this disclosure. Figure 2 As shown, the specific implementation methods are as follows.

[0030] In step 201, a communication group is constructed, which includes a tensor parallel communication group. The tensor parallel communication group includes at least two computing devices (workers). The sparse parameters of each computing device in the same tensor parallel communication group are obtained by tensor parallel partitioning.

[0031] In step 202, a hybrid expert model is trained based on the communication group.

[0032] In the above-described method embodiments, by using tensor parallel partitioning for sparse parameters, the excessive size of sparse parameters (such as expert network parameters) can be avoided, thereby preventing problems such as memory overflow caused by the inability of computing devices to support them, and ensuring the normal progress of model training.

[0033] In one embodiment of this disclosure, the constructed communication group may further include: a data parallel communication group, which includes at least two computing devices. Each computing device in the same data parallel communication group adopts a data parallel approach. In addition, for any tensor parallel communication group, each computing device is included in a data parallel communication group, and the first set of computing devices composed of computing devices included in all data parallel communication groups is equal to the second set of computing devices composed of computing devices included in all tensor parallel communication groups.

[0034] This allows the construction of hybrid communication groups that include tensor parallel communication groups and data parallel communication groups. Accordingly, the number of tensor parallel communication groups and data parallel communication groups can be greater than or equal to 2.

[0035] By introducing data parallelism into the hybrid expert model, the overall training throughput can be improved. Correspondingly, by simultaneously introducing data parallelism and tensor parallel partitioning, the advantages of both methods can be combined, thereby further improving the training effect of the model.

[0036] The following is based on Figure 1 The hybrid expert model shown illustrates the specific implementation of the data parallelism approach. Figure 3 This diagram illustrates the data processing method of a hybrid expert model after introducing data parallelism. Figure 3 As shown, assuming that computing device 0 and computing device 1 each correspond to 3 expert networks, the total number of expert networks is 6. The parameters of each expert network are different. The gating network can select the expert network as the routing network from the 6 expert networks, so that cross-card communication (i.e., cross-computing device communication) can be performed. As shown in the figure, assuming that expert network 4 and expert network 1 are selected respectively, then taking computing device 0 as an example, the processing result H0 of the backbone network can be sent to the selected expert network 4. After being processed by expert network 4, the processing result H0' is returned to computing device 0, thus obtaining the final output result 0.

[0037] Accordingly, Figure 4 This diagram illustrates the data processing method of the hybrid expert model after introducing tensor parallel partitioning and data parallelism as described in this disclosure.

[0038] like Figure 4As shown, in one embodiment of this disclosure, any computing device may include: a backbone network for performing predetermined processing on input data to obtain a first processing result; a gating network for selecting an expert network as a routing network from the expert networks of each computing device in the data parallel communication group, and sending the first processing result to the selected expert network; and an expert network for performing predetermined processing on the obtained first processing result to obtain a second processing result, and returning the return result determined according to the second processing result to the computing device corresponding to the obtained first processing result; different computing devices in the same data parallel communication group correspond to the same backbone network and expert network.

[0039] For ease of distinction, the processing results of the backbone network and the processing results of the expert network will be referred to as the first processing result and the second processing result, respectively. Furthermore, the specific type of predetermined processing is not limited and can be determined according to the actual situation.

[0040] like Figure 4 As shown, in one embodiment of this disclosure, the dense parameters of each computing device in the same tensor parallel communication group can also be obtained by tensor parallel partitioning. The sparse parameters may include expert network parameters, and the dense parameters may include backbone network parameters.

[0041] like Figure 4 As shown, there are four computing devices: computing device 0, computing device 1, computing device 2, and computing device 3. The parallelism is 2-way data parallelism and 2-way tensor parallelism. There are also two expert networks: expert network 0 and expert network 1. [Computing device 0, computing device 1] and [computing device 2, computing device 3] form two tensor parallel communication groups, and [computing device 0, computing device 2] and [computing device 1, computing device 3] form two data parallel communication groups. The two tensor parallel communication groups and the two data parallel communication groups together form the constructed hybrid communication group.

[0042] like Figure 4 As shown, in each tensor parallel communication group, the backbone network parameters are tensor parallelly partitioned. Computing device 0 and computing device 1 share the same backbone network, as do computing device 2 and computing device 3. Furthermore, computing device 0 and computing device 1 read the same input data 0, and after processing by the backbone network, they obtain the same first processing result H_0. Similarly, computing device 2 and computing device 3 read the same input data 1, and after processing by the backbone network, they obtain the same first processing result H_1.

[0043] like Figure 4As shown, computing devices 0 and 1 share the same expert network 0, and computing devices 2 and 3 share the same expert network 1. In each data parallel communication group, the gating network can select an expert network as the routing network for its respective first processing result. Assuming that one expert network is selected, there are two expert networks to choose from in the data parallel communication group [computing device 0, computing device 2]. The gating network in computing device 0 selects expert network 1 in computing device 2, and the gating network in computing device 2 selects expert network 0 in computing device 0. Furthermore, the gating networks in computing devices 0 and 2 can send their respective first processing results to the selected expert networks for processing. Similarly, in the data parallel communication group [computing device 1, computing device 3], there are two expert networks to choose from. The gated network in computing device 1 selects expert network 1 in computing device 3, and the gated network in computing device 3 selects expert network 0 in computing device 1. Furthermore, the gated networks in computing devices 1 and 3 can respectively send their corresponding first processing results to the selected expert network for processing. Since the backbone network parameters use tensor parallel partitioning, the routing results for computing devices 1 and 0 are consistent, and the routing results for computing devices 3 and 2 are consistent.

[0044] like Figure 4 As shown, since the expert network parameters adopt tensor parallel partitioning, after processing by the expert network, the expert networks in the same tensor parallel communication group will obtain the same second processing result. That is, the expert network 0 in computing device 0 and computing device 1 will obtain the same second processing result H_3, and the expert network 1 in computing device 2 and computing device 3 will obtain the same second processing result H_4.

[0045] The obtained second processing result also needs to be returned to the corresponding computing device. Specifically, the expert network can return the result determined based on the second processing result to the computing device corresponding to the obtained first processing result.

[0046] The returned result can refer to the second processing result, meaning the second processing result can be directly returned as the return result. Alternatively, in one embodiment of this disclosure, the returned result may include a portion of the second processing result. Accordingly, for any single-quantum parallel communication group, the portions of the obtained second processing result can be combined to obtain a complete second processing result. The portions are returned by the same expert network corresponding to each computing device in the same single-quantum parallel communication group.

[0047] The specific method used can be determined according to actual needs, which is very flexible and convenient. Preferably, the latter method can be used.

[0048] In the latter method, such as Figure 4 As shown, since the second processing result obtained by the expert network in the same quantity parallel communication group is the same, when returning the second processing result, only a part of the content can be returned. For example, computing device 0 can only return the first half of the second processing result H_3[:2] to computing device 2, and computing device 1 can only return the second half of the second processing result H_3[2:] to computing device 3. Similarly, computing device 2 can only return the first half of the second processing result H_4[:2] to computing device 0, and computing device 3 can only return the second half of the second processing result H_4[2:] to computing device 1, thereby reducing the amount of data to be transmitted and thus reducing resource consumption.

[0049] The above example uses two computing devices in the same parallel communication group. If there are three computing devices, then one-third of the same second processing result can be returned, that is, the first, middle and last parts can be returned respectively. Usually, the returned parts do not overlap with each other, so as to further reduce resource consumption.

[0050] Since only partial content is returned, for any tensor parallel communication group, it is necessary to combine the acquired portions belonging to the same second processing result to obtain the complete second processing result. This requires an all-gather communication within the tensor parallel communication group to restore the original second processing result. For example... Figure 4 As shown, computing device 0 and computing device 1 obtained H_4[:2] and H_4[2:] respectively. Then, H_4[:2] and H_4[2:] can be combined to obtain the complete second processing result H_4. Similarly, computing device 2 and computing device 3 obtained H_3[:2] and H_3[2:] respectively. Then, H_3[:2] and H_3[2:] can be combined to obtain the complete second processing result H_3.

[0051] based on Figure 4 The hybrid communication group shown can complete the training of hybrid expert models, and the training method can be the same as that in existing technologies.

[0052] It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0053] In summary, the scheme described in the embodiments of this disclosure can avoid the problem of excessively large sparse parameters (such as expert network parameters), thereby avoiding problems such as memory overflow caused by the inability of computing devices to support it, ensuring the normal progress of model training, and making more effective use of storage space to support larger-scale model training, etc.

[0054] The above is an introduction to the method embodiments. The following system embodiments will further illustrate the solution described in this disclosure.

[0055] Figure 5 This is a schematic diagram of the structural composition of the first embodiment 500 of the hybrid expert model implementation system described in this disclosure. Figure 5 As shown, it includes a communication group 501 for training a hybrid expert model based on the communication group 501.

[0056] The communication group 501 may include a tensor parallel communication group; the tensor parallel communication group includes at least two computing devices, and the sparse parameters of each computing device in the same tensor parallel communication group are partitioned using tensor parallel methods.

[0057] In the above system embodiment, by using tensor parallel partitioning for sparse parameters, the excessive size of sparse parameters (such as expert network parameters) can be avoided, thereby preventing problems such as memory overflow caused by the inability of computing devices to support them, and ensuring the normal progress of model training.

[0058] In one embodiment of this disclosure, the communication group 501 may further include a data parallel communication group, which includes at least two computing devices. Each computing device in the same data parallel communication group adopts a data parallel mode. In addition, for any tensor parallel communication group, each computing device is included in a data parallel communication group, and the first set of computing devices composed of computing devices included in all data parallel communication groups is equal to the second set of computing devices composed of computing devices included in all tensor parallel communication groups.

[0059] This allows the construction of hybrid communication groups that include tensor parallel communication groups and data parallel communication groups. Accordingly, the number of tensor parallel communication groups and data parallel communication groups can be greater than or equal to 2.

[0060] In one embodiment of this disclosure, any computing device may include: a backbone network for performing predetermined processing on input data to obtain a first processing result; a gating network for selecting an expert network as a routing network from the expert networks of each computing device in the data parallel communication group, and sending the first processing result to the selected expert network; and an expert network for performing predetermined processing on the obtained first processing result to obtain a second processing result, and returning the return result determined according to the second processing result to the computing device corresponding to the obtained first processing result; different computing devices in the same data parallel communication group correspond to the same backbone network and expert network.

[0061] In another embodiment of this disclosure, the dense parameters of each computing device in the same tensor parallel communication group can also be obtained by tensor parallel partitioning, wherein the sparse parameters may include: expert network parameters; and the dense parameters may include: backbone network parameters.

[0062] As mentioned earlier, the obtained second processing result also needs to be returned to the corresponding computing device. Specifically, the expert network can return the result determined based on the second processing result to the computing device corresponding to the obtained first processing result.

[0063] The returned result can refer to the second processing result, that is, the second processing result can be directly returned as the returned result.

[0064] Alternatively, in one embodiment of this disclosure, the returned result may include a portion of the second processing result, accordingly. Figure 6 This is a schematic diagram of the composition structure of the second embodiment 600 of the hybrid expert model implementation system described in this disclosure.

[0065] like Figure 6 As shown, compared to Figure 5 The embodiment shown also includes: a convergence module 502, used to combine the parts of the acquired second processing result for any single-quantum parallel communication group to obtain a complete second processing result, wherein the parts are returned by the same expert network corresponding to each computing device in the same single-quantum parallel communication group.

[0066] Figure 5 and Figure 6 The specific workflow of the system embodiment shown can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0067] In summary, the solution described in the system embodiments of this disclosure can avoid the problem of excessively large sparse parameters (such as expert network parameters), thereby avoiding problems such as memory overflow caused by the inability of computing devices to support it, ensuring the normal progress of model training, and making more effective use of storage space to support larger-scale model training, etc.

[0068] The solutions described in this disclosure can be applied to the field of artificial intelligence, particularly deep learning and distributed storage. Artificial intelligence is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. Artificial intelligence hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0069] The data and other information described in the embodiments of this disclosure are not targeted at any specific user and do not reflect the personal information of any specific user. The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solutions of this disclosure all comply with relevant laws and regulations and do not violate public order and good morals.

[0070] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0071] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0072] like Figure 7As shown, device 700 includes a computing device 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing device 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0073] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0074] The computing device 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing device 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing devices running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing device 701 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing device 701, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing device 701 can be configured to perform the methods described in this disclosure by any other suitable means (e.g., by means of firmware).

[0075] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0076] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0077] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0078] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0079] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0080] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0081] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0082] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A distributed hybrid expert model implementation method, comprising: A communication group is constructed, comprising a tensor parallel communication group and a data parallel communication group. The tensor parallel communication group includes at least two computing devices. The sparse parameters of each computing device in the same tensor parallel communication group are partitioned using a tensor parallel method, and the dense parameters of each computing device in the same tensor parallel communication group are also partitioned using the tensor parallel method. The sparse parameters include expert network parameters, and the dense parameters include backbone network parameters. The data parallel communication group includes at least two computing devices. The computing devices in the same data parallel communication group use a data parallel method. For any tensor parallel communication group, each computing device is included in a data parallel communication group. Furthermore, the first set of computing devices comprised of all computing devices included in all data parallel communication groups is equal to the second set of computing devices comprised of all computing devices included in all tensor parallel communication groups. Based on the communication group, a hybrid expert model is trained using the data. Each of these computing devices includes: The backbone network is used to perform predetermined processing on the input data to obtain the first processing result of the data; A gated network is used to select an expert network as a routing network from the expert networks of each computing device in the data parallel communication group, and send the first processing result to the selected expert network. An expert network is used to perform predetermined processing on the acquired first processing result to obtain a second processing result of the data, and return the return result determined according to the second processing result to the computing device corresponding to the acquired first processing result. The return result includes a portion of the second processing result, so that for any single-parallel communication group, the portions of the acquired data belonging to the same second processing result are combined to obtain a complete second processing result. Each portion of the data is returned by the same expert network corresponding to each computing device in the same single-parallel communication group. Different computing devices in the same single-parallel communication group correspond to the same backbone network and expert network.

2. A distributed hybrid expert model implementation system, comprising a convergence module and a communication group, used to train a hybrid expert model using data based on the communication group; The communication group includes a tensor parallel communication group and a data parallel communication group. The tensor parallel communication group includes at least two computing devices. The sparse parameters of each computing device in the same tensor parallel communication group are partitioned using tensor parallel methods, and the dense parameters of each computing device in the same tensor parallel communication group are partitioned using the same tensor parallel methods. The sparse parameters include expert network parameters, and the dense parameters include backbone network parameters. The data parallel communication group includes at least two computing devices. The computing devices in the same data parallel communication group use data parallel methods. For any tensor parallel communication group, each computing device is included in a data parallel communication group. Furthermore, the first set of computing devices composed of the computing devices included in all data parallel communication groups is equal to the second set of computing devices composed of the computing devices included in all tensor parallel communication groups. in, Each computing device includes: The backbone network is used to perform predetermined processing on the input data to obtain the first processing result of the data; A gated network is used to select an expert network as a routing network from the expert networks of each computing device in the data parallel communication group, and send the first processing result to the selected expert network. An expert network is used to perform predetermined processing on the acquired first processing result to obtain a second processing result of the data, and return the return result determined according to the second processing result to the computing device corresponding to the acquired first processing result; Different computing devices in the same quantitative parallel communication group correspond to the same backbone network and expert network; The returned result includes a portion of the second processing result; The aggregation module is used to combine the parts of the acquired second processing result for any single parallel communication group to obtain a complete second processing result. The parts are returned by the same expert network corresponding to each computing device in the same single parallel communication group.

3. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of claim 1.

4. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method of claim 1.

5. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method of claim 1.

Images