Hybrid expert model communication optimization method, device, equipment, medium and program

Abstract

The embodiment of the present invention discloses a hybrid expert model communication optimization method, device, equipment, medium and program, including: establishing a first current inter-machine communication connection between a current first inter-machine communication data block and a first aligned card machine inter-machine communication data block to realize the first inter-machine communication between the current first inter-machine communication data block and the first aligned card machine inter-machine communication data block; when the current first inter-machine communication data block is completed, updating the current first intra-machine communication data block and the current first inter-machine communication data block, and re-determining the first aligned card machine inter-machine communication data block corresponding to the current first inter-machine communication data block; while the current first intra-machine communication data block performs intra-machine communication based on the first intra-machine communication mode, re-establishing the first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card machine inter-machine communication data block to realize the first inter-machine communication between the current first inter-machine communication data block and the first aligned card machine communication data block in parallel.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a hybrid expert model communication optimization method, device, equipment, medium and program. Background Art

[0002] The Mixture of Expert (MoE) model is a sparse neural network that can greatly increase the number of model parameters without increasing the amount of model calculations, thereby improving the model's capabilities. At the same time, due to the sparsity of the model, usually 1-2 experts are selected for processing for each token (the smallest unit of input data used by the model), and different data are processed by different experts.

[0003] Currently, in the hybrid expert model, data needs to be sent to the corresponding expert through AllToAll (a collective communication operation in which all processes participating in the communication send the same amount of data to all other processes in the communication domain and receive the same amount of data from all these processes). After the expert receiving the data completes the calculation processing locally, the result is transmitted back to the corresponding expert through AllToAll. The hybrid expert model determines which expert network the data selects for processing through the gating network and completes the data transmission through the AllToAll communication mechanism.

[0004] In the process of implementing the present invention, the inventors found that the prior art has the following defects: Since each expert model in MoE is large, different experts are distributed in different nodes. AllToAll communication uses inter-machine bandwidth, but the inter-machine bandwidth is usually smaller than the intra-machine bandwidth, and communication accounts for a relatively high proportion. The data communication method of the existing MoE model does not utilize the parallelism of intra-machine communication and inter-machine communication, and does not fully utilize different hardware architectures to execute different communication engines in parallel, making AllToAll communication a performance bottleneck of the MoE model. Summary of the invention

[0005] The embodiments of the present invention provide a hybrid expert model communication optimization method, device, equipment, medium and program, which can reduce resource waiting when the hybrid expert model processes data, thereby optimizing the communication efficiency and operation speed of the hybrid expert model, and further improving the hardware resource utilization rate and the execution efficiency of the model when the hybrid expert model processes data.

[0006] According to one aspect of the present invention, a hybrid expert model communication optimization method is provided, comprising:

[0007] Determine a current first inter-machine communication data block and a first alignment card inter-machine communication data block corresponding to the current first inter-machine communication data block;

[0008] Establishing a first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine communication data block, and realizing a first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine communication data block based on the first current inter-machine communication connection;

[0009] When it is determined that the current first inter-machine communication data block completes the first inter-machine communication, the current first inter-machine communication data block is updated to the current first intra-machine communication data block, the next first inter-machine communication data block is updated to the current first inter-machine communication data block, and the first alignment card inter-machine communication data block corresponding to the current first inter-machine communication data block is re-determined;

[0010] While the current first intra-machine communication data block is performing intra-machine communication based on the first intra-machine communication mode, the first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is re-established, and the first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is implemented in parallel based on the first current inter-machine communication connection.

[0011] According to another aspect of the present invention, there is provided a hybrid expert model communication optimization device, comprising:

[0012] A first inter-machine communication data block determining module, used to determine a current first inter-machine communication data block and a first alignment card inter-machine communication data block corresponding to the current first inter-machine communication data block;

[0013] A first inter-machine communication module, used for establishing a first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card inter-machine communication data block, and realizing a first inter-machine communication between the current first inter-machine communication data block and the first aligned card inter-machine communication data block based on the first current inter-machine communication connection;

[0014] A first inter-machine internal data block updating module is used to update the current first inter-machine communication data block to the current first internal communication data block, update the next first inter-machine communication data block to the current first inter-machine communication data block, and re-determine the first alignment card inter-machine communication data block corresponding to the current first inter-machine communication data block when it is determined that the current first inter-machine communication data block completes the first inter-machine communication;

[0015] The first inter-machine intra-machine parallel communication module is used to re-establish the first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block while the current first inter-machine communication data block performs intra-machine communication based on the first intra-machine communication mode, and to implement the first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block in parallel based on the first current inter-machine communication connection.

[0016] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the hybrid expert model communication optimization method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the hybrid expert model communication optimization method described in any embodiment of the present invention when executed.

[0021] According to another aspect of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the hybrid expert model communication optimization method described in any embodiment of the present invention is implemented.

[0022] The embodiment of the present invention establishes a first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block after determining the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block, and implements the first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block based on the first current inter-machine communication connection. When it is determined that the current first inter-machine communication data block completes the first inter-machine communication, the current first inter-machine communication data block is updated to the current first intra-machine communication data block, the next first inter-machine communication data block is updated to the current first inter-machine communication data block, and the first aligned card-machine inter-machine communication data block corresponding to the current first inter-machine communication data block is re-determined. While the current first intra-machine communication data block performs intra-machine communication based on the first intra-machine communication mode, the first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is re-established, and the first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is implemented in parallel based on the first current inter-machine communication connection. The above technical solution can solve the problem of low communication efficiency in the existing hybrid expert model, reduce the resource waiting when the hybrid expert model processes data, thereby optimizing the communication efficiency and computing speed of the hybrid expert model, and further improve the hardware resource utilization and model execution efficiency when the hybrid expert model processes data.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0025] Figure 1 is a flow chart of a hybrid expert model communication optimization method provided by Embodiment 1 of the present invention;

[0026] Figure 2 It is a schematic diagram of the calculation and communication process of the MoE module under the tensor parallel strategy in the prior art;

[0027] Figure 3 This is a schematic diagram of the effect of optimizing the inter-machine communication mode provided by the first embodiment of the present invention;

[0028] Figure 4It is a schematic diagram of the calculation and communication flow of a MoE module after communication optimization provided by the first embodiment of the present invention;

[0029] Figure 5 It is a schematic diagram of another calculation and communication process of the MoE module after communication optimization provided by the first embodiment of the present invention;

[0030] Figure 6 is a flow chart of a hybrid expert model communication optimization method provided by Embodiment 2 of the present invention;

[0031] Figure 7 It is a schematic diagram of the effect of an inter-machine communication and intra-machine communication pipeline working mode provided by the second embodiment of the present invention;

[0032] Figure 8 This is a schematic diagram of the effect of multiple data streams running provided by the second embodiment of the present invention;

[0033] Fig. 9 It is a schematic diagram of a communication semantic priority configuration process provided by Embodiment 2 of the present invention;

[0034] Fig.10 This is a schematic diagram of the effect of configuring communication priorities for different parallel strategies provided by the second embodiment of the present invention;

[0035] Fig.11 is a schematic diagram of a hybrid expert model communication optimization device provided in Embodiment 3 of the present invention;

[0036] Fig.12 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first", "second", "third", "fourth" and "target" in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] Embodiment 1

[0040] Figure 1 This is a flow chart of a hybrid expert model communication optimization method provided by the first embodiment of the present invention. This embodiment is applicable to the case where hybrid expert model inter-machine communication and intra-machine communication are executed in parallel to improve communication efficiency. The method can be executed by a hybrid expert model communication optimization device, which can be implemented by software and / or hardware and can generally be integrated in an electronic device. The electronic device can be a terminal device or a server device. As long as it can execute the hybrid expert model communication optimization method, the embodiment of the present invention does not limit the specific device type of the electronic device. Accordingly, Figure 1 As shown, the method includes the following operations:

[0041] S110, determining a current first inter-machine communication data block and a first aligned card-machine communication data block corresponding to the current first inter-machine communication data block.

[0042] Among them, the current first inter-machine communication data block may be a data block currently processed by a training card in a cluster node, and the processing process of this data block requires the first inter-machine communication process with the training cards corresponding to other cluster nodes. The first alignment card inter-machine communication data block may be a data block currently processed by training cards in other cluster nodes, corresponding to the current first inter-machine communication data block. The training card may be a hardware device configured in a cluster node for training a hybrid expert model, which is intended to support the training of a deep learning model and requires processing of a large amount of data and complex calculations, such as a chip such as a GPU. The alignment card may be a training card that belongs to a different cluster node from the current training card and needs to perform inter-machine communication with the current training card. It is understandable that the number of alignment cards corresponding to the current training card may be multiple.

[0043] Optionally, the cluster used to train the hybrid expert model may include multiple nodes, and each node may use multiple training cards to perform distributed parallel training on the hybrid expert model. In the process of distributed parallel training of the hybrid expert model, the training cards in each cluster node often need to perform inter-machine communication and intra-machine communication processes when processing data blocks. Among them, the inter-machine communication process requires the corresponding training card to exchange data between different cluster nodes for the currently processed data block. The intra-machine communication process requires data exchange between the training cards in a cluster node.

[0044] It can be understood that the current first inter-machine communication data block and the first alignment card inter-machine communication data block are relative concepts. Exemplarily, assuming that the training card A1 of node A is the current training card, the data block currently processed by the training card A1 that needs to perform the first inter-machine communication can be used as the current first inter-machine communication data block. If the training card B1 of node B is the alignment card of the training card A1 of node A, the data block currently processed by the training card B1 of node B that needs to perform the first inter-machine communication can be used as the first alignment card inter-machine communication data block. If the training card B1 of node B and the training card C1 of node C are both the alignment cards of the training card A1 of node A, the data blocks currently processed by the training card B1 of node B and the training card C1 of node C that need to perform the first inter-machine communication are both used as the first alignment card inter-machine communication data blocks. Correspondingly, assuming that the training card B1 of node B is used as the current training card, the data block that is currently processed by the training card B1 and needs to be used for the first inter-machine communication can be used as the current first inter-machine communication data block, then the data block that is currently processed by the training card A1 of node A and needs to be used for the first inter-machine communication can be used as the first alignment card inter-machine communication data block, or the data blocks that are currently processed by the training card A1 of node A and the training card C1 of node C and need to be used for the first inter-machine communication can both be used as the first alignment card inter-machine communication data block.

[0045] S120, establishing a first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine communication data block, and realizing a first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine communication data block based on the first current inter-machine communication connection.

[0046] Among them, the first current inter-machine communication connection can be the first inter-machine communication connection established between the current training card and the alignment card. The first inter-machine communication can be the inter-machine communication between the current first inter-machine communication data block and the first alignment card inter-machine communication data block based on the first established inter-machine communication connection. Optionally, the communication mode of the first inter-machine communication can be, for example, an inter-machine communication mode in the form of AllToAll, etc. The embodiment of the present invention does not limit the specific communication mode of the first inter-machine communication. It can be understood that the cluster node performs the first inter-machine communication between the current first inter-machine communication data block and the first alignment card inter-machine communication data block before performing the expert calculation.

[0047] Figure 2 is a schematic diagram of the calculation and communication flow of the MoE module under the tensor parallel strategy in the prior art. In a specific example, Figure 2 As shown, the communication process of the hybrid expert model is described by taking two cluster nodes and the inter-machine communication mode of AllToAll between the nodes as an example. The first inter-machine communication occurs between GPU1 and GPU3 of node 1 and node 2 (such as Figure 2 A1 and A2 in ), and between GPU2 and GPU4 (such as Figure 2 B1 and B2 in ). AllGather (a collective communication algorithm, whose main function is to collect data from all nodes to a master node, and then distribute the collected data to other nodes) and ReduceScatter (a communication primitive in distributed computing, all participating nodes will perform some kind of reduction operation, such as summing, finding the maximum value, etc., and then distribute the reduced results to each node) are two specific intra-machine communication methods, which occur between GPU1 and GPU2 in node 1, and between GPU3 and GPU4 in node 2. The second AllToAll inter-machine communication method is analogous. The communication method of the above-mentioned existing hybrid expert model does not utilize the parallelism of intra-machine communication and inter-machine communication, and does not fully utilize different hardware architectures to execute different communication engines in parallel.

[0048] In the embodiment of the present invention, in order to optimize the communication mode of the hybrid expert model, the single inter-machine communication data can be divided into different data blocks, and the training card performs data processing in units of data blocks each time. At the same time, the inter-machine communication mode can be converted into a combination of Drop (discard module), inter-machine communication and intra-machine communication.

[0049] Figure 3 1 is a schematic diagram of the effect of optimizing the inter-machine communication mode provided by the first embodiment of the present invention. Figure 4 Schematic diagram of the calculation and communication flow of the MoE module after communication optimization provided by the first embodiment of the present invention. Figure 3As shown in , if the AllToAll inter-machine communication method is adopted, the original AllToAll inter-machine communication method can be converted into Drop, AllToAll (inter-machine) and AllGather (intra-machine) methods. Among them, the Drop module refers to a single training card taking a slice of the global activation value to ensure that the activation value data on each training card in the tensor parallel group is non-redundant. Correspondingly, as Figure 2 and Figure 4 As shown, the first AllToAll converted Drop is Figure 2 The AllGather combination after the gated network can save one AllGather operation, thereby reducing redundant communication volume and effectively utilizing the high-bandwidth communication resources within the machine.

[0050] Figure 5 FIG. 1 is another schematic diagram of the calculation and communication flow of the MoE module after communication optimization provided by the first embodiment of the present invention. Figure 3 After optimizing the inter-machine communication in the manner shown in Figure 5 The calculation and communication flow of the MoE module after communication optimization is shown in FIG. Figure 5 As shown, modules such as gating and permute are omitted. For GPU1 of node 1, if the data block marked by A1 (referred to as data block A1, and the same applies to subsequent data blocks) is the current first inter-machine communication data block, then the first alignment card inter-machine communication data block corresponding to the current first inter-machine communication data block is data block A2 processed by GPU3 of node 2. At the same time, data block A2 is also the current first inter-machine communication data block of GPU3. It can be understood that if there are training cards of multiple cluster nodes that communicate with GPU1 as alignment cards at the same time, such as GPU5 and GPU7, then the current first inter-machine communication data block of GPU5 and GPU7 is also the first alignment card inter-machine communication data block of GPU1. Taking GPU1 as the current training card as an example, after GPU1 determines the current first inter-machine communication data block and the first alignment card inter-machine communication data block corresponding to the current first inter-machine communication data block, it can establish the first current inter-machine communication connection between the current first inter-machine communication data block and the first alignment card inter-machine communication data block, and based on the established first current inter-machine communication connection, realize the first inter-machine communication between the current first inter-machine communication data block and each first alignment card inter-machine communication data block. For example, GPU1 can establish the first current inter-machine communication connection between data block A1 and data block A2, and based on the established first current inter-machine communication connection, realize the first inter-machine communication between data block A1 and data block A2. Similarly, GPU2 can establish the first current inter-machine communication connection between data block B1 and data block B2, and based on the established first current inter-machine communication connection, realize the first inter-machine communication between data block B1 and data block B2.

[0051] S130. When it is determined that the current first inter-machine communication data block completes the first inter-machine communication, the current first inter-machine communication data block is updated to the current first intra-machine communication data block, the next first inter-machine communication data block is updated to the current first inter-machine communication data block, and the first alignment card inter-machine communication data block corresponding to the current first inter-machine communication data block is re-determined.

[0052] The first intra-machine communication data block may be a data block currently processed by a training card in a cluster node, and the processing of the data block requires intra-machine communication with other training cards in the cluster node. The first inter-machine communication data block may be a communication data block that requires a first inter-machine communication process.

[0053] It is understandable that after the current training card completes the first inter-machine communication process between the current first inter-machine communication data block and the first alignment card inter-machine communication data block, it needs to communicate with other training cards in the same node, so as to perform the data exchange process in the machine with other training cards in the same node. Correspondingly, if the current training card determines that the current first inter-machine communication data block completes the first inter-machine communication process, the current first inter-machine communication data block can be updated to the current first intra-machine communication data block, so as to perform the intra-machine communication process between the current first intra-machine communication data block and other training cards of the communication node. It should be noted that updating the current first inter-machine communication data block to the current first intra-machine communication data block is to use the data block after completing the first inter-machine communication as the current first intra-machine communication data block, which is only an update of the data block role, and does not update the data content included in the data block after completing the first inter-machine communication. That is, the data block obtained after the current first inter-machine communication data block completes the inter-machine communication can be used as the current first intra-machine communication data block to continue the intra-machine communication process. At the same time, the current training card can update the next first inter-machine communication data block to the current first inter-machine communication data block, and re-determine the first alignment card inter-machine communication data block corresponding to the current first inter-machine communication data block.

[0054] In a specific example, Figure 5As shown, assuming that GPU1 is the current training card, data block A1 is the current first inter-machine communication data block. When GPU1 completes the first inter-machine communication process for data block A1, data block A1 is updated to the current first intra-machine communication data block G1, and the second data block A3 is updated to the current first inter-machine communication data block, and the first alignment card inter-machine communication data block A4 corresponding to data block A3 is re-determined. Similarly, assuming that GPU2 is the current training card, data block B1 is the current first inter-machine communication data block. When GPU2 completes the first inter-machine communication process for data block B1, data block B1 is updated to the current first intra-machine communication data block G2, and the second data block B3 is updated to the current first inter-machine communication data block, and the first alignment card inter-machine communication data block B4 corresponding to data block B3 is re-determined.

[0055] S140. While the current first intra-machine communication data block is performing intra-machine communication based on the first intra-machine communication mode, the first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is re-established, and the first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is implemented in parallel based on the first current inter-machine communication connection.

[0056] The first intra-machine communication mode may be an intra-machine communication mode adopted by each cluster node before performing expert calculation on the hybrid expert model.

[0057] In an embodiment of the present invention, in order to realize the parallel execution of inter-machine communication and intra-machine communication, when the current training card completes the first inter-machine communication for the current first inter-machine communication data block, it can establish the first intra-machine communication between the current first intra-machine communication data block and the current first intra-machine communication data blocks of other training cards in the same cluster node, and while the current first intra-machine communication data block performs intra-machine communication based on the first intra-machine communication mode, the first current inter-machine communication connection between the current first inter-machine communication data block and the first alignment card inter-machine communication data block is re-established, so that the current training card can realize the first inter-machine communication between the current first inter-machine communication data block and the first alignment card inter-machine communication data block in parallel based on the first current inter-machine communication connection, thereby realizing the parallel execution of inter-machine communication and intra-machine communication.

[0058] In a specific example, Figure 5As shown, assuming that GPU1 is the current training card, data block A1 is the current first inter-machine communication data block. After GPU1 completes the first inter-machine communication process for data block A1, it updates data block A1 to the current first intra-machine communication data block G1, and updates the second data block A3 to the current first inter-machine communication data block, and re-determines the first alignment card inter-machine communication data block A4 corresponding to data block A3. At this time, GPU1 can perform an intra-machine communication process on data block G1, such as an AllGather communication process, and can also perform an inter-machine communication process on the current first inter-machine communication data block A3 and the corresponding first alignment card inter-machine communication data block A4, such as an AllToAll communication process. In this way, the inter-machine communication processing of the hybrid expert model can be converted into a parallel processing flow of inter-machine communication and intra-machine communication, that is, the serial execution of inter-machine communication and intra-machine communication in the hybrid expert model can be optimized to a parallel execution mode, thereby making full use of the intra-machine and inter-machine communication resources during the training of the hybrid expert model.

[0059] It can be seen that the above technical solution effectively utilizes the separated hardware architecture resources of inter-machine communication and intra-machine communication by converting the inter-machine communication mode of the hybrid expert model into inter-machine communication and intra-machine communication, realizes pipelining of inter-machine communication and intra-machine communication, hides the inter-machine communication masking, avoids waiting for communication resources, and improves the model training efficiency.

[0060] The embodiment of the present invention establishes a first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block after determining the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block, and implements the first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block based on the first current inter-machine communication connection. When it is determined that the current first inter-machine communication data block completes the first inter-machine communication, the current first inter-machine communication data block is updated to the current first intra-machine communication data block, the next first inter-machine communication data block is updated to the current first inter-machine communication data block, and the first aligned card-machine inter-machine communication data block corresponding to the current first inter-machine communication data block is re-determined. While the current first intra-machine communication data block performs intra-machine communication based on the first intra-machine communication mode, the first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is re-established, and the first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is implemented in parallel based on the first current inter-machine communication connection. The above technical solution can solve the problem of low communication efficiency in the existing hybrid expert model, reduce the resource waiting when the hybrid expert model processes data, thereby optimizing the communication efficiency and computing speed of the hybrid expert model, and further improve the hardware resource utilization and model execution efficiency when the hybrid expert model processes data.

[0061] Embodiment 2

[0062] Figure 6 is a flow chart of a hybrid expert model communication optimization method provided by the second embodiment of the present invention. This embodiment is specific based on the above embodiment. In this embodiment, multiple specific optional implementation methods are provided for calculating the number of communication data blocks to be segmented, optimizing the second intra-machine communication mode, running the corresponding kernel function through the data stream, and configuring the priority of the communication primitive to resolve the communication conflict. Figure 6 As shown, the method of this embodiment may include:

[0063] S210, testing the inter-machine communication efficiency of the inter-machine communication mode and the intra-machine communication efficiency of the first intra-machine communication mode; calculating the inter-machine communication time of a single data block according to the inter-machine communication efficiency, and calculating the intra-machine communication time of a single data block according to the intra-machine communication efficiency.

[0064] Among them, the inter-machine communication efficiency can be used to characterize the efficiency of inter-machine communication. The intra-machine communication efficiency can be used to characterize the efficiency of intra-machine communication. The inter-machine communication time of a single data block can be the communication time required for a single data block to communicate between machines, for example, it can be a single slice (data block) AllToAll time. The intra-machine communication time of a single data block can be the communication time required for a single data block to communicate within a machine, for example, it can be a single slice AllGather time.

[0065] In the embodiment of the present invention, the inter-machine communication efficiency and the intra-machine communication efficiency may be data obtained by pre-testing. Optionally, the inter-machine communication efficiency and the intra-machine communication efficiency may be a corresponding relationship curve between the communication volume and the corresponding communication efficiency, that is, the inter-machine communication efficiency may be searched and obtained through the corresponding relationship curve between the communication volume and the inter-machine communication efficiency, and one communication volume corresponds to one inter-machine communication efficiency. Similarly, the intra-machine communication efficiency may also be searched and obtained through the corresponding relationship curve between the communication volume and the intra-machine communication efficiency, and one communication volume corresponds to one intra-machine communication efficiency. The corresponding relationship curve between the communication volume and the inter-machine communication efficiency, and the corresponding relationship curve between the communication volume and the intra-machine communication efficiency may be drawn by pre-testing.

[0066] In a specific example, by giving the communication volume and measuring the communication time, the actual communication bandwidth is calculated according to the communication time, and the quotient of the actual communication bandwidth and the theoretical bandwidth is further calculated to obtain the relationship curve between the inter-machine communication efficiency and the intra-machine communication efficiency and the communication volume.

[0067] S220. Calculate the number of segments of the communication data block according to the inter-machine communication time of the single data block and the intra-machine communication time of the single data block.

[0068] Correspondingly, after obtaining the relevant data of the inter-machine communication efficiency of the inter-machine communication mode and the intra-machine communication efficiency of the first intra-machine communication mode, the inter-machine communication time of a single data block can be calculated according to the inter-machine communication efficiency, and the intra-machine communication time of a single data block can be calculated according to the intra-machine communication efficiency, and then the number of segments of the communication data block (which can be referred to as a data block) can be further calculated according to the inter-machine communication time of a single data block and the intra-machine communication time of a single data block. Among them, the number of segments of the communication data block is also the number of segments of the sequence input to the hybrid expert model. Exemplarily, assuming that N is the number of segments of the communication data block, when N is 2, it indicates that the sequence input to the hybrid expert model is segmented into 2 blocks. When the first data block completes the first inter-machine communication as the current first inter-machine communication data block and is updated to the current first intra-machine communication data block, the second data block is updated to the current first inter-machine communication data block, and while the current first intra-machine communication data block performs intra-machine communication based on the first intra-machine communication mode, the process of the first inter-machine communication is executed in parallel.

[0069] In an optional embodiment of the present invention, the calculation of the number of segments of the communication data block based on the inter-machine communication time of the single data block and the intra-machine communication time of the single data block may include: calculating the target overall pipeline communication time based on the inter-machine communication time of the single data block and the intra-machine communication time of the single data block; calculating the minimum value of the target overall pipeline communication time, and calculating the number of segments of the communication data block based on the minimum value of the target overall pipeline communication time.

[0070] Among them, the target overall pipeline communication time can be the time for the expert to process the full amount of communication data in a single parallel communication. Specifically, it can be the time for the expert to process the full amount of communication data in a single parallel communication before the expert calculation, or it can also be the time for the expert to process the full amount of communication data in a single parallel communication after the expert calculation is completed.

[0071] Specifically, the size relationship between the inter-machine communication time of a single data block and the intra-machine communication time of a single data block can be compared, so as to determine the summing method of the inter-machine communication time of a single data block and the intra-machine communication time of a single data block according to the size relationship between the inter-machine communication time of a single data block and the intra-machine communication time of a single data block, and further compare the sum of the inter-machine communication time of a single data block and the intra-machine communication time of a single data block through the determined summing method to obtain the target overall pipeline communication time. The calculated target overall pipeline communication time includes the unknown quantity of the number of segments of the communication data block. Furthermore, the target overall pipeline communication time can be traversed and calculated in a loop, and the numbers of segments of different communication data blocks can be compared to obtain the target overall pipeline communication time under different numbers of segments of the communication data blocks, and the number of segments of the communication data blocks corresponding to the minimum value of the target overall pipeline communication time is taken. Optionally, the target overall pipeline communication time can also be referred to as the total pipeline communication time.

[0072] In an optional embodiment of the present invention, the calculation of the target overall pipeline communication time based on the single data block inter-machine communication time and the single data block intra-machine communication time may include: when it is determined that the single data block intra-machine communication time is greater than or equal to the single data block inter-machine communication time, calculating a first product value between the single data block intra-machine communication time and the number of segments of the communication data block, and calculating the sum of the single data block inter-machine communication time and the first product value to obtain the target overall pipeline communication time; when it is determined that the single data block intra-machine communication time is less than the single data block inter-machine communication time, calculating a second product value between the single data block inter-machine communication time and the number of segments of the communication data block, and calculating the sum of the single data block intra-machine communication time and the second product value to obtain the target overall pipeline communication time.

[0073] The first product value may be the product value between the intra-machine communication time of a single data block and the number of segments of the communication data block. The second product value may be the product value between the inter-machine communication time of a single data block and the number of segments of the communication data block.

[0074] In a specific example, assume that the inter-machine communication time of a single data block is the AllToAll single data block communication time, which means the time consumed by a single data block for the AllToAll communication process; the intra-machine communication time of a single data block is the AllGather single data block communication time, which means the time consumed by a single data block for the AllGather communication process. Accordingly, the AllToAll single data block communication time can be expressed by the following formula:

[0075]

[0076] The communication time of a single data block of AllGather can be expressed by the following formula:

[0077]

[0078] Among them, the communication time of a single slice of AllToAll refers to the communication time of a single data block of AllToAll, and the communication time of a single slice of AllGather refers to the communication time of a single data block of AllGather. , b is microbatch (a microbatch is a subset of a batch, usually used in pipeline parallel training to reduce the memory requirements on each computing device, and at the same time, multiple devices can process a microbatch of data in parallel), s is sequence length, h is hidden size (number of hidden units), BPE is the number of bytes occupied by each data element. N is the number of segments of the communication data block, e is the expert parallelism, t is the tensor parallelism, B1 is the AllToAll theoretical bandwidth, and R1 is The corresponding AllToAll communication efficiency, that is, according to The value of is used to find the corresponding relationship curve between the communication volume and the inter-machine communication efficiency, and the inter-machine communication efficiency is obtained. B2 is the AllGather theoretical bandwidth, and R2 is The corresponding AllGather communication efficiency is based on The value of is used to find the corresponding relationship curve between communication volume and intra-machine communication efficiency and obtain the intra-machine communication efficiency.

[0079] Correspondingly, when the time of a single slice AllGather is greater than the time of a single slice AllToAll, that is, when the communication time within a single data block is greater than or equal to the communication time between single data blocks, the total pipeline communication time can be: AllToAll single slice communication time + AllGather single slice communication time * N; when the time of a single sliceAllToAll is greater than the time of a single slice AllGather, that is, the communication time within a single data block is less than the communication time between single data blocks, the total pipeline communication time is: AllToAll single slice communication time * N + AllGather single slice communication time. By selecting a suitable N, the communication efficiency of AllToAll and AllGather is guaranteed, and the total pipeline communication time is minimized, thereby obtaining the specific value of the number of segments of the communication data block.

[0080] S230, determining a current first inter-machine communication data block and a first aligned card-machine communication data block corresponding to the current first inter-machine communication data block.

[0081] S240, establishing a first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine communication data block, and realizing a first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine communication data block based on the first current inter-machine communication connection.

[0082] S250. When it is determined that the current first inter-machine communication data block completes the first inter-machine communication, the current first inter-machine communication data block is updated to the current first intra-machine communication data block, the next first inter-machine communication data block is updated to the current first inter-machine communication data block, and the first alignment card inter-machine communication data block corresponding to the current first inter-machine communication data block is re-determined.

[0083] S260. While the current first intra-machine communication data block is performing intra-machine communication based on the first intra-machine communication mode, the first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is re-established, and the first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is implemented in parallel based on the first current inter-machine communication connection.

[0084] In a specific example, Figure 5 As shown in the figure, assuming that an input sequence is divided into two data blocks, the parallel execution of the data flow process mainly includes the following operations:

[0085] Step 1, execute the AllToAll process: the first communication data block performs inter-machine AllToAll communication, that is, data block A1 in node 1 and data block A2 in node 2, and data block B1 in node 1 and data block B2 in node 2 perform AllToAll communication.

[0086] Step 2: Execute the AllGather & AllToAll process: the first communication data block performs intra-machine AllGather communication, that is, data block G1 in node 1 performs intra-machine AllGather communication with data block G2, and data block G5 in node 2 performs intra-machine AllGather communication with data block G6. At the same time, the second data block performs inter-machine AllToAll communication, that is, data block A3 in node 1 performs AllToAll communication with data block A4 in node 2, and data block B3 in node 1 performs AllToAll communication with data block B4 in node 2. In this way, the pipeline parallelism of AllToAll and AllGather is realized.

[0087] Step 3, execute the AllGather process: the second data block performs intra-machine AllGather communication, that is, data block G3 in node 1 performs intra-machine AllGather communication with data block G4, and data block G7 in node 2 performs intra-machine AllGather communication with data block G8.

[0088] At this point, the AllToAll communication is completed, and each training card performs expert calculations on the processed communication data blocks.

[0089] Figure 7 1 is a schematic diagram of the effect of an inter-machine communication and intra-machine communication pipeline working mode provided by the second embodiment of the present invention. In a specific example, Figure 7 As shown, in the related art, the hybrid expert model first performs the inter-machine communication AlltoAll process for an input sequence, waits for the AlltoAll communication to be completed, and then sequentially performs the intra-machine communication AllGather process. The entire communication process of the input sequence requires a long communication time. In the embodiment of the present invention, by dividing the input sequence input to the hybrid expert model into multiple communication data blocks, and each communication data block executes the inter-machine communication AlltoAll process and the intra-machine communication AllGather process in parallel, the pipeline execution of the inter-machine communication AllToAll and the intra-machine communication AllGather can be achieved. Figure 7 As shown, it is assumed that the input sequence input to the hybrid expert model is divided into 4 communication data blocks, wherein the inter-machine communication AllToAll and the intra-machine communication AllGather of the same communication data block are executed serially, while the inter-machine communication AllToAll or intra-machine communication AllGather of different communication data blocks are executed in parallel. For example, after the first communication data block completes the inter-machine communication AllToAll, it continues to execute the intra-machine communication AllGather. When the first communication data block starts to execute the intra-machine communication AllGather, the second communication data block can execute the inter-machine communication AllToAll in parallel at the same time. At the same time, the second communication data block needs to wait for the first communication data block to complete the intra-machine communication AllGather before continuing to execute the intra-machine communication AllGather process, and the communication process of communication data block 3 and communication data block 4 is analogous. If the intra-machine communication AllGather takes longer than the inter-machine communication AllToAll, the subsequent communication data block needs to wait for a certain time after the inter-machine communication AllToAll is completed before executing the intra-machine communication AllGather process, and the later the order of the communication data block, the longer the waiting time interval.

[0090] S270, after determining that all communication data blocks between target machines have completed intra-machine communication based on the first intra-machine communication mode, the communication data blocks that have completed the intra-machine communication are sent to the expert network, and expert calculation is performed on the received communication data blocks through the expert network.

[0091] The total communication data blocks between target machines may be the total communication data processed by the experts in the hybrid expert model in a single parallel communication, and specifically may be the total communication data processed by the experts in a single parallel communication before the expert calculation. Exemplarily, when the input sequence of the hybrid expert model is divided into 4 blocks, the number of total communication data blocks between target machines may be 4.

[0092] Correspondingly, when all communication data blocks divided into an input sequence have completed the first intra-machine communication process, all communication data blocks between target machines corresponding to the input sequence can be sent to the corresponding expert network, so that the expert network can perform expert calculation on the received communication data blocks. In a specific example, Figure 4 As shown, after each cluster node completes the intra-machine communication AllGather process, it sends the communication data block to the corresponding expert network, so that the expert network performs expert calculation on the received communication data block.

[0093] S280: Receive the current second intra-machine communication data block sent by the expert network, and perform second intra-machine communication on the current second intra-machine communication data block based on a second intra-machine communication mode.

[0094] Among them, the current second intra-machine communication data block can be a communication data block obtained after the expert network completes the expert calculation process on the received communication data block. The second intra-machine communication can be the intra-machine communication method adopted by each cluster node after performing expert calculation on the hybrid expert model, for example, it can be the communication method of ReduceScatter (Reduce-Scatter, a communication operation in distributed computing, all participating nodes will perform some reduction operations such as summing, finding the maximum value, etc., and then distribute the reduced results to each node). The current second inter-machine communication data block can be a data block currently processed by the training card in the cluster node, and the processing process of this data block requires inter-machine communication again between the training cards corresponding to other cluster nodes.

[0095] In the embodiment of the present invention, the second inter-machine communication process in the hybrid expert model can also be converted into a mode in which inter-machine communication and intra-machine communication are executed in parallel. In a specific example, Figure 3 As shown in , the second inter-machine communication AllToAll process in the hybrid expert model can also be converted into Drop, AllToAll and AllGather. Figure 2 and Figure 4As shown in the figure, the Drop after the second AllToAll conversion is combined with AllReduce (all-reduce, a communication operation commonly used in parallel and distributed computing) and converted to ReduceScatter. If the subsequent transformer block (non-MoE modules and MoE modules in the next layer) adopts Sequence Parallel (SP, sequential parallel strategy), the AllGather after the second AllToAll conversion can be omitted, thereby reducing redundant communication volume and effectively utilizing the high-bandwidth communication resources within the machine.

[0096] Correspondingly, after the hybrid expert model completes the expert calculation on the received communication data block through the expert network, the subsequent communication processing flow can be performed on the communication data block processed by the expert network. Specifically, the communication data block can be first used as the current second intra-machine communication data block, and the second intra-machine communication can be performed on the current second intra-machine communication data block based on the second intra-machine communication mode. It can be understood that the current second intra-machine communication data block can be one of the corresponding communication data blocks after the input sequence is divided.

[0097] S290. When it is determined that the current second intra-machine communication data block completes the second intra-machine communication, the current second intra-machine communication data block is updated to the current second inter-machine communication data block, the next second intra-machine communication data block is updated to the current second intra-machine communication data block, and the second alignment card inter-machine communication data block corresponding to the current second inter-machine communication data block is determined.

[0098] The current second inter-machine communication data block may be the current second intra-machine communication data block that completes the second intra-machine communication, and the communication data block needs to undergo the inter-machine communication process again. The second inter-machine communication data block may be a communication data block that needs to undergo the second inter-machine communication process. The second alignment card inter-machine communication data block may be a data block currently processed by a training card in another cluster node, corresponding to the current second inter-machine communication data block.

[0099] If the current second intra-machine communication data block completes the second intra-machine communication, the current second intra-machine communication data block can be updated to the current second inter-machine communication data block, so as to continue to perform the second inter-machine communication process on the current second inter-machine communication data block. At the same time, the next second intra-machine communication data block can be updated to the current second intra-machine communication data block, and the second alignment card inter-machine communication data block corresponding to the current second inter-machine communication data block can be determined, so as to perform the inter-machine communication and intra-machine communication process of the communication data block in parallel again.

[0100] S2110, establishing a second current inter-machine communication connection between the current second inter-machine communication data block and the second aligned card-machine communication data block, and while realizing the second inter-machine communication between the current second inter-machine communication data block and the second aligned card-machine communication data block based on the second current inter-machine communication connection, in parallel, performing intra-machine communication on the current second intra-machine communication data block based on the second intra-machine communication method.

[0101] Among them, the second current inter-machine communication connection can be the second inter-machine communication connection established between the current training card and the alignment card, or the second current inter-machine communication connection can also be other subsequent inter-machine communication connections established between the current training card and the alignment card except the first inter-machine communication connection. The second inter-machine communication can be the inter-machine communication between the current second inter-machine communication data block and the second alignment card inter-machine communication data block based on the established second inter-machine communication connection or other subsequent inter-machine communication connections. Optionally, the communication mode of the second inter-machine communication can be, for example, an inter-machine communication mode in the form of AllToAll, etc., and the embodiment of the present invention does not limit the specific communication mode of the second inter-machine communication. The second intra-machine communication mode can be the intra-machine communication mode adopted by each cluster node after performing expert calculation on the hybrid expert model. It can be understood that the cluster node performs the second inter-machine communication between the current second inter-machine communication data block and the second alignment card inter-machine communication data block after performing expert calculation.

[0102] Specifically, in order to implement the inter-machine communication and intra-machine communication process of the communication data block in parallel again, a second current inter-machine communication connection can be established between the current second inter-machine communication data block and the second aligned card-machine communication data block, so as to execute the second inter-machine communication process between the current second inter-machine communication data block and the second aligned card-machine communication data block through the established second current inter-machine communication connection. In the process of executing the second inter-machine communication on the current second inter-machine communication data block and the second aligned card-machine communication data block, the intra-machine communication process can be performed on the current second intra-machine communication data block in parallel based on the second intra-machine communication method.

[0103] In a specific example, Figure 5As shown, assuming that GPU1 is the current training card, data block G1 is used as the current second intra-machine communication data block after the complete expert calculation, and data block G3 is used as the next intra-machine communication data block after the complete expert calculation. When GPU1 completes the first inter-machine communication process for data block G1, data block G1 is updated to the current second intra-machine communication data block R1, and the second intra-machine communication process is continued for the current second intra-machine communication data block R1, such as the ReduceScatter communication process. After the current second intra-machine communication data block R1 completes the second intra-machine communication process, it is updated to the current second intra-machine communication data block C1, and the next second intra-machine communication data block R3 is updated to the current second intra-machine communication data block, and the second alignment card inter-machine communication data block C2 corresponding to the current second inter-machine communication data block C1 is determined. Further, GPU1 can continue to implement the second inter-machine communication process for the current second inter-machine communication data block C1 and the second alignment card inter-machine communication data block C2, and at the same time, perform the intra-machine communication process for the current second intra-machine communication data block R3 based on the second intra-machine communication method in parallel.

[0104] It should be noted that multiple inter-machine communication processes may be required in the subsequent process after the expert calculation of each communication data block is completed. Each subsequent inter-machine communication process can be converted into Drop, AllToAll and AllGather methods using the above method, thereby realizing a pipeline communication method and achieving overlapping coverage of inter-machine communication and intra-machine communication, thereby improving communication efficiency and reducing communication delay.

[0105] In an optional embodiment of the present invention, the hybrid expert model communication optimization method may also include: running the kernel function of inter-machine communication through the first data stream; running the kernel function of intra-machine communication through the second data stream; wherein the intra-machine communication may include first intra-machine communication and second intra-machine communication; running the kernel function of target calculation through the third data stream; wherein the target calculation may include expert calculation and slice calculation; wherein each data stream is executed serially internally, and the timing between the data streams is synchronized through a synchronization mechanism.

[0106] The first data stream, the second data stream and the third data stream may be three different data streams, for example, three different streams (stream is an abstract concept for processing data sequences, allowing data sets to be processed in a declarative manner).

[0107] Since inter-machine communication uses inter-machine bandwidth communication, and intra-machine communication (including first intra-machine communication and second intra-machine communication) uses intra-machine bandwidth communication, in the embodiment of the present invention, different streams can be used to decouple each communication process and computing process. Specifically, the kernel function of inter-machine communication can be run through the first data stream, the kernel functions of intra-machine communication such as the first intra-machine communication and the second intra-machine communication can be run through the second data stream, and the kernel functions of target computing such as expert computing and slice computing can be run through the third data stream. The computing processes within different data streams can be executed serially, and the timing between data streams can be synchronized through a synchronization mechanism.

[0108] Figure 8 Schematic diagram of the effect of multiple data streams running provided by the second embodiment of the present invention. Figure 8 As shown, assuming that the input sequence is divided into two data blocks, Figure 8 Steam1 can be the first data stream, steam2 can be the second data stream, and steam3 can be the third data stream. Inside steam1, the first two AllToAlls represent the communication data blocks processed by the two AllToAlls before the expert calculation, and the last two AllToAlls represent the communication data blocks processed by the two AllToAlls after the expert calculation. Stream1 runs the kernel (kernel function) of inter-machine communication AllToAll, stream2 runs the kernel of intra-machine communication including AllGather / ReduceScatter communication, and stream3 runs the kernel of calculation, including expert calculation and unslice (data slicing) operations. Each stream is executed in parallel, and the internal stream is executed serially. Figure 8 The arrows between different communication data blocks represent event (a mechanism for record synchronization) dependencies. For example, the AllGather process of the communication data block in stream2 depends on the AllToAll process before the expert calculation of stream1. The expert calculation process depends on the completion of the AllGather process of all data blocks before the expert calculation can be performed. The ReduceScatter process in stream2 depends on the expert calculation result, and the AllToAll process after the expert calculation depends on the ReduceScatter process. The data slicing operation of stream3 can only be performed after the second AllToAll of all communication data blocks is completed.

[0109] In an optional embodiment of the present invention, the hybrid expert model communication optimization method may also include: when a communication conflict occurs in inter-machine communication and / or intra-machine communication, obtaining communication primitive priority information; resolving the communication conflict according to the communication primitive priority information; wherein the communication primitive priority information includes a first communication priority of expert parallelism, a second communication priority of pipeline parallelism, a third communication priority of context parallelism, and a fourth communication priority of data parallelism; the first communication priority is higher than the second communication priority, the second communication priority is higher than the third communication priority, and the third communication priority is higher than the fourth communication priority.

[0110] Among them, the communication primitive priority information may be priority information set for different communication primitives. The first communication priority may be the priority for expert parallel strategy configuration. The second communication priority may be the priority for pipeline parallel strategy configuration. The third communication priority may be the priority for context parallel strategy configuration. The fourth communication priority may be the priority for data parallel strategy configuration.

[0111] In an optional embodiment of the present invention, before obtaining the communication primitive priority information, it may also include: sending the communication kernel functions of different communication primitives to the execution unit through the scheduling unit; setting corresponding priority information for the communication QP of different kernel functions according to the communication primitive priority information through the execution unit; and configuring communication for the communication unit ports corresponding to the communication QP of different kernel functions.

[0112] In the existing MoE training framework, a distributed training method of 6D parallel training is involved. The so-called 6D parallel training is to use EP (Expert Parallel), DP (Data Parallel), PP (Pipeline Parallel), SP (Sequence Parallel), CP (ContextParallel) and TP (Tensor Parallel) at the same time. In the process of distributed parallel training of hybrid expert models, different parallel training strategies will cause asynchronous communication conflicts. Therefore, in order to ensure the training effect of the hybrid expert model, before the distributed parallel training of the hybrid expert model, the communication primitive priority information can be configured for it first. In this way, during the training of the hybrid expert model, if there is a communication conflict between machines and / or within the machine, the communication primitive priority information can be obtained, and the communication conflict can be resolved according to the obtained communication primitive priority information.

[0113] Fig. 9is a schematic diagram of a communication semantic priority configuration process provided by Embodiment 2 of the present invention. In a specific example, Fig. 9 As shown, the hardware implementation method for configuring the priority information of the communication primitives can be: the communication kernels of different communication primitives (the above-mentioned EP, DP, CP and DP etc. all have corresponding kernel functions) are sent to the execution unit (execution kernel function) by the scheduling unit (a hardware module on the GPU card, used to complete kernel scheduling). The execution unit sets different priority information for the QueuePair (communication QP is the basic unit for data transmission between two endpoints (for example, two nodes) and can be used to manage data exchange between GPUs to achieve efficient collective communication operations) of the communication kernel functions of different communication primitives according to the priority configuration information of the communication primitives, and at the same time, performs communication configuration on the communication unit ports corresponding to the communication QPs of different kernel functions.

[0114] Fig.10 Schematic diagram of the effect of configuring communication priorities for different parallel strategies provided by the second embodiment of the present invention. Fig.10 As shown, the communication primitive priority information can be specifically: the first communication priority of expert parallel is higher than the second communication priority of pipeline parallel, the second communication priority of pipeline parallel is higher than the third communication priority of context parallel, and the third communication priority of context parallel is higher than the fourth communication priority of data parallel. That is, when the EP AllToAll communication of MoE conflicts with the AllReduce communication of data parallel, AllToAll communication takes precedence. When the Context parallel communication (CP) conflicts with the data (DP) parallel AllReduce communication, the Send / Recv of the Context Parallel communication takes precedence. Other communication conflicts are analogous.

[0115] The above technical solution divides the data into different communication data blocks according to the cluster communication bandwidth, and uses different data streams to isolate the inter-machine communication and intra-machine communication processes, and combines pipeline technology to achieve overlapping coverage of inter-machine communication and intra-machine communication. Since the intra-machine bandwidth is much larger than the inter-machine bandwidth, combining pipeline technology to achieve overlapping coverage of inter-machine communication and intra-machine communication can achieve efficient utilization of communication resources, improve computing power utilization and communication efficiency, reduce latency, and thus improve the training efficiency of the hybrid expert model. At the same time, by setting different communication priorities for communication primitives, conflicts caused by asynchronous communication can be reduced, ensuring the inter-machine communication efficiency of the hybrid expert model.

[0116] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0117] It should be noted that any arrangement and combination of the technical features in the above embodiments also falls within the protection scope of the present invention.

[0118] Embodiment 3

[0119] Fig.11 is a schematic diagram of a hybrid expert model communication optimization device provided by Embodiment 3 of the present invention, such as Fig.11 As shown, the device includes: a first inter-machine communication data block determination module 310, a first inter-machine communication module 320, a first inter-machine intra-machine data block update module 330 and a first inter-machine intra-machine parallel communication module 340, wherein:

[0120] A first inter-machine communication data block determining module 310 is used to determine a current first inter-machine communication data block and a first aligned card inter-machine communication data block corresponding to the current first inter-machine communication data block;

[0121] A first inter-machine communication module 320 is used to establish a first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card inter-machine communication data block, and to implement a first inter-machine communication between the current first inter-machine communication data block and the first aligned card inter-machine communication data block based on the first current inter-machine communication connection;

[0122] The first inter-machine internal data block updating module 330 is used to update the current first inter-machine communication data block to the current first internal communication data block, update the next first inter-machine communication data block to the current first inter-machine communication data block, and re-determine the first alignment card inter-machine communication data block corresponding to the current first inter-machine communication data block when it is determined that the current first inter-machine communication data block completes the first inter-machine communication;

[0123] The first inter-machine intra-machine parallel communication module 340 is used to re-establish the first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block while the current first inter-machine communication data block is performing intra-machine communication based on the first intra-machine communication mode, and to implement the first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block in parallel based on the first current inter-machine communication connection.

[0124] The embodiment of the present invention establishes a first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block after determining the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block, and implements the first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block based on the first current inter-machine communication connection. When it is determined that the current first inter-machine communication data block completes the first inter-machine communication, the current first inter-machine communication data block is updated to the current first intra-machine communication data block, the next first inter-machine communication data block is updated to the current first inter-machine communication data block, and the first aligned card-machine inter-machine communication data block corresponding to the current first inter-machine communication data block is re-determined. While the current first intra-machine communication data block performs intra-machine communication based on the first intra-machine communication mode, the first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is re-established, and the first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is implemented in parallel based on the first current inter-machine communication connection. The above technical solution can solve the problem of low communication efficiency in the existing hybrid expert model, reduce the resource waiting when the hybrid expert model processes data, thereby optimizing the communication efficiency and computing speed of the hybrid expert model, and further improve the hardware resource utilization and model execution efficiency when the hybrid expert model processes data.

[0125] Optionally, the above-mentioned device also includes a split quantity calculation module, which is used to: test the inter-machine communication efficiency of the inter-machine communication mode and the intra-machine communication efficiency of the first intra-machine communication mode; calculate the inter-machine communication time of a single data block according to the inter-machine communication efficiency, and calculate the intra-machine communication time of a single data block according to the intra-machine communication efficiency; calculate the number of splits of the communication data block according to the inter-machine communication time of the single data block and the intra-machine communication time of the single data block.

[0126] Optionally, the split quantity calculation module is also used to: calculate the target overall pipeline communication time based on the inter-machine communication time of the single data block and the intra-machine communication time of the single data block; calculate the minimum value of the target overall pipeline communication time, and calculate the number of splits of the communication data block based on the minimum value of the target overall pipeline communication time.

[0127] Optionally, the split quantity calculation module is also used to: when it is determined that the intra-machine communication time of the single data block is greater than or equal to the inter-machine communication time of the single data block, calculate the first product value between the intra-machine communication time of the single data block and the number of splits of the communication data block, and calculate the sum of the single data block inter-machine communication time and the first product value to obtain the target overall pipeline communication time; when it is determined that the intra-machine communication time of the single data block is less than the inter-machine communication time of the single data block, calculate the second product value between the single data block inter-machine communication time and the number of splits of the communication data block, and calculate the sum of the intra-machine communication time of the single data block and the second product value to obtain the target overall pipeline communication time.

[0128] Optionally, the above-mentioned device also includes an expert calculation module, which is used to: after determining that all communication data blocks between target machines have completed intra-machine communication based on the first intra-machine communication method, send the communication data blocks that have completed the intra-machine communication to the expert network; and perform expert calculation on the received communication data blocks through the expert network.

[0129] Optionally, the above-mentioned device also includes: a second intra-machine communication module, which is used to receive the current second intra-machine communication data block sent by the expert network, and perform intra-machine communication on the current second intra-machine communication data block based on the second intra-machine communication mode; a second inter-machine intra-machine data block update module, which is used to update the current second intra-machine communication data block to the current second inter-machine communication data block and the next second intra-machine communication data block to the current second intra-machine communication data block when it is determined that the current second intra-machine communication data block completes the second intra-machine communication, and determine the second alignment card-machine communication data block corresponding to the current second inter-machine communication data block; a second inter-machine intra-machine parallel communication module, which is used to establish a second current inter-machine communication connection between the current second inter-machine communication data block and the second alignment card-machine communication data block, and while realizing the second inter-machine communication between the current second inter-machine communication data block and the second alignment card-machine communication data block based on the second current inter-machine communication connection, perform intra-machine communication on the current second intra-machine communication data block in parallel based on the second intra-machine communication mode.

[0130] Optionally, the above-mentioned device also includes a kernel function running module, which is used to: run the kernel function of inter-machine communication through the first data stream; run the kernel function of intra-machine communication through the second data stream; wherein the intra-machine communication includes first intra-machine communication and second intra-machine communication; run the kernel function of target calculation through the third data stream; wherein the target calculation includes expert calculation and slice calculation; wherein each data stream is executed serially internally, and the timing between the data streams is synchronized through a synchronization mechanism.

[0131] Optionally, the above-mentioned device also includes a communication conflict resolution module, which is used to: obtain communication primitive priority information when a communication conflict occurs in inter-machine communication and / or intra-machine communication; resolve the communication conflict according to the communication primitive priority information; wherein the communication primitive priority information includes a first communication priority of expert parallelism, a second communication priority of pipeline parallelism, a third communication priority of context parallelism, and a fourth communication priority of data parallelism; the first communication priority is higher than the second communication priority, the second communication priority is higher than the third communication priority, and the third communication priority is higher than the fourth communication priority.

[0132] Optionally, the above-mentioned device also includes a communication configuration module, which is used to: send communication kernel functions of different communication primitives to the execution unit through the scheduling unit; set corresponding priority information for the communication QP of different kernel functions according to the communication primitive priority information through the execution unit; and perform communication configuration on the communication unit ports corresponding to the communication QP of different kernel functions.

[0133] The hybrid expert model communication optimization device can execute the hybrid expert model communication optimization method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the hybrid expert model communication optimization method provided by any embodiment of the present invention.

[0134] Since the hybrid expert model communication optimization device introduced above is a device that can execute the hybrid expert model communication optimization method in the embodiment of the present invention, based on the hybrid expert model communication optimization method introduced in the embodiment of the present invention, the technical personnel of the field can understand the specific implementation of the hybrid expert model communication optimization device of this embodiment and its various variations, so how the hybrid expert model communication optimization device implements the hybrid expert model communication optimization method in the embodiment of the present invention is not introduced in detail here. As long as the technical personnel of the field implement the device adopted by the hybrid expert model communication optimization method in the embodiment of the present invention, it belongs to the scope of protection of this application.

[0135] Embodiment 4

[0136] Fig.12A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0137] like Fig.12 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0138] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0139] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the hybrid expert model communication optimization method.

[0140] Optionally, the hybrid expert model communication optimization method may include: determining a current first inter-machine communication data block and a first aligned card-machine communication data block corresponding to the current first inter-machine communication data block;

[0141] Establishing a first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine communication data block, and realizing a first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine communication data block based on the first current inter-machine communication connection;

[0142] When it is determined that the current first inter-machine communication data block completes the first inter-machine communication, the current first inter-machine communication data block is updated to the current first intra-machine communication data block, the next first inter-machine communication data block is updated to the current first inter-machine communication data block, and the first alignment card inter-machine communication data block corresponding to the current first inter-machine communication data block is re-determined;

[0143] While the current first intra-machine communication data block is performing intra-machine communication based on the first intra-machine communication mode, the first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is re-established, and the first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is implemented in parallel based on the first current inter-machine communication connection.

[0144] In some embodiments, the hybrid expert model communication optimization method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the hybrid expert model communication optimization method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the hybrid expert model communication optimization method in any other appropriate manner (e.g., by means of firmware).

[0145] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0147] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0148] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0149] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0150] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0151] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0152] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A hybrid expert model communication optimization method, characterized in that: include: Determine a current first inter-machine communication data block and a first alignment card inter-machine communication data block corresponding to the current first inter-machine communication data block; Establishing a first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine communication data block, and realizing a first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine communication data block based on the first current inter-machine communication connection; When it is determined that the current first inter-machine communication data block completes the first inter-machine communication, the current first inter-machine communication data block is updated to the current first intra-machine communication data block, the next first inter-machine communication data block is updated to the current first inter-machine communication data block, and the first alignment card inter-machine communication data block corresponding to the current first inter-machine communication data block is re-determined; While the current first intra-machine communication data block is performing intra-machine communication based on the first intra-machine communication mode, the first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is re-established, and the first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block is implemented in parallel based on the first current inter-machine communication connection.

2. The method according to claim 1, characterized in that Before determining the current first inter-machine communication data block and the first aligned card inter-machine communication data block corresponding to the current first inter-machine communication data block, the method further includes: Testing the inter-machine communication efficiency of the inter-machine communication mode and the intra-machine communication efficiency of the first intra-machine communication mode; Calculating the inter-machine communication time of a single data block according to the inter-machine communication efficiency, and calculating the intra-machine communication time of a single data block according to the intra-machine communication efficiency; The number of segments of the communication data block is calculated according to the inter-machine communication time of the single data block and the intra-machine communication time of the single data block.

3. The method according to claim 2, characterized in that The calculating the number of segments of the communication data block according to the inter-machine communication time of the single data block and the intra-machine communication time of the single data block includes: Calculate the target overall pipeline communication time according to the inter-machine communication time of the single data block and the intra-machine communication time of the single data block; The minimum value of the target overall pipeline communication time is calculated, and the number of segments of the communication data block is calculated based on the minimum value of the target overall pipeline communication time.

4. The method according to claim 3, characterized in that The calculating the target overall pipeline communication time according to the inter-machine communication time of the single data block and the intra-machine communication time of the single data block includes: In the case where it is determined that the intra-machine communication time of the single data block is greater than or equal to the inter-machine communication time of the single data block, a first product value between the intra-machine communication time of the single data block and the number of segments of the communication data block is calculated, and a sum of the inter-machine communication time of the single data block and the first product value is calculated to obtain the target overall pipeline communication time; When it is determined that the intra-machine communication time of the single data block is less than the inter-machine communication time of the single data block, a second product value between the inter-machine communication time of the single data block and the number of divisions of the communication data block is calculated, and the sum of the intra-machine communication time of the single data block and the second product value is calculated to obtain the target overall pipeline communication time.

5. The method according to claim 2, characterized in that: The method further comprises: After determining that all communication data blocks between target machines have completed intra-machine communication based on the first intra-machine communication mode, the communication data blocks that have completed the intra-machine communication are sent to the expert network; The expert network performs expert calculation on the received communication data block.

6. The method according to claim 5, characterized in that After performing expert calculation on the received communication data block through the expert network, the method further includes: receiving a current second intra-machine communication data block sent by the expert network, and performing a second intra-machine communication on the current second intra-machine communication data block based on a second intra-machine communication mode; When it is determined that the current second intra-machine communication data block completes the second intra-machine communication, the current second intra-machine communication data block is updated to the current second inter-machine communication data block, the next second intra-machine communication data block is updated to the current second intra-machine communication data block, and the second alignment card inter-machine communication data block corresponding to the current second inter-machine communication data block is determined; A second current inter-machine communication connection is established between the current second inter-machine communication data block and the second aligned card-machine communication data block, and while realizing the second inter-machine communication between the current second inter-machine communication data block and the second aligned card-machine communication data block based on the second current inter-machine communication connection, the current second intra-machine communication data block is simultaneously subjected to intra-machine communication based on the second intra-machine communication method.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Running a kernel function of inter-machine communication through a first data stream; Run the kernel function of the intra-machine communication through the second data stream; wherein the intra-machine communication includes the first intra-machine communication and the second intra-machine communication; Running a kernel function of a target calculation through a third data stream; wherein the target calculation includes an expert calculation and a slice calculation; Among them, each data stream is executed serially internally, and the timing between data streams is synchronized through a synchronization mechanism.

8. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: In case of communication conflicts in inter-machine communication and / or intra-machine communication, obtaining communication primitive priority information; Resolving the communication conflict according to the communication primitive priority information; Among them, the communication primitive priority information includes a first communication priority of expert parallelism, a second communication priority of pipeline parallelism, a third communication priority of context parallelism and a fourth communication priority of data parallelism; the first communication priority is higher than the second communication priority, the second communication priority is higher than the third communication priority, and the third communication priority is higher than the fourth communication priority.

9. The method according to claim 8, characterized in that Before obtaining the communication primitive priority information, the method further includes: The communication kernel functions of different communication primitives are sent to the execution unit through the scheduling unit; Setting corresponding priority information for communication QPs of different kernel functions according to the communication primitive priority information by an execution unit; Perform communication configuration on the communication unit ports corresponding to the communication QPs of different kernel functions.

10. A hybrid expert model communication optimization device, characterized in that: include: A first inter-machine communication data block determining module, used to determine a current first inter-machine communication data block and a first alignment card inter-machine communication data block corresponding to the current first inter-machine communication data block; A first inter-machine communication module, used for establishing a first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card inter-machine communication data block, and realizing a first inter-machine communication between the current first inter-machine communication data block and the first aligned card inter-machine communication data block based on the first current inter-machine communication connection; A first inter-machine internal data block updating module is used to update the current first inter-machine communication data block to the current first internal communication data block, update the next first inter-machine communication data block to the current first inter-machine communication data block, and re-determine the first alignment card inter-machine communication data block corresponding to the current first inter-machine communication data block when it is determined that the current first inter-machine communication data block completes the first inter-machine communication; The first inter-machine intra-machine parallel communication module is used to re-establish the first current inter-machine communication connection between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block while the current first inter-machine communication data block performs intra-machine communication based on the first intra-machine communication mode, and to implement the first inter-machine communication between the current first inter-machine communication data block and the first aligned card-machine inter-machine communication data block in parallel based on the first current inter-machine communication connection.

11. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the hybrid expert model communication optimization method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the hybrid expert model communication optimization method described in any one of claims 1-9 when executed.

13. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the hybrid expert model communication optimization method according to any one of claims 1 to 9 is implemented.

Images