Data balanced distribution method, electronic device and storage medium based on MOE scenario

Abstract

The present invention relates to the field of chip design technology, and in particular to a data balanced distribution method, electronic device and storage medium based on a MOE scenario. The average data volume avg of the averagely distributed word unit at each expert is calculated according to the total number of experts G in the hybrid expert model MOE, the data sequence length U of the training text processed in each iteration and the number of experts assigned to process each word unit Q, and the upper limit threshold th0 of each expert assigned word unit is obtained according to the avg. The actual data volume M of each expert assigned word unit is obtained, and the data volume of the target word unit assigned by each expert is adjusted to th0 according to the adjustment strategy formulated by M and th0, thereby solving the problem of unbalanced calculation and communication.

Description

Technical Field

[0001] The present invention relates to the technical field of chip design, and in particular to a data balanced distribution method, electronic device and storage medium based on a MOE scenario. Background Art

[0002] Mixture of Experts (MoE) is an integrated learning technology and machine learning method developed in the field of neural networks, mainly used to improve the computational efficiency and performance of the model. The core idea is to divide the model into multiple experts, each of which focuses on processing a specific type of data or task; then use the gating network to select some experts to process the input data instead of letting all experts process the data at the same time. This method enables the model to dynamically select the most suitable experts to participate in the calculation when processing large-scale data, so as to maximize resource utilization. Among them, the gating network is a module used to determine which expert processes the input data. It dynamically selects appropriate experts to process the data based on the characteristics of the input. For example, if the input is a picture, the gating network may select some experts who specialize in image processing.

[0003] In actual training, there are the following defects:

[0004] First, some experts receive and process significantly more token data than other experts. These experts have long calculation times and heavy loads, which leads to the problem of unbalanced calculations.

[0005] Second, due to the unbalanced load distribution, when each expert transmits data to other nodes, the amount of data transmitted varies greatly, resulting in the problem of unbalanced communication. Summary of the invention

[0006] In view of the above technical problems, the technical solution adopted by the present invention is: a method for balanced data distribution based on the MOE scenario, the method comprising the following steps: obtaining the total number of experts G in the hybrid expert model MOE. Obtaining the data sequence length of the training text processed in each iteration as U words. Obtaining the number of experts assigned to process each word Q. Calculating the average data volume avg of the word evenly distributed at each expert based on the G, U and Q, and obtaining the upper limit threshold th0 of the word allocated by each expert based on the avg. Obtaining the actual data volume M of the word allocated by each expert, and adjusting the data volume of the target word allocated by each expert to th0 according to the adjustment strategy, the adjustment strategy is: when the M is greater than the th0, delete (M-th0) words; when M is less than th0, add (th0-M) fillers.

[0007] In addition, the present invention also provides a non-transitory computer-readable storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the above method.

[0008] In addition, the present invention also provides an electronic device, including a processor and the above-mentioned non-transitory computer-readable storage medium.

[0009] The present invention has at least the following beneficial effects:

[0010] The data balanced distribution method, electronic device and storage medium based on the MOE scenario provided by the embodiments of the present invention adjust the data volume of all experts to the same level through the online threshold and combined with the adjustment strategy, thereby achieving a balance in the task volume between different experts, that is, the amount of data that each expert needs to process is the same, and the amount of data transmitted between nodes is also the same, thereby solving the problem of unbalanced communication load; since the data volume of each expert is th0, there is no expert who processes significantly more data than other experts, thereby solving the problem of unbalanced calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A flow chart of a data balanced distribution method based on an MOE scenario provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0014] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present invention have the same meanings as commonly understood by those skilled in the art.

[0015] See also Figure 1 , which shows a flow chart of a data balanced distribution method based on MOE scenario, the method comprising the following steps:

[0016] S100, obtaining the total number G of experts in the hybrid expert model MOE.

[0017] The Mixture of Experts (MoE) model includes G experts. Each expert is a different machine learning model, algorithm or computing unit, and each expert focuses on processing a specific type of data or task.

[0018] It should be noted that the workflow of MOE usually includes: the MOE model receives input data; the gating network calculates the probability of each expert model being selected based on the characteristics of the input data. According to the output of the gating network, one or more expert models with the highest probability are selected for subsequent processing; the selected expert model processes the input data and generates corresponding output; if multiple expert models are selected, their outputs need to be weighted and aggregated to obtain the final output result.

[0019] S200, obtaining a data sequence length of U words of the training text processed in each iteration.

[0020] In the training process of MOE, training texts are usually input in batches to improve training efficiency and utilize parallel computing capabilities. That is, MOE processes a batch of data each iteration.

[0021] A token is a basic unit in a text after processing, and the basic unit may be a word, a punctuation mark, a number or any other meaningful character sequence. Other types of tokens also fall within the protection scope of the present invention.

[0022] In one embodiment, the step of obtaining the length of the data sequence includes: before the training text is input into the MOE, the training text is segmented by a tokenizer to obtain word units, and the number of word units is counted to obtain U.

[0023] In one implementation, the word segmenter is WordPiece of BERT or Byte-Pair Encoding of GPT. Other types of word segmenters also fall within the protection scope of the present invention.

[0024] S300, obtaining the number Q of experts assigned to process each word.

[0025] Among them, in the parallel processing scenario of MOE, especially when processing large-scale data, it is not completely processed by a single expert, but through the corresponding distribution strategy, each input word needs to be processed by Q experts instead of all experts, and the distributed expert system is used to improve the computing efficiency and effect. The architecture of MOE can process data in parallel, and each expert calculates in parallel, and each expert only processes part of the assigned input.

[0026] In one embodiment, the allocation strategy for assigning tokens to various experts is a gating network, which assigns an expert to each word. The gating network receives all tokens in a whole batch of text data as input, generates scores of multiple experts for each input token, usually outputs a probability distribution through a softmax function, and assigns it to the k experts with the highest scores.

[0027] In another embodiment, the allocation strategy for allocating tokens to various experts is random selection, which can avoid expert overload and enhance the diverse use of experts by randomly allocating input tokens to certain experts instead of always based on the highest score.

[0028] The allocation strategy may also be feature-based selection, load balancing routing, etc. Other types of allocation strategies for allocating tokens to various experts fall within the protection scope of the present invention.

[0029] S400, calculating the average data volume avg of the averagely distributed word-units at each expert according to the G, U and Q, and obtaining the upper limit threshold th0 of the word-units distributed by each expert according to the avg.

[0030] Each word needs to be assigned to Q experts, that is, each word provides a processing task for Q experts. Since each token is processed by Q experts, the total amount of data is: Q×U.

[0031] When the load is evenly distributed, the amount of data processed by each expert is: Q×U / G. That is, avg=Q×U / G.

[0032] In one embodiment, th0=avg. In another embodiment, th0=avg±t, where t is a preset adjustment factor. Other ways of configuring th0 according to avg also fall within the protection scope of the present invention.

[0033] It should be noted that the upper threshold ensures that when allocating data to different experts, the amount of work for any expert will not exceed this standard, thus avoiding huge differences in the amount of data between individuals and providing a basis for subsequent balanced allocation.

[0034] S500, obtaining the actual data volume M of each expert-assigned word unit, and adjusting the data volume of each expert-assigned target word unit to th0 according to the adjustment strategy, wherein the adjustment strategy is: when the M is greater than the th0, deleting (M-th0) word units; when M is less than th0, adding (th0-M) fillers.

[0035] Among them, the pad token is a special symbol used to fill the amount of data assigned by each expert. MOE usually ignores the pad token. The pad token only serves as a placeholder in the input layer and is only used to align the sequence length, but will not be used by the model in actual training or reasoning. Usually, the model will skip these pad tokens when calculating the loss.

[0036] It should be noted that by adjusting the data volume of all experts to the same level through the adjustment strategy, the balance of task volume among different experts is achieved, that is, the amount of data that each expert needs to process is the same, and the amount of data transmitted between nodes is also the same, thus solving the problem of unbalanced communication load; since the data volume of each expert is th0, there is no expert who processes significantly more data than other experts, thus solving the problem of unbalanced computing.

[0037] In one embodiment, in MOE, each expert is responsible for processing different parts of the input data, usually performing matrix operations (such as GEMM), which need to be efficiently divided into thread blocks and thread groups when executed on the GPU. The method also includes a split optimization step:

[0038] S600, obtaining attribute information shape of the input data according to th0 and dimension of the input data.

[0039] It should be noted that each expert will receive a portion of the input token data and perform calculations within the expert. Although the actual data received by each expert may have a different shape, after adjusting the strategy, the shape of all experts is the same, and each expert performs specific calculation tasks based on the same shape.

[0040] Among them, shape is a multidimensional vector, including th0 and dimension, and may also include other component information.

[0041] S700: query a segmentation strategy list according to the shape to obtain a target segmentation strategy for segmenting the data. The target segmentation strategy is used to segment the data to obtain multiple data blocks.

[0042] In one embodiment, the segmentation strategy list includes a preset range of shape and a preset segmentation strategy bound to the shape. The preset range of shape refers to the preset range of each component information in the shape. When querying, each component of the shape needs to fall within each preset range of the shape.

[0043] It should be noted that when performing GEMM operations on the GPU, in order to fully utilize the computing resources of the GPU, it is necessary to divide the large matrix into smaller blocks, and one or more thread groups will process all blocks of a matrix. When selecting a splitting strategy, the GPU will select the optimal block size and thread group configuration based on the shape. Therefore, choosing a suitable splitting strategy can improve the computational efficiency within the expert and the overall training efficiency of the MOE model. If the amount of data for each expert is uncertain, the optimal splitting strategy cannot be selected. Because different shapes will lead to different workloads, the GPU kernel needs to select the optimal block size, thread group configuration, etc. according to the shape when executing tasks. If the shape of the input data is not clear, the data cannot be reasonably divided into different threads and blocks, resulting in low resource utilization or unbalanced load.

[0044] In one embodiment, the relevant computing tasks and parameter storage of the optimizer are transferred from the computing unit to the central processor.

[0045] Among them, the optimizer is a key component used to update the model weights during deep learning training. In each iteration, the optimizer adjusts the model weights based on the gradient information so that the loss function gradually decreases. The optimizer usually needs to store a large number of intermediate parameters, such as momentum, gradient, etc., which consume a lot of video memory.

[0046] In one embodiment, the computing unit is a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU) or a neural processing unit (NPU), and other computing units for processing data fall within the scope of protection of the present invention.

[0047] In one embodiment, the optimizer is Adam or SGD. Other types of optimizers also fall within the protection scope of the present invention.

[0048] It should be noted that transferring the relevant computing tasks and parameter storage of the optimizer from the computing unit to the central processor can reduce the burden on the computing unit and release the video memory of the computing unit, so that the model that originally may require multiple computing units to run can now use fewer computing units to complete the training task. At the same time, the communication requirements between computing units are reduced, which reduces the communication overhead across nodes and makes the overall training efficiency higher.

[0049] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiment.

[0050] An embodiment of the present invention further provides an electronic device, comprising a processor and the aforementioned non-transitory computer-readable storage medium.

[0051] An embodiment of the present invention further provides a computer program product, which includes program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method according to various exemplary embodiments of the present invention described above in this specification.

[0052] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0053] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. The data balanced distribution method based on MOE scenario is characterized by: The method comprises the following steps: Get the total number of experts G in the hybrid expert model MOE; The length of the data sequence of the training text obtained for each iteration is U words; Get the number of experts assigned to process each word, Q; Calculate the average data volume avg of the word-units allocated by each expert on average according to G, U and Q, and obtain the upper limit threshold th0 of the word-units allocated by each expert according to avg; The actual data volume M of each expert-assigned word unit is obtained, and the data volume of each expert-assigned target word unit is adjusted to th0 according to the adjustment strategy. The adjustment strategy is: when M is greater than th0, (M-th0) word units are deleted; when M is less than th0, (th0-M) fillers are added.

2. The method according to claim 1, characterized in that The step of obtaining the data sequence length includes: before the training text is input into the MOE, the training text is segmented by a word segmenter to obtain the word units, and the number of the word units is counted to obtain the U.

3. The method according to claim 1, characterized in that The allocation strategy for assigning word units to various experts is a gating network.

4. The method according to claim 1, characterized in that: The relationship between avg and th0 satisfies: th0=avg, or th0=avg±t, where t is a preset adjustment factor.

5. The method according to claim 1, characterized in that The method also includes a segmentation optimization step: Get the attribute information shape of the input data according to th0 and dimension of the input data; Query the segmentation strategy list according to the shape to obtain a target segmentation strategy for segmenting the data; The target segmentation strategy is used to segment data to obtain multiple data blocks.

6. The method according to claim 5, characterized in that The segmentation strategy list includes a preset range of shapes and a preset segmentation strategy bound to the shapes.

7. The method according to claim 1, characterized in that Transfer the optimization-related computational tasks and parameter storage from the computational unit to the central processor.

8. The method according to claim 7, characterized in that The computing unit is a graphics processing unit, a general-purpose graphics processing unit or a neural processing unit.

9. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: The invention comprises a processor and the non-transitory computer-readable storage medium as claimed in claim 9.

Images