A method, apparatus and device for optimizing a hybrid expert model

Abstract

Embodiments of the present application disclose a method, device and equipment for optimizing a hybrid expert model. In the embodiments of the present application, the average number of tokens of the hybrid expert model is obtained, wherein the token is a basic unit for the hybrid expert model to process input data; the N times of the average number of tokens of the hybrid expert model are determined, wherein the N is an integer or a decimal number greater than 1; and the target video memory is applied according to the N times of the average number of tokens, wherein the target video memory is used to store tokens in a forward propagation process. Through the above method, the N times of the average number of tokens are applied when the video memory is applied in the forward propagation, so that the video memory fragmentation problem can be solved, the video memory occupation of the hybrid expert model is reduced, and the optimization of the hybrid expert model is realized.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a method, apparatus, and device for optimizing hybrid expert models. Background Technology

[0002] With the rapid development of artificial intelligence, large language models (LLMs) involved in artificial intelligence have also developed rapidly. For example, the development of fully connected (dense) models in LLMs has reached a bottleneck. How to reduce the training difficulty and inference cost of LLMs has become an important issue that needs to be addressed.

[0003] In existing technologies, Mixture of Experts (MoE) models are used to replace Dense models because they can train faster and larger models with the same computing resources. However, MoE itself has some drawbacks, such as higher communication costs and more complex models. These drawbacks result in slower training speeds and higher GPU memory usage.

[0004] In summary, optimizing MOE to reduce GPU memory usage and improve training speed is a problem that needs to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method, apparatus, and device for optimizing hybrid expert models, which can solve the problem of video memory fragmentation, reduce the video memory usage of hybrid expert models, and achieve optimization of hybrid expert models.

[0006] In a first aspect, embodiments of the present invention provide a method for optimizing a hybrid expert model. The method includes obtaining the average number of tokens in the hybrid expert model, wherein the token is a basic unit for the hybrid expert model to process input data; determining N times the average number of tokens in the hybrid expert model, wherein N is an integer or a decimal greater than 1; and applying for target memory based on the N times the average number of tokens, wherein the target memory is used to store tokens during forward propagation.

[0007] Optionally, obtaining the average number of tokens for the hybrid expert model specifically includes: obtaining the model parameters of the hybrid expert model; and determining the average number of tokens based on the model parameters of the hybrid expert model.

[0008] Optionally, the method further includes: using the Computational Unified Device Architecture (CUDA) to deeply fuse the accessed operators of the hybrid expert model during forward and backward propagation.

[0009] Optionally, the method further includes: determining at least one special operator; and rewriting the at least one special operator.

[0010] Optionally, the special operator includes a tensor operation function that selects specific elements or sub-tensors from a specified dimension of the tensor based on the index tensor, and or, a tensor operation function that accumulates the values ​​in the source tensor to the target tensor according to the positions specified by the index tensor.

[0011] Optionally, when the hybrid expert model includes shared experts and vertical experts, the method further includes:

[0012] The shared expert and the vertical expert perform calculations simultaneously.

[0013] Optionally, the shared expert and the vertical expert perform calculations simultaneously, specifically including:

[0014] The first linear transformation operation and activation function operation of the shared expert are performed during the initial communication process of the vertical expert;

[0015] The second linear transformation operation of the shared expert is performed during the end communication process of the vertical expert.

[0016] Optionally, the method further includes:

[0017] In response to each backpropagation, the activation function and / or the signaling protocol layer in the Fibre Channel layered structure are recalculated.

[0018] Secondly, embodiments of the present invention provide an apparatus for optimizing a hybrid expert model, the apparatus comprising: an acquisition unit for acquiring the average number of tokens in the hybrid expert model, wherein the tokens are the basic units for the hybrid expert model to process input data; a determination unit for determining N times the average number of tokens in the hybrid expert model, wherein N is an integer or a decimal greater than 1; and a processing unit for requesting target memory based on the N times the average number of tokens, wherein the target memory is used to store tokens during forward propagation.

[0019] Optionally, the acquisition unit is specifically used for: acquiring the average number of tokens of the hybrid expert model, specifically including: acquiring the model parameters of the hybrid expert model; and determining the average number of tokens based on the model parameters of the hybrid expert model.

[0020] Optionally, the processing unit is further configured to: employ the Computational Unified Device Architecture (CUDA) to deeply fuse the accessed operators of the hybrid expert model during forward and backward propagation.

[0021] Optionally, the determining unit is further configured to: determine at least one special operator; and rewrite the at least one special operator.

[0022] Optionally, the special operator includes a tensor operation function that selects specific elements or sub-tensors from a specified dimension of the tensor based on the index tensor, and or, a tensor operation function that accumulates the values ​​in the source tensor to the target tensor according to the positions specified by the index tensor.

[0023] Optionally, when the hybrid expert model includes shared experts and vertical experts, the processing unit is further configured to: perform calculations simultaneously on the shared experts and the vertical experts.

[0024] Optionally, the processing unit is specifically used to: perform the first linear transformation operation and activation function operation of the shared expert during the start communication process of the vertical expert; and perform the second linear transformation operation of the shared expert during the end communication process of the vertical expert.

[0025] Optionally, the processing unit is further configured to: recalculate the activation function and / or the signaling protocol layer in the Fibre Channel layered structure in response to each backpropagation.

[0026] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect or any one of the possible methods of the first aspect.

[0027] Fourthly, embodiments of the present invention provide a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method as described in the first aspect or any one of the possibilities of the first aspect.

[0028] In this embodiment of the invention, the average number of tokens in the hybrid expert model is obtained, where the token is the basic unit for the hybrid expert model to process input data; N times the average number of tokens in the hybrid expert model is determined, where N is an integer or decimal greater than 1; and in response to each forward propagation, memory is requested based on the size of N times the average number of tokens. By using the above method, requesting memory according to N times the average number of tokens during forward propagation can solve the memory fragmentation problem, reduce the memory usage of the hybrid expert model, and optimize the hybrid expert model. Attached Figure Description

[0029] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0030] Figure 1 This is a schematic diagram of video memory usage in an embodiment of the present invention;

[0031] Figure 2 This is another schematic diagram of video memory usage in an embodiment of the present invention;

[0032] Figure 3 This is another schematic diagram of video memory usage in an embodiment of the present invention;

[0033] Figure 4 This is another schematic diagram of video memory usage in an embodiment of the present invention;

[0034] Figure 5 This is a flowchart of a method for optimizing a hybrid expert model according to an embodiment of the present invention;

[0035] Figure 6 This is a schematic diagram of shared experts and vertical experts in a MoE according to an embodiment of the present invention;

[0036] Figure 7 This is an overlapping flowchart of shared experts and vertical experts in an embodiment of the present invention;

[0037] Figure 8 This is a schematic diagram of an apparatus for optimizing a hybrid expert model according to an embodiment of the present invention;

[0038] Figure 9 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0039] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0040] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0041] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0042] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0043] In existing technologies, the fully connected (Dense) model is a core architecture of Large Language Models (LLMs). In this Dense model, each neuron is connected to all neurons in the previous layer, ensuring that all parameters participate in gradient updates and avoiding model collapse caused by local parameter failures. It is one of the basic architectures of traditional Transformer models and their variants. The LLM is based on deep learning and Natural Language Processing (NLP) techniques, using large-scale data training to obtain a model capable of understanding and generating natural language. However, the development of LLM has reached a bottleneck, encountering problems such as "illusion" issues arising from disregarding facts and the need for deep logical understanding and mathematical reasoning abilities. To address these issues, the complexity of large language models continues to increase. With the actual needs of different application scenarios, the parameters of large language models become increasingly larger, and their complexity and scale continuously increase, increasing both the training difficulty and the inference cost. To reduce the training difficulty and inference cost of LLMs, a Mixture of Expert Models (MELs) are adopted. Experts (MoE) replaces the Dense model. MoE is a sparse-gated deep learning model, primarily composed of a set of expert models and a gating model. MoE divides the input data into multiple regions based on task type and assigns one or more expert models to each region. Each expert model can focus on processing that specific part of the input data, thereby improving the overall model performance. Compared to the Dense model, MoE trains faster and can train larger models with the same computing resources. Therefore, MoE is used instead of the Dense model. However, MoE itself has some drawbacks, such as higher communication costs and greater model complexity, leading to slower training speeds and higher GPU memory usage.

[0044] The reason for the high memory usage mentioned above is as follows: In the PyTorch framework, the memory management mechanism requests a memory block (Segment) from the Graphics Processing Unit (GPU) via cudaMalloc, and then separates sub-blocks (Blocks) from the Segment, specifically as follows: Figure 1As shown, Block1, Block2, and Block3 are separated within the GPU memory segment. During the forward propagation of the model training, these Blocks are used to store intermediate results. Specifically, Block1, Block2, and Block3 are allocated to the allocated GPU memory pool. Then, the remaining GPU memory in the segment is allocated to the unallocated GPU memory pool. The Blocks allocated to the allocated memory pool are gradually released during the backpropagation process. For the Dense model described above, the size of each intermediate result is fixed during each forward propagation. Figure 1 Each block in the model is fully utilized; however, the architecture of the multiple expert models in the hybrid expert model MOE is expert parallel (EP), which causes the number of tokens routed to each expert to vary during the computation process. Consequently, the total number of tokens (experttokens_num) of the multiple expert models in the MOE varies. Assuming the MOE requires three forwarding processes, although the total number of tokens in the intermediate results is not fixed, the average number of tokens can be calculated based on the MOE model parameters, and the calculated value is T. If the expert tokens_num in the three forwarding processes of the MOE are T, T1, and T2 respectively, where T2 > T > T1; in the first forwarding process, expert tokens_num is T, and the usage of GPU memory and the existing memory pool is as follows... Figure 2 As shown, under specific circumstances, each memory segment requested from the video memory can be treated as a block. Assume that Segment1, Segment2, and Segment3 are requested from the video memory and allocated to the video memory pool as blocks. Since T is pre-calculated, the allocation of segments is based on T. Segment1, Segment2, and Segment3 are multiples of T. Therefore, Segment1 will be fully utilized by intermediate result 1, Segment2 will be fully utilized by intermediate result 2, and Segment3 will be fully utilized by intermediate result 3. In the second forword process, expert tokens_num is T1. Since T1 is less than T, the usage of video memory and the existing memory pool is as follows. Figure 3As shown, Segment 1 will not be fully used by intermediate result 1, resulting in a small amount of idle time; Segment 2 will not be fully used by intermediate result 2, resulting in a small amount of idle time; Segment 3 will not be fully used by intermediate result 3, resulting in a small amount of idle time. During the third forword process, expert tokens_num is T2, and the usage of the display memory and existing memory pool is as follows: Figure 4 As shown, if Segment 1 is insufficient for intermediate result 1, then intermediate result 1 needs to find a suitable Segment among Segment 1, Segment 2, and Segment 3. Segment 3 is used by intermediate result 1, leaving a small amount of free space. Similarly, intermediate result 1 needs to find a suitable Segment among Segment 1 and Segment 2. Segment 1 is used by intermediate result 2, leaving a small amount of free space. Segment 2 cannot be used by intermediate result 3, so a new Segment 4 needs to be allocated in video memory for intermediate result 3. At this point, Segment 2 is completely unused. Therefore, the above process leads to video memory fragmentation, causing a rise in video memory spikes and resulting in out-of-memory (OOM) problems, making it impossible to fully utilize video memory.

[0045] The reasons for the slow training speed are as follows: MoE's unique computational logic causes intermediate computation results to occupy a large amount of memory. The unique computational logic is the expert network and the top-k selection mechanism. If a complete recalculation is used during backpropagation, the memory usage during training can be significantly reduced, but it will inevitably lead to a significant decrease in training speed.

[0046] Therefore, optimizing MOE to reduce memory usage and improve training speed is a problem that needs to be solved.

[0047] In this embodiment of the invention, to address the above-mentioned problems, a method for optimizing a hybrid expert model is proposed, specifically as follows: Figure 5 As shown, the method includes:

[0048] Step S501: Obtain the average number of tokens for the hybrid expert model.

[0049] Specifically, the token is the basic unit for the hybrid expert model to process input data.

[0050] In one possible implementation, the model parameters of the hybrid expert model are first obtained, and then the average number of tokens is determined based on the model parameters of the hybrid expert model, denoted as avg_token_num.

[0051] Step S502: Determine N times the average number of tokens in the hybrid expert model.

[0052] Wherein, N is an integer or decimal greater than 1, and N is determined based on the historical token count statistics of the hybrid expert model.

[0053] Assuming N is 1.5, then the average number of tokens in the hybrid expert model is determined to be N times the average number of tokens in the hybrid expert model, which is 1.5 times the average number of tokens in the avg_token_num. This is only an example for illustration, and the value of N should be determined according to the actual situation.

[0054] Step S503: Apply for target memory based on the N times the average number of tokens, wherein the target memory is used to store tokens during the forward propagation process.

[0055] Specifically, after allocating target memory, the tokens need to occupy memory during each forward propagation. During forward propagation, the pre-allocated target memory size is N times the average number of tokens; that is, the allocated segment size can hold N times the average number of tokens. Therefore, the intermediate results will only exhibit the aforementioned behavior when occupying memory. Figure 2 and Figure 3 In this situation, there is no need to re-apply for video memory, meaning the above-mentioned issues will not occur. Figure 4 In this way, the problem of video memory fragmentation is effectively avoided, ensuring that video memory resources can be fully utilized in scenarios with dynamically changing token quantities.

[0056] In one possible implementation, to improve training speed, the Compute Unified Device Architecture (CUDA) is used to deeply fuse the memory-accessed operators of the hybrid expert model during forward and backward propagation. CUDA is a parallel computing platform and programming model provided by NVIDIA to leverage the powerful computing capabilities of NVIDIA GPUs to accelerate tasks such as scientific computing, deep learning, and image processing.

[0057] Specifically, to ensure broad applicability, PyTorch operators are relatively slow in computation. Therefore, it is necessary to deeply fuse the memory-accessed operators (also known as kernels) of SwigLU (Swish-Gated Linear Unit) during forward and backward propagation to improve computation speed. The PyTorch operators are the core components implementing tensor operations in the PyTorch framework, serving as the basic units of the computation graph in deep learning models. They are responsible for executing specific computational logic, such as addition, matrix multiplication, and convolution, and support features like automatic differentiation and device acceleration. SwigLU is a hybrid activation function that combines the Swish activation function and the Gated Linear Unit (GLU).

[0058] In one possible implementation, to improve training speed, at least one special operator can be identified and rewritten using the CUDA described above.

[0059] Specifically, the special operators include a tensor operation function (index_select) that selects specific elements or sub-tensors from a specified dimension of the tensor based on the index tensor, and a tensor operation function (scatter_add) that adds the values ​​in the source tensor to the target tensor based on the positions specified by the index tensor.

[0060] In this embodiment of the invention, rewriting the index_select and / or scatter_add can reduce the number of memory accesses and optimize computational efficiency.

[0061] In one possible implementation, during the MoE training process, an expert parallel (EP) strategy is adopted. EPs need to communicate with each other. Since the hybrid expert model includes shared experts and routed experts, it is necessary to achieve overlap between the computation of shared experts and the communication of routed experts, that is, the shared experts and the routed experts perform calculations simultaneously.

[0062] Specifically, the shared experts are experts who possess more generalized or public knowledge, thereby reducing knowledge redundancy in each fine-grained expert. The number of shared experts is fixed and they are always active. The vertical experts are obtained by dividing a feedforward neural network (FFN) into mm parts through fine-grained expert segmentation. The number of parameters for each expert is reduced, but the professional level of the experts can be improved.

[0063] In one possible implementation, a schematic diagram of shared experts and vertical experts in the MoE is shown below. Figure 6 As shown, Figure 6 The "1" in the text represents a shared expert. Figure 6 In the diagram, “2”, “3”, “4”, ..., “2N-1” and “2N” represent vertical experts. The input hidden layer sends data to the shared expert and the vertical expert, and outputs the data to the output hidden layer. The calculations of the shared expert and the vertical expert are independent of each other and have no data dependency. Based on the above characteristics, the calculation of the shared expert and the communication of the vertical expert are overlapped.

[0064] In one possible implementation, the shared expert and the vertical expert perform calculations simultaneously, specifically including: performing the first linear transformation operation (Linear1) and activation function (Act) operation of the shared expert during the initial communication process of the vertical expert; and performing the second linear transformation operation (Linear2) of the shared expert during the final communication process of the vertical expert, specifically as follows... Figure 7 As shown, the Figure 7 This is a flowchart illustrating the overlap between shared experts and vertical experts. Specifically, the routed expert's initial communication process overlaps with the shared expert's Linear1 and Act processes. Then, the routed expert undergoes token permutation, expert forward propagation, and token unpermutation calculations. Finally, the routed expert's communication process ends and overlaps with the shared expert's Linear2 process. The expert forward propagation also includes the calculation of Linear1, Act, and Linear2. Figure 7 It also includes input and output.

[0065] In this embodiment of the invention, due to the unique computational logic of MoE, the intermediate computation results occupy a large space in the GPU memory. The unique computational logic is the expert network and the top-k selection mechanism. During backpropagation, a partial recalculation method is adopted. Specifically, since the calculation time of the activation function is very short, but its output result occupies a large space in the GPU memory, similarly, the calculation time of the expert FC2 layer accounts for a small proportion of the entire training process, but its output result also occupies a large space in the GPU memory. Therefore, during each backpropagation, the activation function (Act) and / or the signaling protocol layer (FC2 layer) in the Fibre Channel layered structure are recalculated.

[0066] In this embodiment of the invention, the above-mentioned MoE can also be called FastMoE, which is a MoE version based on Megatron Core MoE that has been deeply optimized, and has achieved memory optimization and speed optimization. The Megatron Core MoE is a MoE based on the NVIDIA Megatron-LM framework.

[0067] In one possible implementation, the MoE in this application can specifically be the MoE 96A23 model, with model parameters shown in Table 1 below, as detailed below:

[0068] Table 1

[0069]

[0070]

[0071] In one possible implementation, the interfaces (APIs) for calculating GPU memory in PyTorch include `torch.cuda.max_memory_allocated` and `torch.cuda.max_memory_reserved`. `torch.cuda.max_memory_allocated` retrieves the maximum amount of memory effectively occupied by tensors on the current CUDA device, while `torch.cuda.max_memory_reserved` returns the maximum amount of memory already reserved on the current CUDA device, in bytes. Memory reservation refers to memory reserved for models or data that is not actually used; that is, the amount of memory requested on the GPU. The ratio `max_memory_reserved / max_memory_allocated` represents the GPU memory fragmentation situation, i.e., the amount of memory requested on the GPU divided by the amount actually used. The larger this ratio, the more severe the GPU memory fragmentation. With the FastMoE described above, the training speedup is 1.12x, .max_memory_allocated is reduced by 16.9%, and the fragmentation rate is reduced from 1.298 to 1.008. This is only an example for illustration.

[0072] In this embodiment of the invention, the memory fragmentation problem of the MoE model is effectively solved by the above-mentioned dynamic memory allocation method; the designed partial recalculation strategy achieves a balance between speed and memory; some PyTorch operators are rewritten using CUDA, which significantly improves the training speed; and the overlap of shared expert computation and vertical expert communication is realized, which further improves the training speed.

[0073] In this embodiment of the invention, an apparatus for optimizing a hybrid expert model is provided, such as... Figure 8 As shown, it specifically includes: an acquisition unit 801, a determination unit 802, and a processing unit 803; wherein, the acquisition unit 801 is used to acquire the average number of tokens of the hybrid expert model, wherein the token is the basic unit for the hybrid expert model to process input data; the determination unit 802 is used to determine N times the average number of tokens of the hybrid expert model, wherein N is an integer or decimal greater than 1; the processing unit 803 allocates target memory according to the N times the average number of tokens, wherein the target memory is used to store tokens during the forward propagation process.

[0074] Furthermore, the acquisition unit is also used to: acquire the model parameters of the hybrid expert model; and determine the average number of tokens based on the model parameters of the hybrid expert model.

[0075] Furthermore, the processing unit is also configured to: employ the Computational Unified Device Architecture (CUDA) to deeply fuse the accessed operators of the hybrid expert model during forward and backward propagation.

[0076] Furthermore, the determining unit is also used to: determine at least one special operator; and rewrite the at least one special operator.

[0077] Furthermore, the special operators include tensor operation functions that select specific elements or sub-tensors from a specified dimension of the tensor based on the index tensor, and tensor operation functions that accumulate values ​​in the source tensor to the target tensor based on the positions specified by the index tensor.

[0078] Furthermore, when the hybrid expert model includes shared experts and vertical experts, the processing unit is also used to: allow the shared experts and the vertical experts to perform calculations simultaneously.

[0079] Furthermore, the processing unit is specifically used to: perform the first linear transformation operation and activation function operation of the shared expert during the start communication process of the vertical expert; and perform the second linear transformation operation of the shared expert during the end communication process of the vertical expert.

[0080] Furthermore, the processing unit is also configured to: recalculate the activation function and / or the signaling protocol layer in the Fibre Channel layered structure in response to each backpropagation.

[0081] Figure 9 This is a schematic diagram of the structure of the electronic device described in an embodiment of the present invention. For example... Figure 9 As shown, it includes a general computer hardware architecture, which includes at least a processor 901 and a memory 902. The processor 901 and the memory 902 are connected via a bus 903. The memory 902 is adapted to store instructions or programs executable by the processor 901. The processor 901 can be a standalone microprocessor or a collection of one or more microprocessors. Thus, the processor 901 executes the instructions stored in the memory 902 to perform the method flow of the embodiments of the present invention as described above, thereby realizing data processing and control of other devices. The bus 903 connects the above-mentioned components together, and also connects the above-mentioned components to a display controller 904, a display device, and an input / output (I / O) device 905. The input / output (I / O) device 905 can be a mouse, keyboard, modem, network interface, touch input device, motion-sensing input device, printer, and other devices known in the art. Typically, the input / output device 905 is connected to the system via an input / output (I / O) controller 906.

[0082] The instructions stored in memory 902 are executed by at least one processor 901 to: obtain the average number of tokens in the hybrid expert model, wherein the tokens are the basic units for the hybrid expert model to process input data; determine N times the average number of tokens in the hybrid expert model, wherein N is an integer or a decimal greater than 1; and request target video memory based on the N times the average number of tokens, wherein the target video memory is used to store tokens during forward propagation.

[0083] Specifically, the electronic device includes: one or more processors 901 and a memory 902. Figure 9 Take a processor 901 as an example. The processor 901 and the memory 902 can be connected via a bus or other means. Figure 9 Taking a bus connection as an example, memory 902, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Processor 901 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 902, thereby implementing the aforementioned method for determining and optimizing the hybrid expert model.

[0084] Memory 902 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, memory 902 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 902 may optionally include memory remotely located relative to processor 901, and these remote memories may be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0085] One or more modules are stored in memory 902, and when executed by one or more processors 901, they perform the method for optimizing the hybrid expert model in any of the above method embodiments.

[0086] As those skilled in the art will recognize, various aspects of the embodiments of the present invention can be implemented as a system, method, or computer program product. Therefore, various aspects of the embodiments of the present invention can take the form of a completely hardware implementation, a completely software implementation (including firmware, resident software, microcode, etc.), or an implementation combining software and hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." Furthermore, various aspects of the embodiments of the present invention can take the form of a computer program product implemented in one or more computer-readable media having computer-readable program code implemented thereon.

[0087] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, (but not limited to) an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include: an electrical connection having one or more wires, a portable computer floppy disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the context of embodiments of the present invention, a computer-readable storage medium can be any tangible medium capable of containing or storing a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0088] Computer-readable signal media may include propagated digital signals having computer-readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and can communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0089] Program code implemented on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, or any suitable combination thereof.

[0090] Computer program code for performing operations relating to various aspects of embodiments of the present invention can be written in any combination of one or more programming languages, including: object-oriented programming languages ​​such as Java, Smalltalk, C++, etc.; and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code can be executed as a standalone software package entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet provided by an Internet service provider).

[0091] The flowchart illustrations and / or block diagrams of the methods, apparatus (systems), and computer program products according to embodiments of the present invention describe various aspects of the embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions (executed via the processor of the computer or other programmable data processing apparatus) create means for implementing the functions / actions specified in the flowchart and / or block diagram blocks or blocks.

[0092] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus or other means to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of writing that includes instructions that implement the functions / actions specified in flowchart and / or block diagram blocks or blocks.

[0093] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operable steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide for implementing the functions / actions specified in flowchart and / or block diagram blocks or blocks.

[0094] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0095] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding access points are provided for users to choose to authorize or refuse processing. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

Claims

1. A method for optimizing a hybrid expert model, characterized in that, The method includes: Obtain the average number of tokens for the hybrid expert model, where the token is the basic unit for the hybrid expert model to process input data; The average number of tokens in the hybrid expert model is determined to be N times, where N is an integer or a decimal greater than 1; Request target memory based on the average number of tokens N times, wherein the target memory is used to store tokens during the forward propagation process.

2. The method according to claim 1, characterized in that, The process of obtaining the average number of tokens for the hybrid expert model specifically includes: Obtain the model parameters of the hybrid expert model; The average number of tokens is determined based on the model parameters of the hybrid expert model.

3. The method according to claim 1, characterized in that, The method further includes: The Computing Unified Device Architecture (CUDA) is used to deeply fuse the operators accessed in the hybrid expert model during forward and backward propagation.

4. The method according to claim 1, characterized in that, The method further includes: Identify at least one special operator; Rewrite at least one of the special operators.

5. The method according to claim 4, characterized in that, The special operators include tensor operation functions that select specific elements or sub-tensors from a specified dimension of a tensor based on an index tensor, and tensor operation functions that accumulate values ​​in a source tensor to a target tensor based on positions specified by the index tensor.

6. The method according to claim 1, characterized in that, When the hybrid expert model includes shared experts and vertical experts, the method further includes: The shared expert and the vertical expert perform calculations simultaneously.

7. The method according to claim 6, characterized in that, The shared expert and the vertical expert perform calculations simultaneously, specifically including: The first linear transformation operation and activation function operation of the shared expert are performed during the initial communication process of the vertical expert; The second linear transformation operation of the shared expert is performed during the end communication process of the vertical expert.

8. The method according to claim 1, characterized in that, The method further includes: In response to each backpropagation, the activation function and / or the signaling protocol layer in the Fibre Channel layered structure are recalculated.

9. An apparatus for optimizing a hybrid expert model, characterized in that, The device includes: An acquisition unit is used to acquire the average number of tokens of the hybrid expert model, wherein the token is the basic unit for the hybrid expert model to process input data; A determining unit is used to determine N times the average number of tokens in the hybrid expert model, where N is an integer or a decimal greater than 1; The processing unit requests target memory based on the N times the average number of tokens, wherein the target memory is used to store tokens during the forward propagation process.

10. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-8.

Images