Optimization method and device of hybrid expert model

Abstract

The embodiments of the present application provide a hybrid expert model optimization method, device, equipment and computer-readable storage medium, which relate to the field of artificial intelligence technology. The method includes: obtaining the operating parameter information of the model, wherein the operating parameter information includes: parameter input quantity information, memory occupancy information, and load distribution information; according to the operating parameter information, executing the corresponding model optimization strategy to optimize the model. In this way, the targeted optimization model can be implemented to improve the model operation performance, shorten the model optimization time, and improve the model optimization efficiency.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to an optimization method, apparatus, device, and computer-readable storage medium for a hybrid expert model. Background Art

[0002] Model optimization technology has been widely used in the fields of machine learning and deep learning, aiming to improve model performance and reduce model computing costs. As data scale and model complexity continue to increase, model optimization technology is becoming increasingly important.

[0003] In the process of model optimization, most related technologies update each parameter based on the back propagation algorithm to improve the performance of the machine learning model. The above methods have the problems of poor model optimization targeting, long time consumption and low efficiency. Summary of the invention

[0004] According to an embodiment of the present application, a hybrid model optimization solution is provided, which can achieve targeted optimization of the model to improve the model operation performance, shorten the model optimization time, and improve the model optimization efficiency.

[0005] In a first aspect of the present application, a hybrid expert model optimization method is provided. The method comprises: obtaining operation parameter information of the model; the operation parameter information comprises: parameter input quantity information, memory occupancy information, and load distribution information; and executing a corresponding model optimization strategy to optimize the model according to the operation parameter information.

[0006] In some feasible implementations, the model includes: a Transformer architecture, when the number of parameter inputs is greater than a preset number threshold, executing a corresponding model optimization strategy to optimize the model based on the operating parameter information includes: replacing the FFN layer in the original encoder layer of the model with a MoE layer, generating a target encoder layer to optimize the model, wherein the target encoder layer includes: at least one gating network and corresponding multiple experts.

[0007] In some feasible implementations, when the memory usage is greater than a preset memory usage threshold, the corresponding model optimization strategy is executed according to the operating parameter information to optimize the model, including: constructing a weight change; when the original weight of the model is less than the preset weight threshold, generating a target weight according to the weight change and the original weight; and according to the target weight, training the model so that the memory usage is less than or equal to the preset memory usage threshold.

[0008] In some feasible implementations, when the original weights of the model are greater than or equal to a preset weight threshold, the above method also includes: decomposing the weight change amount to generate a first matrix and a second matrix, wherein the product of the first matrix and the second matrix is ​​equal to the original weights; fixing the original weights, training the parameters of the first matrix and / or the parameters of the second matrix so that the memory usage is less than or equal to the preset memory usage threshold.

[0009] In some feasible implementations, executing the corresponding model optimization strategy to optimize the model according to the operating parameter information further includes: determining the load distribution balance according to the number of tokens received by each expert; when the load distribution balance is less than a preset load distribution balance threshold, constructing a loss function to make the load distribution balance greater than or equal to the preset load distribution balance threshold; the loss function is determined according to the following formula:

[0010] ;

[0011] in, is the loss function, is the balance factor, is the number of experts, Assign to experts Token ratio, To assign the gating network to the expert in the data sample barch The mean of the routing coefficients;

[0012] Assign to Expert Token ratio Determined according to the following formula:

[0013] ;

[0014] in, Assign to experts Token ratio, To assign experts The number of tokens, is the total number of tokens in the data sample barch;

[0015] Gating network assigned to experts in data sample barch The mean value of the routing coefficient Determined according to the following formula:

[0016] ;

[0017] in, To assign the gating network to the expert in the data sample barch The mean value of the routing coefficient, is the total number of tokens in the data sample barch, To assign the gating network to the expert in the data sample barch The routing coefficient.

[0018] In some feasible implementations, the above method also includes: when the gating network is not differentiable, generating corresponding gumbel noise for each discrete sampling data to change the data sampling method from discrete sampling to continuous sampling, wherein the expected value of discrete sampling is equal to the expected value of continuous sampling; changing the sampling function to a gumbel softmax function to make the gating network differentiable.

[0019] In some feasible implementations, the above method also includes: preheating the model to a target number of rounds so that the model learns the target features; copying the weight information of the model; copying a preset number of FFN layers to serve as multiple experts, wherein the preset number is equal to the number of multiple experts; corresponding to the multiple experts, inserting a gated network; freezing the remaining parameters, and training the gated network and experts to optimize the model.

[0020] In a second aspect of the present application, an optimization device for a hybrid expert model is provided, comprising: an acquisition unit for acquiring operating parameter information of the model, wherein the operating parameter information includes: parameter input quantity information, memory usage information, and load distribution information; an execution unit for executing a corresponding model optimization strategy to optimize the model according to the operating parameter information.

[0021] In a third aspect of the present application, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method according to the first aspect of the present application is implemented.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method of the first aspect of the present application is implemented.

[0023] The optimization method of the hybrid expert model provided in the embodiment of the present application obtains the operating parameter information of the model, wherein the operating parameter information includes: parameter input quantity information, memory usage information, and load distribution information; according to the operating parameter information, the corresponding model optimization strategy is executed to optimize the model, so as to achieve targeted optimization of the model to improve the model operation performance, shorten the model optimization time, and improve the model optimization efficiency. It should be understood that the content described in the invention content section is not intended to limit the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0025] Figure 1 A schematic diagram of a process of optimizing a hybrid expert model provided in an embodiment of the present application;

[0026] Figure 2 A schematic block diagram of a MoE layer provided in an embodiment of the present application;

[0027] Figure 3 A schematic block diagram of a model preheating provided in an embodiment of the present application;

[0028] Figure 4 A structural schematic diagram of an optimization device for a hybrid expert model provided according to an embodiment of the present application;

[0029] Figure 5 A structural schematic diagram of an electronic device provided according to an embodiment of the present application;

[0030] Figure 6 It is a schematic diagram of the structure of a terminal device or a server suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0032] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0033] Model optimization technology has been widely used in the fields of machine learning and deep learning, aiming to improve model performance and reduce model computing costs. As data scale and model complexity continue to increase, model optimization technology is becoming increasingly important.

[0034] In the process of model optimization, most related technologies update each parameter based on the back propagation algorithm to improve the performance of the machine learning model. The above methods have the problems of poor model optimization targeting, long time consumption and low efficiency.

[0035] In view of this, an embodiment of the present application provides a model optimization solution that can achieve targeted optimization of the model to improve model operating performance, shorten model optimization time, and improve model optimization efficiency.

[0036] In a first aspect of the present application, a method for optimizing a hybrid expert model is provided. Figure 1 A schematic diagram of a hybrid expert model optimization method 100 provided in an embodiment of the present application is shown in FIG. Figure 1 The method 100 comprises the following steps:

[0037] Step S110, obtaining the operating parameter information of the model, wherein the operating parameter information includes: parameter input quantity information, memory usage information, and load distribution information.

[0038] In some feasible implementations, the operating parameter information of the model can be obtained in real time in the background or in the cloud.

[0039] Exemplarily, the parameter input quantity information can be obtained in real time based on the model definition code or using tools provided by the framework. Specifically, the definition code or tools provided by the framework can include: TensorFlow / Keras, PyTorch, etc.

[0040] The above memory usage information can be obtained in real time based on monitoring tools. Specifically, the above monitoring tools can include: TensorFlow / Keras, such as: tf.profiler or tf.summary, etc., PyTorch, nvidia-smi, etc.

[0041] The above load distribution information can be obtained based on TensorFlow / Keras or PyTorch.

[0042] It should be noted that the above parameter input quantity information, memory usage information, and load distribution information can be combined with visualization tools and visualized on the user's PC to facilitate the user to intuitively monitor the running status of the model.

[0043] Step S120: Execute the corresponding model optimization strategy to optimize the model according to the operation parameter information.

[0044] Exemplarily, when it is determined that the number of parameter inputs of the model is greater than a preset threshold, a corresponding model optimization strategy can be executed to improve the applicability of the model to large-scale data set tasks and improve the high computing power of the model.

[0045] When it is determined that the memory usage of the model is greater than a preset memory usage threshold, a corresponding model optimization strategy can be executed to reduce the memory usage of the model and improve the applicability of the model to resource-constrained operating environments.

[0046] When it is determined that the model load distribution balance is less than the preset load distribution balance threshold, the corresponding model optimization strategy can be executed to improve the model load balance, thereby improving the model training efficiency and shortening the model training time.

[0047] Based on this, the optimization method of the hybrid expert model provided in the present application obtains the operating parameter information of the model, wherein the operating parameter information includes: parameter input quantity information, memory usage information, and load distribution information; according to the operating parameter information, the corresponding model optimization strategy is executed to optimize the model, so as to achieve targeted optimization of the model to improve the applicability of the model for large-scale data set tasks and improve the model computing power; targeted reduction of the model's memory usage and improvement of the model's applicability to resource-constrained operating environments; targeted improvement of the load distribution balance during the model operation process, thereby improving the model training efficiency, shortening the model training time, and improving the model optimization efficiency.

[0048] In some feasible implementations, the model includes: a Transformer architecture, when the number of parameter inputs is greater than a preset number threshold, executing a corresponding model optimization strategy to optimize the model based on the operating parameter information includes: replacing the FFN layer in the original encoder layer of the model with a MoE layer, generating a target encoder layer to optimize the model, wherein the target encoder layer includes: at least one gating network and corresponding multiple experts.

[0049] In some feasible implementations, the above includes: the model of the Transformer architecture can be a large language pre-training model (Large Language Model, LLM).

[0050] Exemplarily, when the number of parameter inputs of the LLM pre-trained model obtained in real time based on the definition code of the LLM pre-trained model or using the tools provided by the framework is greater than the preset number threshold, the feed-forward network (FFN) layer in the original encoder layer of the LLM pre-trained model is replaced with a mixture of experts (MoE) layer to generate a target encoder layer to optimize the model, wherein the target encoder layer includes: at least one gated network and corresponding multiple experts. It should be noted that the above-mentioned preset number threshold can be defined according to the actual needs of the user and is not specifically limited here.

[0051] For example, Figure 2 A schematic block diagram of a MoE layer provided in an embodiment of the present application. Figure 2 As shown in Figure 2, a MoE layer consists of a gating network and corresponding n experts. The input parameters of the MoE layer are Usually it is the intermediate representation vector of the original input. The above parameters The dimensions of are (batch_size, seq_length, hidden_size). Specifically, the parameters Input to the gating network, which is usually a fully connected layer, and the weight size of the gating network is hidden_size×n. It should be noted that for a 1×hidden_size vector, the output of the gating network is a 1×n vector, which is used to control the selected expert. For hard routing, the output is a 0 / 1 vector. For soft routing, the output is the probability value of selecting an expert, specifically, Figure 2 As shown, experts 1, 2, and n-1 are selected with probabilities of 0.1, 0.3, and 0.6, respectively.

[0052] It should be noted that the input parameter Input to the experts selected by the gating network, The output of the experts is , expert in gating network selection The probability of , then the output result of the MoE layer can be determined according to the following formula (1):

[0053] (1);

[0054] in, is the number of experts, For experts The probability of For experts Output.

[0055] Based on this, the above method replaces the FFN layer in the original encoder layer of the model with the MoE layer when it is determined that the number of parameter inputs of the model containing the Transformer architecture is greater than the preset number threshold, and generates a target encoder layer to optimize the model. This can achieve targeted improvement of the model's applicability to large-scale data set tasks and improve the model's computing power and adaptability.

[0056] In some feasible implementations, when the memory usage is greater than a preset memory usage threshold, the corresponding model optimization strategy is executed according to the operating parameter information to optimize the model, including: constructing a weight change; when the original weight of the model is less than the preset weight threshold, generating a target weight according to the weight change and the original weight; and according to the target weight, training the model so that the memory usage is less than or equal to the preset memory usage threshold.

[0057] In some feasible implementations, the above-mentioned preset memory usage threshold and the above-mentioned preset weight threshold can be customized according to the actual needs of the user, and are not specifically limited here.

[0058] In some feasible implementations, the above-mentioned weight change amount can be constructed according to the user's memory optimization requirements. Specifically, the above-mentioned weight change amount is negatively correlated with the user's memory optimization requirements, that is, the higher the user's memory optimization requirements, the smaller the above-mentioned weight change amount. In some feasible implementations, the user can perform quantization operations or weight pruning operations on the weight change amount according to actual needs. Alternatively, the above-mentioned weight change amount can be constructed according to the user's requirements for the convergence speed of the model. Specifically, the above-mentioned weight change amount is positively correlated with the user's requirements for the convergence speed of the model, that is, the higher the user's requirements for the convergence speed of the model, the larger the above-mentioned weight change amount. In some feasible implementations, the user can perform gradient pruning on the weight change amount according to actual needs.

[0059] For example, the weight change can be expressed as Indicates that the above original weights can be used The above target weights can be expressed as express.

[0060] For example, the target weight It can be determined according to the following formula (2):

[0061] (2);

[0062] Based on this, the above method constructs a weight change when the memory usage is greater than a preset memory usage threshold, and generates a target weight based on the weight change and the original weight when the original weight of the model is less than the preset weight threshold; the model is trained according to the target weight, so that the model weight can be fine-tuned and the memory usage during the model training process can be reduced so that the model memory usage is less than the preset memory usage threshold, thereby improving the model training efficiency and enhancing the applicability of the model to resource-constrained operating environments.

[0063] In some feasible implementations, when the original weights of the model are greater than or equal to a preset weight threshold, the above method also includes: decomposing the weight change amount to generate a first matrix and a second matrix, wherein the product of the first matrix and the second matrix is ​​equal to the original weights; fixing the original weights, training the parameters of the first matrix and / or the parameters of the second matrix so that the memory usage is less than or equal to the preset memory usage threshold.

[0064] It should be noted that in the original weight of the model When the weight is greater than or equal to the preset weight threshold, the weight change can be decomposed based on the efficient parameter method. , to generate the first matrix and the second matrix .

[0065] For example, the original weights The size can be , the first matrix The size can be , the second matrix The size can be , where the first matrix and the second matrix The product is equal to the original weight .

[0066] In some feasible implementations, the above-mentioned multiple experts can be replaced by an efficient parameter layer. For example, the above-mentioned original weights are fixed , train the first matrix The parameters and / or the second matrix The parameters of the input parameters are set so that the memory usage is less than or equal to the preset memory usage threshold. It should be noted that in this case, for the input parameters , then The output of the experts It can be determined according to the following formula (3):

[0067] (3);

[0068] in, For the The output of an expert, is the original weight, is the first matrix, is the second matrix, is the input parameter.

[0069] It should be noted that the first matrix And the second matrix above Rank It can be determined according to the user's memory optimization requirements. Specifically, the first matrix And the second matrix above Rank It is negatively correlated with the user's memory optimization requirements. That is, the higher the user's memory optimization requirements, the higher the first matrix And the second matrix above Rank The smaller. Among them, the first matrix And the second matrix above Rank The smaller it is, the fewer parameters the model needs to train, and the better the memory optimization effect.

[0070] Based on this, the above method can be achieved by decomposing the weight change, generating the first matrix and the second matrix, fixing the original weight, training the various parameters of the first matrix and / or the various parameters of the second matrix to optimize the model. When the original weight of the model is greater than or equal to the preset weight threshold, the memory occupancy of the redundant information in the weight change during the model training process is reduced, so that the memory occupancy during the model training process is less than or equal to the preset memory occupancy threshold, thereby improving the model training efficiency and enhancing the applicability of the model to resource-constrained operating environments.

[0071] In some feasible implementations, executing the corresponding model optimization strategy to optimize the model according to the operating parameter information further includes: determining the load distribution balance according to the number of tokens received by each expert; when the load distribution balance is less than a preset load distribution balance threshold, constructing a loss function to make the load distribution balance greater than or equal to the preset load distribution balance threshold; wherein the loss function is determined according to the following formula (4):

[0072] (4);

[0073] in, is the loss function, is the balance factor, is the number of experts, Assign to experts Token ratio, To assign the gating network to the expert in the data sample barch The mean of the routing coefficients.

[0074] It should be noted that the above balance factor It is positively correlated with the user's load balancing requirements. That is, the higher the user's load balancing requirements, the higher the above balance factor The bigger.

[0075] Assign to Expert Token ratio Determined according to the following formula (5):

[0076] (5);

[0077] in, Assign to experts Token ratio, To assign experts The number of tokens, is the total number of tokens in the data sample barch.

[0078] Gating network assigned to experts in data sample barch The mean value of the routing coefficient Determined according to the following formula (6):

[0079] (6);

[0080] in, To assign the gating network to the expert in the data sample barch The mean value of the routing coefficient, is the total number of tokens in the data sample barch, To assign the gating network to the expert in the data sample barch The routing coefficient.

[0081] It should be noted that, based on the number of tokens received by each expert, a matrix of the number of tokens received by the expert can be constructed, and the row data variance and / or column data variance of the above matrix can be calculated to determine the load distribution balance. Alternatively, based on the matrix of the number of tokens received by the above expert, the row data Euclidean distance and / or column data Euclidean distance of the above matrix can be calculated to determine the load distribution balance.

[0082] The above preset load distribution balance threshold can be defined according to the actual needs of the user and is not specifically limited here. In some feasible implementations, the variance threshold or the Euclidean distance threshold can be determined according to the preset load distribution balance threshold.

[0083] Specifically, when the variance of the row data or the variance of the column data of the above matrix is ​​greater than a preset variance threshold, a loss function is constructed so that the row data variance and the column data variance are less than or equal to the preset variance threshold, so that the load distribution balance is greater than or equal to the preset load distribution balance threshold. When the Euclidean distance of the row data and / or the Euclidean distance of the column data are greater than the preset Euclidean distance threshold, a loss function is constructed so that the Euclidean distance of the row data and the Euclidean distance of the column data are less than or equal to the preset Euclidean distance threshold, so that the load distribution balance is greater than or equal to the preset load distribution balance threshold.

[0084] In some feasible implementations, the training loss value of the MoE layer can be determined based on the loss value of the original model training data sample Batch and the above loss function based on the following formula (7) to improve the convergence speed and performance of the model:

[0085] (7);

[0086] in, is the training loss value of the MoE layer, is the loss value of the original model when training the data sample Batch, is the loss function.

[0087] Based on this, the above method determines the load distribution balance according to the number of tokens received by each expert; when the load distribution balance is less than the preset load distribution balance threshold, constructs a loss function to make the load distribution balance greater than or equal to the preset balance threshold; it can reduce the imbalance of the number of tokens received by each expert, improve the response speed of the model to training data samples, improve the continuity and stability of the model operation process, improve the resource utilization of each expert during the model operation process, avoid network congestion at a node during the model operation process, and improve the data throughput and convergence speed during the model operation process.

[0088] In some feasible implementations, the above method also includes: when the gating network is not differentiable, generating corresponding gumbel noise for each discrete sampling data to change the data sampling method from discrete sampling to continuous sampling, wherein the expected value of discrete sampling is equal to the expected value of continuous sampling; changing the sampling function to a gumbel softmax function to make the gating network differentiable.

[0089] It should be noted that before the gating network is selected k When an expert performs gradient update, that is, when the gated network is not differentiable, corresponding gumbel noise can be generated for each discrete sampling data, and the original discrete sampling method can be converted into a continuous sampling method based on the reparameterization technology.

[0090] For example, the vector output by the original gating model can be set to , used to represent the original n The sampling distribution of experts, where i The probability of an expert being selected is In some feasible implementations, the original selection Top k Discrete distribution sampling method for experts Change to

[0091] Specifically, The sampling method can be determined according to the following formula (8):

[0092] (8);

[0093] in, log represents the logarithmic function, For the i The probability of an expert being selected is for gumbel noise.

[0094] It should be noted that gumbel noise , sampled from ,in, is a uniformly distributed sample data point belonging to the interval [0, 1].

[0095] It should be noted that, as shown in the following formula (9), the expected value of discrete sampling is equal to the expected value of continuous sampling.

[0096] (9);

[0097] For example, The sampling function is changed to the gumbel softmax function, which can be approximated to Operation to make the gating network induced.

[0098] It should be noted that based on the gumbel softmax function, the selection expert of the gating network The probability of It can be determined according to the following formula (10):

[0099] (10);

[0100] in, exp is an exponential function, is the temperature coefficient.

[0101] It should be noted that The value range of is (0, +∞). Specifically, when When the value of approaches 0, the probability of selecting an expert is closer to the one-hot vector, that is, the Top 1 operation; when The closer the value of is to +∞, the more uniform the probability of selecting each expert.

[0102] Based on this, the above method generates corresponding gumbel noise for each discrete sampling data when the gating network is not differentiable, so as to change the data sampling mode from discrete sampling to continuous sampling, wherein the expected value of discrete sampling is equal to the expected value of continuous sampling; the sampling function is changed to gumbel softmax function to make the gating network differentiable, which is conducive to facilitating end-to-end training of the model, thereby simplifying the model training process and improving the model training efficiency.

[0103] In some feasible implementations, the above method also includes: preheating the model to a target number of rounds so that the model learns the target features; copying the weight information of the model; copying a preset number of FFN layers to serve as multiple experts, wherein the preset number is equal to the number of multiple experts; corresponding to the multiple experts, inserting a gated network; freezing the remaining parameters, and training the gated network and experts to optimize the model.

[0104] It should be noted that the above target number of rounds can be determined based on the user's requirements for the number of target features learned by the model and the quality of the target features learned. The target number of rounds is positively correlated with the user's requirements for the number of target features learned by the model and the quality of the target features learned, respectively, that is, the higher the user's requirements for the number and / or quality of the target features, the greater the target number of rounds.

[0105] For example, Figure 3 A schematic block diagram of a model preheating method provided in an embodiment of the present application, such as Figure 3As shown in the figure, the weight information of the original encoder layer can be copied, where the weight information includes: the weight information of the multi-head attention mechanism and the weight information of the addition normalization (Add&Norm). The position-wise feed-forward network (FFN) layer is copied n times as n experts, and the gated network is inserted before the multiple experts as the preheated encoder layer. Specifically, during the model training process, the remaining parameters are frozen, and the gated network and experts are trained to optimize the model.

[0106] Based on this, the above method preheats the model to the target number of rounds so that the model learns the target features; copies the weight information of the model; copies a preset number of FFN layers to serve as multiple experts, where the preset number is equal to the number of multiple experts; inserts a gating network corresponding to the multiple experts; freezes the remaining parameters, trains the gating network and the experts to optimize the model, so that the representation space divided by each expert is optimal, thereby achieving the optimal routing network of the model in the entire training process, and then enabling the gating network to learn a better solution to improve the operating performance of the model.

[0107] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0108] The above is an introduction to the method embodiment. The following is a further explanation of the scheme described in this application through an apparatus embodiment.

[0109] Figure 4 A structural schematic diagram of an optimization device 400 of a hybrid expert model provided in an embodiment of the present application is shown as follows: Figure 4 As shown, the device 400 includes: an acquisition unit 410 and an execution unit 420.

[0110] The acquisition unit 410 is used to acquire the operation parameter information of the model; the operation parameter information includes: parameter input quantity information, memory usage information, and load distribution information.

[0111] The execution unit 420 is used to execute the corresponding model optimization strategy to optimize the model according to the operating parameter information.

[0112] Figure 5is a structural schematic diagram of an electronic device 500 provided according to an embodiment of the present application, such as Figure 5 As shown, the electronic device 500 includes: a memory 520 and a processor 510, wherein the memory 520 stores a computer program, and when the processor executes the program, the method of the first aspect of the present application is implemented.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0114] Figure 6 A schematic diagram of the structure of a terminal device or a server suitable for implementing an embodiment of the present application is shown.

[0115] like Figure 6 As shown, the terminal device or server includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 into the random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the terminal device or server are also stored. The CPU 601, ROM 602 and RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0116] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0117] In particular, according to an embodiment of the present application, the above method flow steps can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present application are executed.

[0118] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0119] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of the code, and the aforementioned module, program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] The units or modules involved in the embodiments described in the present application may be implemented by software or hardware. The units or modules described may also be arranged in a processor. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

[0121] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the method described in the present application.

[0122] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.

Claims

1. A hybrid expert model optimization method, characterized in that: include: Get the model's operating parameter information; The operation parameter information includes: parameter input quantity information, memory usage information and / or load distribution information; According to the operating parameter information, executing a corresponding model optimization strategy to optimize the model; The model includes: a Transformer architecture, and when the number of parameter inputs is greater than a preset number threshold, executing a corresponding model optimization strategy according to the operating parameter information to optimize the model includes: Replacing the FFN layer in the original encoder layer of the model with the MoE layer, generating a target encoder layer to optimize the model, wherein the target encoder layer includes: at least one gating network and corresponding multiple experts; When the memory usage is greater than a preset memory usage threshold, executing a corresponding model optimization strategy to optimize the model according to the operating parameter information includes: Build weight changes; When the original weight of the model is less than a preset weight threshold, generating a target weight according to the weight change amount and the original weight; According to the target weight, training the model so that the memory usage is less than or equal to a preset memory usage threshold; The target weight is determined according to the following formula: ； in, is the target weight, is the original weight, is the weight change amount; In the case where the original weight of the model is greater than or equal to a preset weight threshold, the method further includes: Decomposing the weight change amount to generate a first matrix and a second matrix, wherein the product of the first matrix and the second matrix is ​​equal to the original weight; Fixing the original weights, and training various parameters of the first matrix and / or various parameters of the second matrix so that the memory occupancy is less than or equal to the preset memory occupancy threshold; According to the operating parameter information, executing a corresponding model optimization strategy to optimize the model further includes: Determine the load distribution balance based on the number of tokens received by each expert; When the load distribution balance degree is less than a preset load distribution balance degree threshold, constructing a loss function to make the load distribution balance degree greater than or equal to the preset load distribution balance degree threshold; The loss function is determined according to the following formula: ； in, is the loss function, is the balance factor, is the number of experts, Assign to experts Token ratio, To assign the gating network to the expert in the data sample barch The mean of the routing coefficients; Assign to Expert Token ratio Determined according to the following formula: ； in, Assign to experts Token ratio, To assign experts The number of tokens, is the total number of tokens in the data sample barch; Gating network assigned to experts in data sample barch The mean value of the routing coefficient Determined according to the following formula: ； in, To assign the gating network to the expert in the data sample barch The mean value of the routing coefficient, is the total number of tokens in the data sample barch, To assign the gating network to the expert in the data sample barch The routing coefficient.

2. The method according to claim 1, characterized in that Also includes: In the case where the gating network is not differentiable, generating corresponding gumbel noise for each discrete sampling data, so that the data sampling mode is changed from discrete sampling to continuous sampling, wherein the expected value of the discrete sampling is equal to the expected value of the continuous sampling; The sampling function is changed to a gumbel softmax function to make the gating network differentiable.

3. The method according to any one of claims 1 to 2, characterized in that include: Preheating the model to a target number of rounds so that the model learns target features; Copying the weight information of the model; Copying a preset number of FFN layers to serve as a plurality of experts, wherein the preset number of times is equal to the number of the plurality of experts; Corresponding to the plurality of experts, inserting the gating network; The remaining parameters are frozen, and the gating network and the expert are trained to optimize the model.

4. A hybrid model optimization device, characterized in that: include: An acquisition unit, used to acquire operation parameter information of the model, wherein the operation parameter information includes: parameter input quantity information, memory usage information, and load distribution information; An execution unit, configured to execute a corresponding model optimization strategy to optimize the model according to the operation parameter information; The model includes: a Transformer architecture, and when the number of parameter inputs is greater than a preset number threshold, executing a corresponding model optimization strategy according to the operating parameter information to optimize the model includes: Replacing the FFN layer in the original encoder layer of the model with the MoE layer, generating a target encoder layer to optimize the model, wherein the target encoder layer includes: at least one gating network and corresponding multiple experts; When the memory usage is greater than a preset memory usage threshold, executing a corresponding model optimization strategy to optimize the model according to the operating parameter information includes: Build weight changes; When the original weight of the model is less than a preset weight threshold, generating a target weight according to the weight change amount and the original weight; According to the target weight, training the model so that the memory usage is less than or equal to a preset memory usage threshold; The target weight is determined according to the following formula: ； in, is the target weight, is the original weight, is the weight change amount; In the case where the original weight of the model is greater than or equal to a preset weight threshold, the method further includes: Decomposing the weight change amount to generate a first matrix and a second matrix, wherein the product of the first matrix and the second matrix is ​​equal to the original weight; Fixing the original weights, and training various parameters of the first matrix and / or various parameters of the second matrix so that the memory occupancy is less than or equal to the preset memory occupancy threshold; According to the operating parameter information, executing a corresponding model optimization strategy to optimize the model further comprises: Determine the load distribution balance based on the number of tokens received by each expert; When the load distribution balance degree is less than a preset load distribution balance degree threshold, constructing a loss function to make the load distribution balance degree greater than or equal to the preset load distribution balance degree threshold; The loss function is determined according to the following formula: ； in, is the loss function, is the balance factor, is the number of experts, Assign to experts Token ratio, To assign the gating network to the expert in the data sample barch The mean of the routing coefficients; Assign to Expert Token ratio Determined according to the following formula: ； in, Assign to experts Token ratio, To assign experts The number of tokens, is the total number of tokens in the data sample barch; Gating network assigned to experts in data sample barch The mean value of the routing coefficient Determined according to the following formula: ； in, To assign the gating network to the expert in the data sample barch The mean value of the routing coefficient, is the total number of tokens in the data sample barch, To assign the gating network to the expert in the data sample barch The routing coefficient.

5. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images