Data processing method and system for low-energy large language model based on momentum mechanism and multi-type experts

Abstract

The present application discloses a data processing method and system for a low-energy large language model based on a momentum mechanism and multiple types of experts, belonging to the technical field of large language models, including obtaining a hybrid expert model, the hybrid expert model including a gating network, a combination network, and multiple expert networks; obtaining target data based on a lazy loading mechanism; obtaining the fitness scores of each expert network corresponding to the target data through the gating network, obtaining the expert network that meets the sorting requirements as the target network; obtaining the output data of the target data in the target network, and weighted summing the multiple output data through the combination network. The present application sets multiple expert networks in the hybrid expert model, ranks the fitness scores of the target data, selects the outputs of the expert networks that meet the ranking requirements, and performs weighted summing to obtain the final output, thereby reducing floating-point operations and computing memory overhead, improving computing efficiency, and reducing memory resource waste.

Description

Technical Field

[0001] The present application belongs to the technical field of large language models, and specifically relates to a data processing method and system for a low-energy large language model based on a momentum mechanism and multiple types of experts. Background Art

[0002] Sparse Mixture of Experts (SMoE) is a machine learning approach that partitions an AI model into separate subnetworks, or “experts,” that each specialize in a subset of the input data in order to jointly perform a task.

[0003] However, in the existing sparse mixture of experts (MoE) architecture, simple inputs are processed by too many experts, while complex inputs fail to fully utilize sufficient expert resources. This unbalanced allocation of computing resources leads to low computing efficiency and waste of resources. Summary of the Invention

[0004] Purpose of the invention: This application develops a data processing method and system for a low-energy large language model based on a momentum mechanism and multi-type experts, aiming to solve the technical problem in the prior art of low computing efficiency and waste of resources caused by the imbalance of computing resource allocation.

[0005] Technical Solution: In a first aspect, embodiments of the present application provide a data processing method for a low-energy, large language model based on a momentum mechanism and multiple types of experts, including:

[0006] Acquire a hybrid expert model, wherein the hybrid expert model includes a gating network, a combination network, and multiple expert networks;

[0007] Acquire target data based on a lazy loading mechanism and input the data into the hybrid expert model;

[0008] Obtaining the fitness scores of the target data corresponding to the respective expert networks through the gating network, and sorting the plurality of expert networks based on the fitness scores, and obtaining the expert network that meets the sorting requirements as the target network;

[0009] Output data of the target data in the target network is obtained, and a plurality of the output data are weightedly summed through the combination network to obtain a final output result.

[0010] In some embodiments, the step of acquiring target data based on the lazy loading mechanism includes:

[0011] Identify the data set to be processed;

[0012] Read the data set line by line, and obtain the data required by the target batch as the data to be processed;

[0013] The data to be processed is segmented, encoded and filled to obtain the target data.

[0014] In some embodiments, the step of sorting the plurality of expert networks based on the fitness scores and obtaining the expert network that meets the sorting requirements as the target network includes:

[0015] pass The strategy with the highest fitness score is selected The expert network is the target network, and its characterization formula includes:

[0016] ;

[0017] in, For selected The set of the expert networks with the highest fitness scores is used to represent the set of target networks; To select the top fitness score from all the expert networks of the expert network.

[0018] In some embodiments, the plurality of expert networks include a routing expert, the routing expert includes a plurality of sub-expert networks, and the data processing method further includes:

[0019] Obtain historical data of the target network selected by the gating network in the expert network;

[0020] The current selection result of the target network by the gating network is optimized based on the historical data, and the characterization formula includes:

[0021] ;

[0022] in, is the current selection result of the target network, used to represent the currently selected expert network; The historical data is used to characterize the expert network selected in the previous round; is the attenuation factor of the historical data, ; The said expert network is currently selected;

[0023] The gating network uses a momentum mechanism to dynamically adjust the routing expert, which is used to optimize the selection process of multiple sub-expert networks in combination with learning rate decay. The characterization formula for updating the momentum mechanism includes:

[0024] ;

[0025] in, for Momentum at the moment, used to characterize the momentum at the current moment; for The gating score at the moment is used to represent the gating score at the current moment; is the momentum coefficient; for Momentum at a moment, used to characterize the momentum at the previous moment; for The gate output at the moment is used to represent the gate output at the current moment. , for The gate output at the moment is used to represent the gate output at the previous moment. is the coefficient of learning rate decay.

[0026] In some embodiments, the data processing method further includes:

[0027] Obtain historical data of the target network selected by the gating network in the expert network;

[0028] The current selection result of the target network by the gating network is optimized based on the historical data, and the characterization formula includes:

[0029] ;

[0030] in, is the current selection result of the target network, used to represent the currently selected expert network; The historical data is used to characterize the expert network selected in the previous round; is the momentum factor, ; is the expert network currently selected.

[0031] In some embodiments, the step of obtaining the hybrid expert model includes:

[0032] Determining the hybrid expert model to be trained and a plurality of computing nodes;

[0033] Splitting the hybrid expert model to be trained and deploying it on multiple computing nodes;

[0034] Based on the ZeRO Stage-3 optimizer in the DeepSpeed ​​training model framework, distributed training is performed on the hybrid expert model to be trained on multiple computing nodes to obtain the trained hybrid expert model.

[0035] In some embodiments, the distributed training step includes:

[0036] Synchronously updating the gradients of the plurality of computing nodes based on a full-subtraction operation;

[0037] The plurality of gradients are merged to update the hybrid expert model.

[0038] In some embodiments, the distributed training step further includes:

[0039] The load balancing mechanism dynamically adjusts the workload of the computing nodes, and its characterization formula includes:

[0040] ;

[0041] in, For the The load of the computing node is used to characterize the The proportion of computing tasks of computing nodes to the total tasks; For the The computing power or task volume of each computing node is used to characterize the The amount of data that a computing node currently needs to process; is the total number of computing nodes.

[0042] In some embodiments, the plurality of expert networks further include shared experts, the shared experts being used to share global features and provide general calculations, and the representation formulas of the shared experts include:

[0043] ;

[0044] in, for said shared experts; is the target data, which is used to characterize the feature vector of the current sample; is the weight matrix of the shared expert, used to represent the transformation of the target data by the shared expert; is the bias term of the shared expert, used to adjust the output result of the shared expert;

[0045] The plurality of sub-expert networks include:

[0046] A feedforward network expert is used to process input data through a multilayer perceptron structure, perform nonlinear mapping, and extract high-order features. The representation formula of the feedforward network expert includes:

[0047] ;

[0048] in, is the feedforward network expert; is the target data, which is used to characterize the feature vector of the current sample; is the output weight matrix of the feedforward network expert, used for performing linear transformation on the target data; is an activation function, used to perform nonlinear transformation on the target data; and are the weights and biases of the hidden layer; is a bias term of the feedforward network expert, used to adjust the output result of the feedforward network expert;

[0049] A constant processing expert is used to perform a weighted combination of the dynamically calculated weight and the constant vector. The characterization formula of the constant processing expert includes:

[0050] ;

[0051] ;

[0052] in, Experts in processing said constants; is the target data, which is used to characterize the feature vector of the current sample; is a fixed constant vector; is a weight matrix used to perform a linear transformation on the target data; is the bias term; To pass The weight coefficient calculated by the normalized exponential function; is the input transformation function;

[0053] A replication expert, the replication expert is used to output the target data or perform nonlinear activation on the target data. The representation formula of the replication expert includes:

[0054] ;

[0055] ;

[0056] in, delivering experts for said copy; is the target data, which is used to characterize the feature vector of the current sample; It is an adjustable coefficient with a value of 1 or 0, which is used to control the proportion of replication. =1, then copy the input directly, if =0, no calculation is performed and the output is output directly; is an activation function used to introduce nonlinear changes; and are the weight matrix and bias term.

[0057] In a second aspect, embodiments of the present application further provide a data processing system for a low-energy, large language model based on a momentum mechanism and multiple types of experts, including:

[0058] A model training unit, wherein the model training unit is used to obtain a hybrid expert model, wherein the hybrid expert model includes a gating network, a combination network, and multiple expert networks;

[0059] A data input unit, configured to obtain target data based on a lazy loading mechanism and input the target data into the hybrid expert model;

[0060] A target network unit, configured to obtain, through the gating network, fitness scores of the target data corresponding to the respective expert networks, and sort the plurality of expert networks based on the fitness scores, and obtain the expert network that meets the sorting requirements as a target network;

[0061] The data output unit is used to obtain the output data of the target data in the target network, and perform weighted summation of multiple output data through the combination network to obtain a final output result.

[0062] Beneficial effect: Compared with the prior art, the embodiment of the present application provides a data processing method for a low-energy large language model based on a momentum mechanism and multiple types of experts, including obtaining a hybrid expert model, the hybrid expert model including a gating network, a combination network and multiple expert networks; obtaining target data based on a lazy loading mechanism and inputting it into the hybrid expert model; obtaining the fitness scores of each expert network corresponding to the target data through the gating network, and sorting the multiple expert networks based on the fitness scores, obtaining the expert network that meets the sorting requirements as the target network; obtaining the output data of the target data in the target network, and weightedly summing the multiple output data through the combination network to obtain the final output result. The present application sets multiple expert networks in the hybrid expert model, performs fitness ranking on the fitness scores of the target data, selects the output of the expert network that meets the ranking requirements and performs weighted summing to obtain the final output, thereby reducing the number of expert networks for forward and reverse calculations, reducing the overhead of floating-point operations and computing memory, improving computing efficiency, and reducing the waste of memory resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1A flowchart of the steps of the data processing method for a low-energy large language model based on a momentum mechanism and multiple types of experts provided in an embodiment of the present application;

[0065] Figure 2 A flowchart of the steps of acquiring target data based on a lazy loading mechanism in a data processing method for a low-energy large language model based on a momentum mechanism and multiple types of experts provided in an embodiment of the present application;

[0066] Figure 3 A flowchart of the steps for optimizing the model by combining historical gating network information and the momentum mechanism in the data processing method for a low-energy large language model based on the momentum mechanism and multiple types of experts provided in an embodiment of the present application;

[0067] Figure 4 A flowchart of the steps for obtaining a hybrid expert model in the data processing method for a low-energy large language model based on a momentum mechanism and multiple types of experts provided in an embodiment of the present application;

[0068] Figure 5 A unit connection diagram of a data processing system for a low-energy, large language model based on a momentum mechanism and multiple types of experts provided in an embodiment of the present application;

[0069] Figure 6 A schematic diagram of a data processing method for a low-energy, large language model based on a momentum mechanism and multiple types of experts provided in an embodiment of the present application;

[0070] Figure numerals: 10, model training unit; 20, data input unit; 30, target network unit; 40, data output unit. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0072] Sparse Mixture of Experts (SMoE) is a machine learning approach that partitions an AI model into separate subnetworks, or “experts,” that each specialize in a subset of the input data in order to jointly perform a task.

[0073] In existing sparse mixture of experts (MoE) architectures, the expert selection mechanism is static and fixed, resulting in some simple inputs being processed by too many experts, while complex inputs fail to fully utilize sufficient expert resources. This imbalance in computing resource allocation leads to low computational efficiency and resource waste.

[0074] At the same time, existing training methods usually require all data to be loaded into memory before training begins, which places extremely high demands on memory and storage space. Especially when facing ultra-large-scale data sets, the memory consumption is extremely large and it is difficult to adapt to scenarios with limited hardware resources.

[0075] In addition, expert selection and routing decisions in traditional MoE architectures are usually calculated based on a static gating mechanism and lack reference to historical routing decisions. This may cause routing decisions to fluctuate dramatically at different training stages, affecting the stability of training and the final performance.

[0076] In addition, in large-scale distributed training, due to uneven loads among different GPUs, the computing resources of some GPUs are not fully utilized, resulting in low computing efficiency and possible computing bottlenecks.

[0077] In view of this, an embodiment of the present application provides a data processing method for a low-energy large language model based on a momentum mechanism and multiple types of experts, including obtaining a hybrid expert model, the hybrid expert model including a gating network, a combination network and multiple expert networks; obtaining target data based on a lazy loading mechanism and inputting it into the hybrid expert model; obtaining the fitness scores of each expert network corresponding to the target data through the gating network, and sorting the multiple expert networks based on the fitness scores, obtaining the expert network that meets the sorting requirements as the target network; obtaining the output data of the target data in the target network, and weightedly summing the multiple output data through the combination network to obtain the final output result. The present application sets multiple expert networks in the hybrid expert model, performs fitness ranking on the fitness scores of the target data, selects the output of the expert network that meets the ranking requirements for weighted summing to obtain the final output, reduces the number of expert networks for forward and reverse calculations, reduces the overhead of floating-point operations and computing memory, improves computing efficiency, and reduces the waste of memory resources.

[0078] In some embodiments, see Figure 1 and Figure 6 , Figure 1 A flowchart of the steps of the data processing method of the low-energy large language model based on the momentum mechanism and multi-type experts provided in the embodiment of the present application, Figure 6This is a schematic diagram of a data processing method for a low-energy large language model based on a momentum mechanism and multiple types of experts provided in an embodiment of the present application. The data processing method for a low-energy large language model based on a momentum mechanism and multiple types of experts provided in an embodiment of the present application is specifically implemented through steps 100 to 400:

[0079] Step 100: Obtain a hybrid expert model, where the hybrid expert model includes a gating network, a combination network, and multiple expert networks.

[0080] In some embodiments, the present application introduces a multi-type expert mechanism to enhance the model's ability to process different inputs, improve computing efficiency, reduce redundant calculations, and enhance the model's flexibility and robustness. Understandably, in a traditional MoE architecture, all experts typically perform the same tasks and functions, but when faced with diverse tasks, this unified expert design easily leads to a waste of computing resources. In order to improve computing efficiency and processing power, the present application proposes a diversified expert mechanism, including:

[0081] Shared experts are used to share global features and provide general computing. Shared experts are responsible for general computing tasks and ensure that global features are shared between inputs. The characterization formula of shared experts includes:

[0082] ;

[0083] in, To be a shared expert; is the target data, which is used to characterize the feature vector of the current sample; is the weight matrix of the shared expert, which is used to represent the transformation of the shared expert on the target data; is the bias term of the shared expert, which is used to adjust the output results of the shared expert;

[0084] Routing experts include multiple sub-expert networks. Based on the different characteristics of the target data, different sub-expert networks are selected for data processing. The multiple sub-expert networks include:

[0085] Feedforward network experts are used to process input data through a multi-layer perceptron structure, perform nonlinear mapping, and extract high-order features. Feedforward network experts are experts at processing complex input data and are particularly suitable for tasks that require deep feature extraction, such as text classification and semantic understanding. The representation formulas for feedforward network experts include:

[0086] ;

[0087] in, Experts in feedforward networks; is the target data, which is used to characterize the feature vector of the current sample; is the output weight matrix of the feedforward network expert, which is used to perform linear transformation on the target data; It is an activation function, such as ReLU, GELU, etc., which is used to perform nonlinear transformation on the target data; and are the weights and biases of the hidden layer; is the bias term of the feedforward network expert, which is used to adjust the output results of the feedforward network expert;

[0088] The Constant Processing Expert is used to perform a weighted combination of dynamically calculated weights and constant vectors. Through the adjustable parameters learned during training, the Constant Processing Expert can adaptively adjust the output of each input based on the characteristics of the input data, thereby optimizing the calculation process and model performance. The Constant Processing Expert is mainly used to standardize input data, introduce a fixed reference signal, or smooth changes in input features. The characterization formula of the Constant Processing Expert includes:

[0089] ;

[0090] ;

[0091] in, Experts in constant processing; is the target data, which is used to characterize the feature vector of the current sample; It is a fixed constant vector, which is usually learned during the training phase and represents a specific standard or benchmark value; is the weight matrix, which is used to perform linear transformation on the target data. The weight matrix is ​​obtained through adaptive learning during the model training process; It is a bias term used to fine-tune the input data to make the output more flexible; To pass The weight coefficient calculated by the normalized exponential function determines the target data and constant vector influence in the final output; The input transformation function may be the target data, depending on the task requirements. Nonlinear mapping, such as through activation functions (such as 、 etc.) to transform.

[0092] Understandably, among constant processing experts, the target data and constant vector By weight coefficient and The constant processing expert controls the influence of input features and constant vectors on the final output by adaptively adjusting these weights. In particular, Function on target data The processing weights and , so that each input can be flexibly adjusted under different tasks, thereby enhancing the adaptability of the model to task requirements.

[0093] Copy transfer experts are used to output target data or perform nonlinear activation on target data. Copy transfer experts are suitable for inputs that do not require processing and can quickly pass input data to downstream layers, saving computing resources and avoiding redundant calculations. When directly copying to the output, the representation formula of the copy transfer expert includes:

[0094] ;

[0095] When nonlinear activation is required, the characterization formula of the copy transfer expert includes:

[0096] ;

[0097] in, for copy delivery experts; is the target data, which is used to characterize the feature vector of the current sample; It is an adjustable coefficient with a value of 1 or 0, which is used to control the proportion of replication. =1, then copy the input directly, if =0, no calculation is performed and the output is output directly; is the activation function, such as or , used to introduce nonlinear changes when needed; and are the weight matrix and bias term.

[0098] Understandably, the purpose of the copy transfer expert is to efficiently transfer input data without performing additional calculations when faced with simple inputs. and activation function , increases the flexibility of the copy transfer expert, making it not only suitable for simple transfer tasks, but also slightly adjusted for specific scenarios. This enables the copy transfer module to provide more computational resource savings for the network without requiring deep calculations.

[0099] In some embodiments, see Figure 4 , Figure 4 This is a flowchart of the steps for obtaining a hybrid expert model in the data processing method for a low-energy large language model based on a momentum mechanism and multiple types of experts provided in an embodiment of the present application. The method for obtaining the hybrid expert model is specifically implemented through steps 110 to 130:

[0100] Step 110: Determine a hybrid expert model to be trained and a plurality of computing nodes.

[0101] Step 120: Split the hybrid expert model to be trained and deploy it on multiple computing nodes.

[0102] Step 130: Based on the ZeRO Stage-3 optimizer in the DeepSpeed ​​training model framework, the hybrid expert model to be trained is distributedly trained on multiple computing nodes to obtain the trained hybrid expert model.

[0103] In some embodiments, during the distributed training process, the present application synchronously updates the gradients of multiple computing nodes based on the full-subtraction operation, merges multiple gradients, and updates the hybrid expert model.

[0104] In some embodiments, during distributed training, the present application dynamically adjusts the workload of computing nodes through a load balancing mechanism to ensure uniform distribution of computing tasks and avoid uneven load between computing nodes. The characterization formula includes:

[0105] ;

[0106] in, For the The load of the computing node is used to characterize the The proportion of computing tasks of computing nodes to the total tasks; For the The computing power or task volume of each computing node is used to characterize the The amount of data that a computing node currently needs to process; is the total number of computing nodes.

[0107] It is understandable that in order to process large-scale data sets and improve training efficiency, this application introduces distributed training and load balancing mechanisms to ensure optimal utilization of computing resources, optimize computing resource allocation, and improve training efficiency under multiple nodes. This application ensures efficient training of large-scale data sets by combining distributed training architecture and load balancing mechanism. Distributed training accelerates the training process through parallel computing, and ensures the consistency of gradient updates through All-Reduce operations. The load balancing mechanism dynamically adjusts the task volume of the computing node to ensure full utilization of computing resources and avoid the occurrence of computing bottlenecks. Through these optimizations, this application can greatly improve the efficiency of large language model training and enhance the stability of the training process.

[0108] In some embodiments, the steps of training the hybrid expert model are as follows:

[0109] Learning rate warm-up: In the early stages of training, the learning rate is warmed up from the initial value It starts to increase gradually, which helps to avoid drastic changes in the early stages of the network and instability in the training process. As the training progresses, the learning rate gradually increases until it reaches a maximum value. , learning rate The characterization formula is:

[0110] ;

[0111] After reaching the maximum learning rate, the learning rate Will gradually return to 0:

[0112] ;

[0113] in, is the maximum learning rate; The duration of the learning rate warmup; is the number of training steps; The maximum number of training steps in the training process.

[0114] Understandably, by gradually increasing the learning rate, the model can better adapt to the training process and avoid instability in training caused by an excessively large learning rate in the early stage.

[0115] Loss Function and Optimizer

[0116] In this application, the cross-entropy loss function (Cross-Entropy Loss, ) to calculate the prediction loss of the model and compare it with the load loss (Load Loss, ) to ensure that the model can be adaptively optimized according to the characteristics and complexity of the data. The formula is as follows:

[0117] ;

[0118] in, is the total loss function; is the cross entropy loss, which is used to calculate the error between the model output and the true label; Load loss is used to control task distribution and avoid overloading; A hyperparameter that controls the balance between load loss and cross entropy loss.

[0119] use or The optimizer, combined with gradient clipping technology, stabilizes model training and ensures the stability of gradient updates:

[0120] ;

[0121] in, is the gradient clipping value; The threshold for gradient clipping is used to prevent gradient explosion; Gradient clipping.

[0122] Performance Evaluation

[0123] Prerequisite: After training is complete, an independent validation set is needed to evaluate the model to check its performance on new data.

[0124] Evaluation Methodology:

[0125] Accuracy: Measures the proportion of samples that the model predicts correctly. The formula is as follows:

[0126] ;

[0127] in, For the The predicted labels of the expert network; For the The true labels of the expert network; For the The indicator function of the expert network indicates whether the prediction is correct; is the sample set of the validation set.

[0128] Perplexity: measures the predictive ability of the model when generating sequences, usually used in language modeling tasks. Its formula is as follows:

[0129] ;

[0130] in, is the model's predicted probability for a given input; For the Target data.

[0131] Early Stopping: During the evaluation process, if the model's performance on the validation set stops improving, the early stopping mechanism is triggered to avoid overfitting and stop training.

[0132] This application designs a learning rate adjustment strategy, a combination of loss function and optimizer, and a model performance evaluation method to ensure the efficiency and stability of the training process. By gradually increasing the learning rate, combining the optimizer, and designing an appropriate loss function, the model can converge more stably. At the same time, combined with evaluation metrics such as accuracy and perplexity, we ensure that the model's performance on the test set meets the expected requirements.

[0133] Step 200: Obtain target data based on the lazy loading mechanism and input it into the hybrid expert model.

[0134] Understandably, in traditional training, the entire dataset typically needs to be loaded into memory at once, placing extremely high demands on memory resources, especially when processing very large datasets. This invention introduces a lazy loading mechanism and uses PyTorch's IterableDataset class to allow data to be read and processed line by line during training. During each training session, only the samples required for the current batch are loaded, and tokenization, encoding, and padding are performed before feeding the model. Each training sample is dynamically loaded and tokenized, encoded, and padded as needed, avoiding the need to load the entire dataset into memory. This effectively reduces memory consumption and improves training efficiency.

[0135] In some embodiments, see Figure 2 , Figure 2 This is a flowchart of the steps of acquiring target data based on a lazy loading mechanism in a data processing method for a low-energy large language model based on a momentum mechanism and multiple types of experts provided in an embodiment of the present application. The method for acquiring target data based on a lazy loading mechanism is implemented through steps 210 to 230:

[0136] Step 210: Determine the data set to be processed.

[0137] Step 220: Read the data set line by line, and obtain the data required by the target batch as the data to be processed.

[0138] Step 230: Perform word segmentation, encoding, and filling processing on the data to be processed to obtain target data.

[0139] Understandably, the dataset loading process can be defined in the following way:

[0140] ;

[0141] in, is a data set where each data point It is a pairing of input features and corresponding labels, used to represent target data. ; Each time you train, only load the samples needed for the current batch , perform word segmentation, encoding and padding before sending it to the model for training.

[0142] Step 300: Obtain the fitness scores of each expert network corresponding to the target data through the gating network, and sort the multiple expert networks based on the fitness scores, and obtain the expert network that meets the sorting requirements as the target network.

[0143] In some embodiments, the target data is used to calculate the fitness score of each expert network through a gating function. The expert network with a higher fitness score will contribute more to the processing of the target data. The calculation formula is as follows:

[0144] ;

[0145] in, is the total number of expert networks; For the The fitness score of an expert network reflects the fitness of the expert network for the target data. adaptability; For the The gating function of the expert network is input by the target data To calculate the fitness score of each expert network, the gating function is used to evaluate the contribution of each expert network to the current task. Experts with higher fitness scores will occupy a more important position in the final calculation.

[0146] In some embodiments, the present application The strategy with the highest fitness score is selected An expert network is used as the target network to reduce computational overhead and optimize the utilization of computing resources. Its characterization formula includes:

[0147] ;

[0148] in, For selected The set of expert networks with the highest fitness scores is used to represent the set of target networks; To select the top fitness score from all expert networks expert network.

[0149] Understandably, this application uses a gating network to select experts. Its core purpose is to dynamically select the most suitable expert for computation based on the characteristics of the input data. By introducing gating functions and the Top-K strategy, we can optimize the allocation of computing resources while ensuring computational efficiency, thereby improving the overall performance and stability of large language models.

[0150] In some embodiments, in order to further optimize the selection process of each round of expert networks and improve the stability and training efficiency of the model, the present application combines the memory function and momentum mechanism of historical gating network information in the routing expert. The gating network can refer to historical information when selecting the expert network, avoiding the drastic fluctuations caused by frequent updates during the training process. For details, please refer to Figure 3 , Figure 3This is a flowchart of the steps for optimizing the model by combining historical gating network information and momentum mechanism in the data processing method for a low-energy large language model based on momentum mechanism and multi-type experts provided in an embodiment of the present application. The method for optimizing the model by combining historical gating network information and momentum mechanism is specifically implemented through steps 310 to 320:

[0151] Step 310: Obtain historical data of the gating network selecting the target network in the expert network;

[0152] Step 320: Optimize the current selection result of the gating network for the target network based on historical data.

[0153] Specifically, the characterization formula for model optimization considering historical gating network information includes:

[0154] ;

[0155] in, is the current selection result of the target network, which is used to characterize the currently selected expert network; is historical data, used to characterize the expert network selected in the previous round; is the attenuation factor of historical data, ; is the currently selected expert network; the gating network uses a momentum mechanism to dynamically adjust the routing expert, which is used to optimize the selection process of multiple sub-expert networks in combination with learning rate decay. The characterization formula for updating the momentum mechanism includes:

[0156] ;

[0157] in, for Momentum at the moment, used to characterize the momentum at the current moment; for The gating score at the moment is used to represent the gating score at the current moment; is the momentum coefficient; for Momentum at a moment, used to characterize the momentum at the previous moment; for The gate output at the moment is used to represent the gate output at the current moment. , for The gate output at the moment is used to represent the gate output at the previous moment. is the coefficient of learning rate decay.

[0158] It can be understood that by memorizing historical selection information, the model can refer to previous decisions when selecting the current layer, making each decision smoother and reducing unnecessary jumps.

[0159] Alternatively, the characterization formula for model optimization considering historical gating network information includes:

[0160] ;

[0161] in, is the current selection result of the target network, which is used to characterize the currently selected expert network; is historical data, used to characterize the expert network selected in the previous round; is the momentum factor, ; The expert network currently selected. As can be understood, the momentum mechanism smoothes each selection by combining the historical decisions of the previous layer with the current decision, reducing the noise caused by gradient updates, avoiding oscillations caused by overly fast update pace, and ensuring training stability.

[0162] Step 400: Obtain output data of the target data in the target network, and perform weighted summation of multiple output data through the combined network to obtain a final output result.

[0163] In some embodiments, the present application selects The outputs of the target networks are weighted and summed to ensure that the contribution of each expert network is weighted according to its score. The weighted summation formula is:

[0164] ;

[0165] in, The final output result is The weighted sum of the outputs of the expert networks; For the The gating weight of the target network represents the activation strength of the target network, usually through The function is calculated; For the Target data of target network The calculation output; understandably, through the weighted sum strategy, the final output result It is a weighted combination of all selected expert networks, ensuring that the most suitable expert network contributes the most.

[0166] It can be understood that the data processing method of the low-energy large language model based on the momentum mechanism and multiple types of experts provided in the embodiment of the present application includes obtaining a hybrid expert model, which includes a gating network, a combination network and multiple expert networks; obtaining target data based on a lazy loading mechanism and inputting it into the hybrid expert model; obtaining the fitness scores of each expert network corresponding to the target data through the gating network, and sorting the multiple expert networks based on the fitness scores, obtaining the expert network that meets the sorting requirements as the target network; obtaining the output data of the target data in the target network, and weighted summing the multiple output data through the combination network to obtain the final output result. The present application sets multiple expert networks in the hybrid expert model, performs fitness ranking on the fitness scores of the target data, selects the output of the expert network that meets the ranking requirements for weighted summing to obtain the final output, reduces the number of expert networks for forward and reverse calculations, reduces the overhead of floating-point operations and computing memory, improves computing efficiency, and reduces the waste of memory resources.

[0167] For example, to verify the feasibility and practical effect of this application, this example completes the large language model training task of sparse mixed expert and multi-type expert mechanism based on a distributed system composed of seven NVIDIA RTX 3090 graphics cards. The following are the detailed implementation steps of this embodiment:

[0168] Step 1: Model framework construction and expert structure design:

[0169] This application example builds a neural network model based on the Sparse Mixture of Experts (MoE) architecture. The backbone network adopts a 16-layer Transformer structure, and an MoE expert sublayer is inserted in the feedforward submodule position of each layer to improve the model's expressiveness and computational efficiency in large-scale tasks.

[0170] In the MoE module, the expert pool contains six functionally heterogeneous expert networks, using a top-2 sparse activation strategy. Each token is assigned only to the two expert networks with the highest scores, significantly reducing computational overhead. The specific expert networks and their functions are as follows:

[0171] Shared Experts (2): Each shared expert consists of two layers of Multi-Layer Perceptrons (MLPs) with a structure of 1024→4096→1024, using the Gaussian Error Linear Unit (GELU) as the activation function, and supports input and output dimensions of [BatchSize, SeqLen, 1024]. Shared experts are primarily responsible for processing high-semantic complexity tokens on the backbone path, such as abstract reasoning or cross-sentence alignment tasks. They have strong nonlinear transformation capabilities, and their parameters can be shared across multiple layers, improving training stability and generalization.

[0172] Routing Experts (4):

[0173] Feedforward Network (FFN) Experts (2): Utilizing a two-layer MLP structure (1024 → 4096 → 1024), these experts are primarily used to supplement or smooth token allocations when gating score differences are small. These experts have strong routing guidance capabilities in the early stages of training, helping the system form a stable expert activation path, reducing training oscillations, and assisting in rapid MoE network convergence.

[0174] Constant Processing Expert (1): Always outputs a fixed vector (such as all zeros or a mean embedding vector). Its role is to handle empty tokens, padding positions, or inputs with small semantic contributions, preventing invalid tokens from perturbing the model computation graph and improving overall routing efficiency.

[0175] Copy and transfer expert (1): Implements identity mapping, meaning the output is equal to the input. This expert is suitable for situations where feature transformation is not required, such as intermediate layer token transfer. It maintains context continuity, reduces information loss, and provides a stable channel during inference.

[0176] All expert network inputs and outputs support tensors of shape [BatchSize, SeqLen, HiddenDim=1024], seamlessly integrating with the output of standard Transformer embedding layers. Each MoE sublayer in the model is equipped with a gating network. Logits output by the linear layer are normalized using softmax, and then two activated expert networks are selected using a top-2 strategy. Their outputs are weightedly fused based on the assigned weights.

[0177] To prevent gating from converging to a single expert assignment, this application introduces a Dropout mechanism (with a probability of 0.1) into the gating network and adds a gating entropy regularization term to maintain expert load balance.

[0178] Initial training parameters were initialized using a normal distribution (mean 0, standard deviation 0.02). The AdamW optimizer was used, with an initial learning rate of 3e-4. Linear warmup (for the first 10,000 steps) and a cosine decay strategy were used, and the epoch count was set to 10. The batch size per round was 64, corresponding to a total of 32,768 tokens processed. The entire model was deployed on a distributed training platform consisting of seven NVIDIA RTX 3090 graphics cards. Expert parallelism and memory optimization were implemented using the DeepSpeed ​​framework, employing the Zero-2 graphics memory conservation strategy.

[0179] Through the rational design and division of labor of the above-mentioned expert structure, the model has stronger dynamic modeling capabilities and computing flexibility, laying the foundation for subsequent routing mechanisms, training strategies and inference acceleration.

[0180] Step 2: Routing strategy and momentum mechanism implementation:

[0181] To address the problems of load imbalance, high activation conflict rate and poor routing stability in the expert selection process in the MoE architecture, this application proposes a momentum-guided Top-2 sparse routing strategy, and combines it with a lightweight gating network structure to implement an efficient and scalable expert scheduling mechanism.

[0182] In this example, each MoE module embeds a lightweight router network, whose core structure is a two-layer multi-layer perceptron (MLP) with Relu activation and LayerNorm normalization. This router takes the hidden state vector of each token as input and outputs a score vector for all expert networks. A top-2 selection mechanism is then used to select the two highest-scoring expert networks from the six expert networks for activation and computational tasks, resulting in the final fused output. To ensure gradient propagation, a gated softmax followed by a top-2 truncation is used, and the Gumbel-Softmax approximate sampling method is applied to achieve sparse routing.

[0183] To address the problem of experts tending to converge to "some hot experts" in the early stages of training, the system introduces a historical momentum correction term into the gated score calculation. This momentum term is dynamically updated based on the frequency of each expert being selected over the past period of time, and a reverse penalty is added to the current routing score, namely:

[0184] ;

[0185] in, is the historical momentum correction term; For the The current score of the expert network; For its recent The average selection frequency (momentum value) of a batch; is the momentum adjustment coefficient (set to 0.3). This strategy can guide routers to avoid continuously selecting high-frequency experts, achieving dynamic balancing of global routing load. At the same time, it uses the momentum mechanism to dynamically adjust routing experts and combines it with learning rate decay to optimize the training process, thereby improving the model convergence speed and stability.

[0186] To further improve routing efficiency and conflict handling, a "conflict identification and reallocation mechanism" has been introduced. When multiple tokens are assigned to the same expert on the same GPU node, an alternate allocation process is triggered based on the expert's current load threshold. If the maximum token capacity (set to 128) is exceeded, the system searches for a backup option among the top two candidate experts and implements "suboptimal expert reuse," thus avoiding local routing bottlenecks and improving computational throughput.

[0187] Step 3: Expert capacity and conflict resolution mechanism:

[0188] During the training and inference of large-scale sparse expert models, dynamic token allocation can easily lead to excessive load on local expert networks, potentially causing cache overflows, increased communication latency, and gradient imbalance. To address this, this application proposes a set of expert capacity limiting mechanisms and dynamic conflict mitigation strategies based on the routing module to ensure stable system operation and optimal computing resource utilization in distributed scenarios.

[0189] First, this application sets a maximum token capacity for each expert. In a cluster configuration of seven RTX 3090 GPUs, taking into account the memory capacity and computing bandwidth, the default capacity for a single expert is set at 128 tokens / step. Any excess is considered a conflict. This value can be adjusted dynamically between 64 and 128 depending on the graphics card configuration and batch size, and the optimal threshold is determined through empirical curve regression.

[0190] To mitigate the imbalance in token distribution during actual training, this system introduces a Token Overflow Predictor mechanism. After completing the initial allocation of the top-two experts, the system calculates the number of tokens currently received by each expert network and labels any expert network that exceeds its capacity as a "congested expert." For tokens assigned to these expert networks, the system prioritizes finding a second-best option among their top-two candidates. If a reconciliation is still unsuccessful, a suboptimal routing fallback mechanism is activated, temporarily transferring the tokens to a "replicated expert" among the shared experts to ensure uninterrupted computation.

[0191] In addition, the system has designed an expert load balancing controller, which records the token allocation of all expert networks in the current mini-batch and updates it with an exponential sliding average. When the expert load is seriously uneven (the difference between the maximum and minimum token numbers is greater than 40%), a mild gate resampling mechanism is automatically triggered, which weights the scores of the top-2 experts with low load. , prompting the gate output to be fine-tuned to it, in order to dynamically guide a more balanced expert usage frequency.

[0192] Experimental results show that when this mechanism is not used, the average number of tokens activated per round by the hottest expert is 143, while that of the coldest expert is only 31. After adopting the capacity control and conflict resolution mechanism proposed in this application, this value is optimized to an average of 17 for the hottest expert and 95 for the coldest expert. The variance of the expert load is reduced by more than 60%, the overall computing load is more evenly distributed, and the memory utilization rate is increased from 78.4% to 91.6%.

[0193] Finally, to achieve modularity and ease of deployment, all conflict detection and rerouting modules are registered as plug-in components in the forward process of the MoE main framework. They are compatible with DeepSpeed ​​and Megatron-LM, and support full token packet transfer and conflict resolution in cross-card all-to-all communication, ensuring stability and high throughput in actual large-scale deployments.

[0194] Step 4: Distributed training and video memory optimization:

[0195] This embodiment is deployed on a cluster consisting of 7 NVIDIA RTX 3090 graphics cards (24GB of video memory). It uses the ZeRO-3 optimizer in the DeepSpeed ​​framework as the core engine for distributed training, and combines expert parallelism (ExpertParallelism) and data parallelism (Data Parallelism) for hybrid parallel acceleration.

[0196] To support the training requirements of the proposed sparse mixture of experts (MoE) model for large-scale semantic tasks, improve computational efficiency, and reduce hardware resource consumption, this step develops a distributed training and memory optimization strategy based on the DeepSpeed ​​framework. This strategy integrates expert parallelism, data parallelism, and mixed-precision training mechanisms to ensure stable and efficient model operation on multi-GPU platforms.

[0197] Specifically, this example is deployed on a cluster consisting of seven NVIDIA RTX 3090 graphics cards, each with 24GB of video memory. The training framework implements multi-expert parallelism based on the DeepSpeed ​​and NCCL communication libraries, and adopts a Top-2 sparse activation strategy to ensure that each token is assigned to only two experts. By allocating the six experts to different GPUs on demand, heterogeneous load distribution among experts is achieved, significantly reducing repeated calculations and cross-card bandwidth pressure. Combined with the expert capacity management mechanism, the system strictly limits the maximum number of tokens processed per GPU in a single step to between 128 and 140, ensuring the stability of the training process.

[0198] In terms of video memory optimization, this application uses ZeRO-3 parameter sharding technology to divide the model weights, gradients, and optimizer states among multiple GPUs for storage, and cooperates with full FP16 mixed precision training to reduce the single-card video memory usage from the original 22.3GB to 13.5GB, saving nearly 40% of video memory space. All training data and gradients are efficiently exchanged through All-to-All reconstruction operations during the communication phase, and only necessary activation data is transmitted to avoid redundant communication. This strategy effectively reduces the cross-GPU communication delay from 41.2ms to 24.5ms, improving communication efficiency by more than 40%.

[0199] To ensure the stability of the training process, this application introduces forward-moving LayerNorm and gradient clipping mechanisms in each layer of the model, and cooperates with dynamic loss scaling strategies to avoid gradient explosion or precision loss problems in mixed precision training. In terms of training scheduling, the system uses dynamic batch scheduling strategies and The cosine decay learning rate adjustment scheme further improves the model convergence speed and generalization ability.

[0200] Experiments show that after enabling the optimization mechanism in this step, the overall training throughput of the MoE model increased by approximately 82%, memory utilization increased to 71.6%, and the expert conflict rate decreased to 3.6%. This training solution not only greatly improves the system's support for large-scale expert models, but also lays a solid foundation for subsequent inference acceleration and multi-task migration.

[0201] Step 5: Model effect comparison and load analysis:

[0202] To validate the performance improvements and system load optimization effects of the proposed sparse mixture of experts (MoE) model in multi-task language understanding, we designed a set of systematic comparative experiments. We analyzed these models from various perspectives, including accuracy assessment, training stability, memory utilization, and expert activation balance. We selected the current mainstream sparse expert systems Minimind, Switch Transformer, and native Dense Transformer models as benchmarks. All experiments were conducted on a unified hardware platform (7×NVIDIA RTX 3090, 98GB of total memory). The tasks involved mixed Chinese-English question-answering and instruction-following fine-tuning. The batch size was set to 64, the number of training epochs was 15, and the same warmup and learning rate decay strategies were used.

[0203] The experiment first evaluated accuracy. In the OpenAssistant-style question-answering task, the proposed model significantly outperformed other mainstream models in BLEU-4, ROUGE-L, and the average sentence readability index (PPL), with improvements of 3.1%, 2.4%, and a perplexity reduction of 9.7%, respectively. In particular, in high-complexity instruction set tasks, the proposed model achieved an accuracy improvement of over 5% while maintaining controllable inference time. Secondly, in terms of expert load balancing, the "momentum routing correction mechanism" and "conflict fallback mechanism" introduced in this application significantly alleviated the problem of expert conflict in Top-2 routing. The token activation gap between the hottest and coldest experts in this model was reduced from 112 (units / step) in the original model to 22, and the load standard deviation decreased by 62%.

[0204] The model's comprehension, reasoning, and generalization capabilities were evaluated on the CEVAL dataset, a comprehensive evaluation dataset for the Chinese Higher Education Entrance Examination (CEE), which covers 52 subjects in STEM, humanities and social sciences, law, and medicine. Using a 5-shot evaluation method, the proposed model achieved an average accuracy of 26.15%, a 1.51% improvement over Minimind and a 1.19% improvement over SwitchTransformer. The specific accuracy scores for each category are shown in the table below:

[0205]

[0206] Further analysis revealed that the improved MoE model performed more robustly in multi-domain knowledge transfer, demonstrating particularly strong reasoning and knowledge integration capabilities in computing (mathematics, physics) and law. This demonstrates that the model not only possesses high-performance generation and comprehension capabilities but also possesses scalable and specialized adaptability for real-world complex tasks.

[0207] Accordingly, the embodiment of the present application also provides a data processing system for a low-energy large language model based on a momentum mechanism and multiple types of experts, see Figure 5 , Figure 5 This is a unit connection diagram of a data processing system for a low-energy, large language model based on a momentum mechanism and multiple types of experts provided in an embodiment of the present application. The data processing system for a low-energy, large language model based on a momentum mechanism and multiple types of experts provided in an embodiment of the present application includes:

[0208] A model training unit 10 is used to obtain a hybrid expert model, where the hybrid expert model includes a gating network, a combination network, and multiple expert networks;

[0209] The data input unit 20 is used to obtain target data based on a lazy loading mechanism and input the target data into the hybrid expert model;

[0210] The target network unit 30 is used to obtain the fitness scores of each expert network corresponding to the target data through the gating network, and sort the multiple expert networks based on the fitness scores, and obtain the expert network that meets the sorting requirements as the target network;

[0211] The data output unit 40 is used to obtain the output data of the target data in the target network, and to obtain the final output result by weighted summing of multiple output data through the combination network.

[0212] In some embodiments, the data input unit 20 is specifically used to:

[0213] Identify the data set to be processed;

[0214] Read the dataset line by line and obtain the data required by the target batch as the data to be processed;

[0215] Perform word segmentation, encoding and filling on the data to be processed to obtain the target data.

[0216] In some embodiments, the target network unit 30 is specifically configured to:

[0217] pass The strategy with the highest fitness score is selected The expert network is the target network, and its characterization formula includes:

[0218] ;

[0219] in, For selected The set of expert networks with the highest fitness scores is used to represent the set of target networks; To select the top fitness score from all expert networks expert network.

[0220] In some embodiments, the data processing system is specifically configured to:

[0221] Obtain historical data of the gating network selecting the target network in the expert network;

[0222] The current selection result of the target network by optimizing the gating network based on historical data is represented by the following formula:

[0223] ;

[0224] in, is the current selection result of the target network, which is used to characterize the currently selected expert network; is historical data, used to characterize the expert network selected in the previous round; is the attenuation factor of historical data, ; is the currently selected expert network; the gating network uses a momentum mechanism to dynamically adjust the routing expert, which is used to optimize the selection process of multiple sub-expert networks in combination with learning rate decay. The characterization formula for updating the momentum mechanism includes:

[0225] ;

[0226] in, for Momentum at the moment, used to characterize the momentum at the current moment; for The gating score at the moment is used to represent the gating score at the current moment; is the momentum coefficient; for Momentum at a moment, used to characterize the momentum at the previous moment; for The gate output at the moment is used to represent the gate output at the current moment. , for The gate output at the moment is used to represent the gate output at the previous moment. is the coefficient of learning rate decay.

[0227] In some embodiments, the data processing system is specifically configured to:

[0228] Obtain historical data of the gating network selecting the target network in the expert network;

[0229] The current selection result of the target network by optimizing the gating network based on historical data is represented by the following formula:

[0230] ;

[0231] in, is the current selection result of the target network, which is used to characterize the currently selected expert network; is historical data, used to characterize the expert network selected in the previous round; is the momentum factor, ; The currently selected expert network.

[0232] In some embodiments, the model training unit 10 is specifically used to:

[0233] Determine the hybrid expert model to be trained and multiple computing nodes;

[0234] Split the hybrid expert model to be trained and deploy it on multiple computing nodes;

[0235] Based on the ZeRO Stage-3 optimizer in the DeepSpeed ​​training model framework, distributed training is performed on the hybrid expert model to be trained on multiple computing nodes to obtain the trained hybrid expert model.

[0236] In some embodiments, the model training unit 10 is specifically used to:

[0237] Synchronously update the gradients of multiple computing nodes based on the full-reduce operation;

[0238] Merge multiple gradients and update the mixture of experts model.

[0239] In some embodiments, the model training unit 10 is specifically used to:

[0240] The load balancing mechanism dynamically adjusts the workload of computing nodes. The characterization formulas include:

[0241] ;

[0242] in, For the The load of the computing node is used to characterize the The proportion of computing tasks of computing nodes to the total tasks; For the The computing power or task volume of each computing node is used to characterize the The amount of data that a computing node currently needs to process; is the total number of computing nodes.

[0243] The present application has provided a detailed introduction to a data processing method and system for a low-energy large language model based on a momentum mechanism and multiple types of experts provided in an embodiment of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A data processing method for a low-energy large language model based on momentum mechanism and multi-type experts, characterized by: include: Acquire a hybrid expert model, wherein the hybrid expert model includes a gating network, a combination network, and multiple expert networks; the multiple expert networks include a routing expert, and the routing expert includes multiple sub-expert networks; Acquire target data based on a lazy loading mechanism and input the data into the hybrid expert model; Obtaining the fitness scores of the target data corresponding to the respective expert networks through the gating network, and sorting the plurality of expert networks based on the fitness scores, and obtaining the expert network that meets the sorting requirements as the target network; Obtain historical data of the target network selected by the gating network in the expert network; The current selection result of the target network by the gating network is optimized based on the historical data, and the characterization formula includes: R current =λ·R previous +(1-λ)·R new ; Among them, R current is the current selection result of the target network, used to characterize the currently selected expert network; R previous is the historical data used to characterize the expert network selected in the previous round; λ is the attenuation factor of the historical data, 0≤λ≤1; R new The said expert network is currently selected; The gating network uses a momentum mechanism to dynamically adjust the routing expert, which is used to optimize the selection process of multiple sub-expert networks in combination with learning rate decay. The characterization formula for updating the momentum mechanism includes: P t =-f(x t )+μ·p t-1 ; Among them, p t is the momentum at time t, which is used to characterize the momentum at the current moment; f(x t ) is the gating score at time t, which is used to represent the gating score at the current moment; μ is the momentum coefficient; p t-1 is the momentum at time t-1, used to represent the momentum at the previous moment; x t is the gate output at time t, which is used to represent the gate output at the current moment, x t =x t-1 +γ·p t-1 , x t-1 is the gate output at time t-1, which is used to represent the gate output at the previous moment, and γ is the coefficient of learning rate attenuation; Output data of the target data in the target network is obtained, and a plurality of the output data are weightedly summed through the combination network to obtain a final output result.

2. The data processing method of low-energy large language model based on momentum mechanism and multi-type experts according to claim 1 is characterized in that: The steps of acquiring target data based on the lazy loading mechanism include: Identify the data set to be processed; Read the data set line by line, and obtain the data required by the target batch as the data to be processed; The data to be processed is segmented, encoded and filled to obtain the target data.

3. The data processing method of low-energy large language model based on momentum mechanism and multi-type experts according to claim 1 is characterized in that: The step of sorting the plurality of expert networks based on the fitness scores and obtaining the expert network that meets the sorting requirements as the target network includes: The Top-K strategy is used to select the K expert networks with the highest fitness scores as the target network. The characterization formula includes: ε K :Top-K(score1,score2,…,score M ); Among them, ε K The set of K selected expert networks with the highest fitness scores is used to represent the set of target networks; Top-K is the expert networks ranked in the top K in fitness scores selected from all the expert networks.

4. The data processing method of low-energy large language model based on momentum mechanism and multi-type experts according to claim 1 is characterized in that: The data processing method further includes: Obtain historical data of the target network selected by the gating network in the expert network; The current selection result of the target network by the gating network is optimized based on the historical data, and the characterization formula includes: R current =R previous +β·(R new -R previous ); Among them, R current is the current selection result of the target network, used to characterize the currently selected expert network; R previous is the historical data used to characterize the expert network selected in the previous round; β is the momentum factor, 0≤β≤1; R new is the expert network currently selected.

5. The data processing method of low-energy large language model based on momentum mechanism and multi-type experts according to claim 1 is characterized in that: The step of obtaining the hybrid expert model includes: Determining the hybrid expert model to be trained and a plurality of computing nodes; Splitting the hybrid expert model to be trained and deploying it on multiple computing nodes; Based on the ZeRO Stage-3 optimizer in the DeepSpeed ​​training model framework, distributed training is performed on the hybrid expert model to be trained on multiple computing nodes to obtain the trained hybrid expert model.

6. The data processing method of low-energy large language model based on momentum mechanism and multi-type experts according to claim 5 is characterized in that: The steps of the distributed training include: Synchronously updating the gradients of the plurality of computing nodes based on a full-subtraction operation; The plurality of gradients are merged to update the hybrid expert model.

7. The data processing method of low-energy large language model based on momentum mechanism and multi-type experts according to claim 6 is characterized in that: The distributed training step further includes: The task load of the computing node is dynamically adjusted based on the load balancing mechanism, and its characterization formula includes: Among them, Load i is the load of the i-th computing node, which is used to represent the proportion of the computing tasks of the i-th computing node to the total tasks; C i is the computing capacity or task volume of the i-th computing node, which is used to represent the amount of data that the i-th computing node currently needs to process; N is the total number of computing nodes.

8. The data processing method of low-energy large language model based on momentum mechanism and multi-type experts according to claim 1 is characterized in that: The multiple expert networks further include shared experts, which are used to share global features and provide general calculations. The representation formula of the shared experts includes: E global (x)=W global x+b global ; Among them, E global is the shared expert; x is the target data, which is used to characterize the feature vector of the current sample; W global is the weight matrix of the shared expert, used to characterize the transformation of the target data by the shared expert; b global is the bias term of the shared expert, used to adjust the output result of the shared expert; The plurality of sub-expert networks include: A feedforward network expert is used to process input data through a multilayer perceptron structure, perform nonlinear mapping, and extract high-order features. The representation formula of the feedforward network expert includes: E FFN (x)=W FFN ·σ(W hidde x+b hidde )+b FFN ; Among them, E FFN is the feedforward network expert; x is the target data, which is used to characterize the feature vector of the current sample; W FFN is the output weight matrix of the feedforward network expert, used to perform linear transformation on the target data; σ is the activation function, used to perform nonlinear transformation on the target data; W hidden and b hidd is the weight and bias of the hidden layer; b FFN is a bias term of the feedforward network expert, used to adjust the output result of the feedforward network expert; A constant processing expert is used to perform a weighted combination of the dynamically calculated weight and the constant vector. The characterization formula of the constant processing expert includes: E constant (x)=α1·f(x)+α2·v; [α1,α2]=softmax(W c ·x+b c ); Among them, E constant is the constant processing expert; x is the target data, which is used to characterize the feature vector of the current sample; v is a fixed constant vector; W c is a weight matrix used to perform linear transformation on the target data; b c is the bias term; α1, α2 are weight coefficients calculated by the Softmax normalized exponential function; f is the input transformation function; A replication expert, the replication expert is used to output the target data or perform nonlinear activation on the target data. The representation formula of the replication expert includes: E replicate (x)=λ x+(1-λ) x; E replicate (x)=λ·σ(W replicate x+b replicate )+(1-λ)·x; Among them, E replicate is the replication expert; x is the target data, which is used to characterize the feature vector of the current sample; λ is an adjustable coefficient, which takes a value of 1 or 0 and is used to control the replication ratio. If λ = 1, the input is directly replicated; if λ = 0, no calculation is performed and the output is directly output; σ is an activation function used to introduce nonlinear changes; W replicate and b replicate are the weight matrix and bias term.

9. A data processing system for a low-energy, large language model based on momentum mechanism and multi-type experts, characterized by: include: A model training unit (10), the model training unit (10) is used to obtain a hybrid expert model, the hybrid expert model includes a gating network, a combination network and multiple expert networks; the multiple expert networks include a routing expert, and the routing expert includes multiple sub-expert networks; A data input unit (20), the data input unit (20) is used to obtain target data based on a lazy loading mechanism and input the target data into the hybrid expert model; A target network unit (30) is used to obtain the fitness scores of the target data corresponding to each of the expert networks through the gating network, and to sort the multiple expert networks based on the fitness scores, and to obtain the expert network that meets the sorting requirements as the target network; and to obtain historical data of the target network selected by the gating network from the expert networks; The current selection result of the target network by the gating network is optimized based on the historical data, and the characterization formula includes: R current =λ·R previous +(1-λ)·R new ; Among them, R current is the current selection result of the target network, used to characterize the currently selected expert network; R previous is the historical data used to characterize the expert network selected in the previous round; λ is the attenuation factor of the historical data, 0≤λ≤1; R new The said expert network is currently selected; The gating network uses a momentum mechanism to dynamically adjust the routing expert, which is used to optimize the selection process of multiple sub-expert networks in combination with learning rate decay. The characterization formula for updating the momentum mechanism includes: P t =-f(x t )+μ·p t-1 ; Among them, p t is the momentum at time t, which is used to characterize the momentum at the current moment; f(x t ) is the gating score at time t, which is used to represent the gating score at the current moment; μ is the momentum coefficient; p t-1 is the momentum at time t-1, used to represent the momentum at the previous moment; x t is the gate output at time t, which is used to represent the gate output at the current moment, x t =x t-1 +γ·p t-1 , x t-1 is the gate output at time t-1, which is used to represent the gate output at the previous moment, and γ is the coefficient of learning rate attenuation; The data output unit (40) is used to obtain output data of the target data in the target network, and perform weighted summation of multiple output data through the combination network to obtain a final output result.

Images