Expert model training method and device, storage medium and electronic equipment

Abstract

The invention discloses an expert model training method and device, a storage medium and electronic equipment, and relates to the technical field of computers, and the method comprises the steps: determining a target expert network group from a plurality of expert network groups included in an initial expert model according to an estimated resource occupation condition of input data; taking a plurality of expert networks in the target expert network group as a current expert network in sequence, and determining an individual difference evaluation coefficient matched with the current expert network; determining at least one target expert network from the plurality of expert networks according to the individual difference evaluation coefficients of the plurality of expert networks in the target expert network group; and processing the input data according to the at least one target expert network, and adjusting the model parameters in the initial expert model according to the load state description information corresponding to the at least one target expert network, thereby solving the technical problem of inaccurate selection of the expert network in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a method and device for training an expert model, a storage medium, and an electronic device. BACKGROUND

[0002] As an innovative architecture for large language model training, Mixture of Experts (MoE) has attracted widespread attention in the field of artificial intelligence in recent years. The core idea of MoE model is to assign input data to different expert sub-models, and then combine the outputs of all sub-models to generate the final result, achieving efficient use of computing resources and improvement of model performance. MoE model, through the characteristic of sparse activation, only a small number of experts most relevant to input data are activated for calculation in each forward propagation process, while most experts are in an inactive state. This sparsity significantly reduces the resources required for each calculation, and the computing efficiency is greatly improved compared to traditional dense models.

[0003] However, the existing technology uses fixed expert activation rules, fixed activation expert numbers, and fixed selection of expert networks, resulting in a situation where the selection of experts does not match the actual task requirements in actual application, and the computing resources cannot be effectively utilized. That is, there is an inaccurate selection of expert networks in the prior art. SUMMARY

[0004] The present application provides a method and device for training an expert model, a storage medium, and an electronic device to at least solve the technical problem of inaccurate selection of expert networks in the prior art.

[0005] The present application provides a method for training an expert model, comprising: determining a target expert network group from a plurality of expert network groups included in an initial expert model according to an estimated resource occupation situation of input data, wherein the expert network group includes a plurality of expert networks for processing input data, and the estimated resource occupation situation is used to indicate the computing resources required by the expert networks to process the input data; sequentially taking the plurality of expert networks in the target expert network group as a current expert network, and determining an individual difference evaluation coefficient matched with the current expert network, wherein the individual difference evaluation coefficient is determined according to the respective network similarity between the current expert network and a plurality of reference expert networks in the target expert network group, and the reference expert network is an expert network in the target expert network group other than the current expert network; determining at least one target expert network from the plurality of expert networks according to the respective individual difference evaluation coefficients of the plurality of expert networks in the target expert network group; processing the input data according to the at least one target expert network, and adjusting the model parameters in the initial expert model according to the respective load state description information of the at least one target expert network.

[0006] The application further provides a training device of an expert model, comprising: a first determination unit configured to determine a target expert network group from a plurality of expert network groups included in an initial expert model according to an estimated resource occupation of input data, wherein the expert network group comprises a plurality of expert networks for processing the input data, and the estimated resource occupation is used to indicate the computing resources required by the expert network for processing the input data; a second determination unit configured to determine an individual difference evaluation coefficient matched with a current expert network in turn by taking the plurality of expert networks in the target expert network group as the current expert network, wherein the individual difference evaluation coefficient is determined according to the respective network similarities between the current expert network and a plurality of reference expert networks in the target expert network group, and the reference expert network is an expert network in the target expert network group except the current expert network; a third determination unit configured to determine at least one target expert network from the plurality of expert networks according to the respective individual difference evaluation coefficients of the plurality of expert networks in the target expert network group; and an adjustment training unit configured to process the input data according to the at least one target expert network, and adjust model parameters in the initial expert model according to the respective load state description information of the at least one target expert network.

[0007] The application further provides an electronic device, comprising: a memory configured to store a computer program; and a processor configured to implement the steps of the training method of the expert model when executing the computer program.

[0008] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the training method of the expert model.

[0009] The application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of the training method of the expert model.

[0010] Through the present application, the target expert network group is determined from the multiple expert network groups included in the initial expert model according to the estimated resource occupation of the input data, ensuring that each data request can be allocated to the expert network group that can effectively process the required resources of the request, thereby directly locating the expert with matching processing capacity and demand, improving the accuracy of selection; the multiple expert networks in the target expert network group are sequentially taken as the current expert network, the individual difference evaluation coefficient matched with the current expert network is determined, the structure and parameter differences between the expert networks are considered, the difference between the current network and other networks in the same group can be quantified, the selection accuracy of the expert networks in the group is enhanced, so that when processing the same task, the system tends to select those experts that are most matched with the task, rather than only based on load or random selection; at least one target expert network is determined from the multiple expert networks according to the individual difference evaluation coefficient of each expert network in the target expert network group, so that the task can be processed by the expert with appropriate resource level, thereby maximizing the processing quality and efficiency; the input data is processed according to the at least one target expert network, and the model parameters in the initial expert model are adjusted according to the load state description information corresponding to the at least one target expert network, ensuring that the selection of the expert network is always based on the latest resource state and demand, solving the technical problem of inaccurate selection of the expert network in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 The hardware structure block diagram of the server device of the training method of the expert model provided by the embodiments of the present application is shown in the figure.

[0013] Figure 2 The flowchart of the training method of the expert model provided by the embodiments of the present application is shown in the figure.

[0014] Figure 3 The flowchart of another training method of the expert model provided by the embodiments of the present application is shown in the figure.

[0015] Figure 4 The structure schematic diagram of the training device of the expert model according to the embodiments of the present application is shown in the figure.

[0016] Figure 5 The structure schematic diagram of the electronic device of the training of the expert model according to the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0017] With reference to the drawings and specific embodiments described below, the technical solutions in the embodiments of the present application will be better understood.

[0018] It should be noted that, in the description of the present application, the terms "comprising", "containing" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0019] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0020] The method embodiments provided in the embodiments of the present application can be executed in a server device or similar computing device. Taking the case of running on a server device, Figure 1 is a hardware structure block diagram of a server device of a method for training an expert model according to an embodiment of the present application. As shown in Figure 1 , the server device can include one or more (only one is shown in Figure 1 ) processor 102 (the processor 102 can include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned server device can also include a transmission device 106 for communication function and an input and output device 108, and those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the above-mentioned server device. For example, the server device can also include more or fewer components than those shown in Figure 1 , or have a different configuration from Figure 1 .

[0021] The memory 104 can be used to store computer programs, for example, software programs of application software and modules, such as a computer program corresponding to the training method of the expert model in the embodiments of the present application. The processor 102 performs various functional applications and data processing by running the computer programs stored in the memory 104, that is, implements the above method. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, which can be connected to a server device through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0022] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network can include a wireless network provided by a communication provider of a server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC) which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module which is used to communicate with the Internet in a wireless manner.

[0023] The embodiments of the present application provide a training method of an expert model. The method is described in detail in combination with the execution flow of the training method of the expert model.

[0024] In order to more clearly understand the technical solutions provided by the embodiments of the present application, the key terms related to the embodiments of the present application are introduced first.

[0025] Mixture of Experts (MoE).

[0026] Large Language Models (LLMs).

[0027] Reinforcement Learning (RL).

[0028] Proximal Policy Optimization (PPO).

[0029] Long Short-Term Memory (LSTM).

[0030] Temporal Difference Error (TD-error).

[0031] Convolutional Neural Network (CNN).

[0032] Transformer Model (Transformer).

[0033] Bidirectional Encoder Representations from Transformers (BERT).

[0034] Generative Pre-trained Transformer (GPT).

[0035] As an optional implementation, as shown in Figure 2 The training method of the above expert model comprises:

[0036] S202, determining a target expert network group from a plurality of expert network groups included in an initial expert model according to an estimated resource occupation of input data, wherein the expert network group includes a plurality of expert networks for processing the input data, and the estimated resource occupation is used to indicate the computing resources required by the expert networks to process the input data;

[0037] S204, sequentially taking the plurality of expert networks in the target expert network group as a current expert network, and determining an individual difference evaluation coefficient matched with the current expert network, wherein the individual difference evaluation coefficient is determined according to the respective network similarities between the current expert network and a plurality of reference expert networks in the target expert network group, and the reference expert network is an expert network other than the current expert network in the target expert network group;

[0038] S206, determining at least one target expert network from the plurality of expert networks according to the respective individual difference evaluation coefficients of the plurality of expert networks in the target expert network group;

[0039] S208, processing the input data according to the at least one target expert network, and adjusting the model parameters in the initial expert model according to the respective load state description information of the at least one target expert network.

[0040] As an optional implementation, in step S202, a target expert network group is determined from a plurality of expert network groups included in the initial expert model according to an estimated resource occupation of the input data, wherein the expert network group includes a plurality of expert networks for processing the input data, and the estimated resource occupation is used to indicate the computing resources required by the expert networks to process the input data.

[0041] Optionally, the input data includes but is not limited to text data, code data, multi-modal data, etc., including but not limited to tasks for performing text understanding, software development, media content analysis, etc.; the resource occupation includes but is not limited to CPU usage, memory consumption, GPU occupation, etc., and the initial expert model contains a plurality of hierarchical expert network group models, each group containing a set of expert networks, which are divided into different groups according to processing capacity or resource consumption, such as lightweight group, balanced group, and high-performance group. The best expert network group is selected according to the estimated resource requirement of the input data to process the data.

[0042] In step S204, a plurality of expert networks in the target expert network group are sequentially taken as a current expert network, and an individual difference evaluation coefficient matching the current expert network is determined, wherein the individual difference evaluation coefficient is determined according to the respective network similarity between the current expert network and a plurality of reference expert networks in the target expert network group, and the reference expert network is an expert network in the target expert network group except the current expert network.

[0043] Optionally, for each current expert network, the system calculates the similarity between it and the remaining networks in the group, such as by comparing their core parameter matrices. If the similarity of the current expert network A with expert networks B, C, and D is 0.8, 0.3, and 0.6, respectively, the individual difference evaluation coefficient of A is, for example, obtained by taking the average reciprocal of the similarity. The higher the coefficient, the greater the difference of the network relative to other networks in the group.

[0044] In step S206, at least one target expert network is determined from the plurality of expert networks according to the respective individual difference evaluation coefficients of the plurality of expert networks in the target expert network group.

[0045] Optionally, if the individual difference evaluation coefficients of expert networks A, B, and C are 0.5, 0.7, and 0.4, respectively, expert network B is selected as the "target expert network" because it has the highest coefficient, indicating that B has the highest difference relative to other networks in the group.

[0046] In step S208, the input data is processed according to the at least one target expert network, and the model parameters in the initial expert model are adjusted according to the respective load state description information of the at least one target expert network.

[0047] Optionally, the load state description information is used to describe information of current processing capacity or resource utilization of the expert network, to monitor whether the expert network is overloaded or idle, and to update parameters of the entire expert model according to the load state and processing result of the expert network, so as to optimize future selection and processing.

[0048] As an optional implementation, after selecting at least one target expert network, taking the expert network B as an example, the image data is processed through the target expert network B, and the current load state description information of the target expert network B is checked, such as CPU usage rate, queue depth, etc. Assuming that the CPU usage rate of the target expert network B reaches 90% in a short time during processing, that is, the effect of the target expert network B in processing image data is poor, the system needs to adjust the parameters of the expert model, such as reducing the selection probability of the target expert network B for processing image data in future tasks, or transferring the task to an expert network with lower load when the load of the expert network B exceeds a certain threshold. The learning rate can be set, the optimizer (such as Adam) is used to update the model parameters, and only the network parameters of a specific part can be updated to quickly adapt to the current load state, which is not limited here.

[0049] Through the above-mentioned embodiments of the present application, the target expert network group is determined from the multiple expert network groups included in the initial expert model according to the estimated resource occupation of the input data, ensuring that each data request can be allocated to an expert network group that can effectively process the required resources of the request, thereby directly locating the expert with matching processing capacity and demand, improving the accuracy of selection; the multiple expert networks in the target expert network group are sequentially taken as the current expert network, the individual difference evaluation coefficient matched with the current expert network is determined, the structural and parameter differences between the expert networks are considered, the difference between the current network and other networks in the same group is quantified, the selection accuracy of the expert networks in the group is enhanced, so that when processing the same task, the system tends to select those experts that are most matched with the task, rather than only based on load or random selection; at least one target expert network is determined from the multiple expert networks according to the individual difference evaluation coefficients of the multiple expert networks in the target expert network group, so that the task can be processed by the expert with appropriate resource level, thereby maximizing the processing quality and efficiency; the input data is processed according to the at least one target expert network, and the model parameters in the initial expert model are adjusted according to the load state description information corresponding to the at least one target expert network, so that the selection of the expert network is always based on the latest resource state and demand, solving the technical problem of inaccurate selection of the expert network in the prior art.

[0050] In an alternative embodiment, a plurality of expert networks in the target expert network group are sequentially taken as the current expert network, and an individual difference evaluation coefficient matching the current expert network is determined, comprising:

[0051] S1, respectively calculating network similarities between the current expert network and a plurality of reference expert networks;

[0052] S2, determining an individual difference evaluation coefficient matching the current expert network according to a first average value of the plurality of network similarities, wherein the individual difference evaluation coefficient has a negative correlation with the first average value.

[0053] In the above steps S1-S2, optionally, assuming that the target expert network group is a high-performance expert network group, containing four expert networks: A, B, C, and D, first taking the A network as the "current expert network", analyzing its uniqueness and difference relative to other networks in the group, and then repeating the process with the B, C, and D networks as the "current expert network".

[0054] The network similarity is used to measure the similarity of two expert networks in terms of parameters, structure or function, and is usually obtained by comparing core parameters or using specific similarity indicators. Taking the A network as the current expert network, the system will calculate the similarity between A and other expert networks B, C, and D. The difference between them can be quantified by extracting the core parameter matrix of each network and using similarity indicators (such as cosine similarity, Euclidean distance, etc.), which are not specifically limited here.

[0055] Assuming that the calculated similarities are 0.75 (A and B networks), 0.60 (A and C networks), and 0.85 (A and D networks), respectively. Further, the average value of the similarities between the current expert network and the reference expert networks is calculated to evaluate the degree of differentiation of the current expert network, such as the first average value = (0.75 + 0.60 + 0.85) / 3 = 0.73. The individual difference evaluation coefficient = 1 - first average value. For the A network, the individual difference evaluation coefficient is 1 - 0.73 = 0.27, i.e. if a certain expert network has a lower first average value, its individual difference evaluation coefficient will be higher, indicating that it has higher uniqueness in the group.

[0056] Through the above embodiments, the similarity calculation can intuitively show the difference between the current expert network and other expert networks. If an expert network is highly similar to most experts in the group, its characteristics may not be prominent enough to meet the processing needs of specific input data. Conversely, an expert network with a lower similarity may have more unique processing capabilities and be more suitable for specific types of data. Establishing a relationship between the individual difference evaluation coefficient and the network similarity can make the selection of experts more inclined to those with higher differences, avoiding the blindness of expert network selection. The system can more accurately match the expert network and the input data, reduce processing errors, and improve the accuracy of expert network selection.

[0057] In an optional embodiment, determining at least one target expert network from the plurality of expert networks according to the individual difference evaluation coefficients of the plurality of expert networks in the target expert network group comprises:

[0058] S1, respectively modifying the initial expert selection vectors matched with the plurality of expert networks in the target expert network group according to the individual difference evaluation coefficients of the plurality of expert networks, to obtain a plurality of target expert selection vectors, wherein the initial expert selection vectors are used to determine the matching degree of the expert networks for processing the input data;

[0059] S2, determining at least one target expert network from the plurality of expert networks according to the activation processing results of the plurality of target expert selection vectors.

[0060] In the above step S1, the initial expert selection vector represents the preliminary probability vector directly allocated by the gating network to each expert network for input data, i.e., the probability of each expert being selected before considering the differentiation of expert networks.

[0061] Optionally, the initial expert selection vectors matched with the plurality of reference expert networks are modified according to the individual difference evaluation coefficients to obtain a plurality of target expert selection vectors, for example: target expert selection vector z'=z+weight×individual difference evaluation coefficient, wherein z is the original topk expert selection Logits (i.e., the initial expert selection vector), and then the modified Logits z' can be brought into the subsequent softmax calculation for topk expert selection.

[0062] In the above step S2, the first few expert networks with the highest probability are selected for parallel processing when processing the input. For example, a topk strategy is used to select the first k expert networks with the highest probability.

[0063] As an optional implementation, it is assumed that the target expert network group includes expert networks A, B, C, and D. The initial expert selection vector is ([0.2, 0.3, 0.1, 0.4]), and the individual difference evaluation coefficient is ([0.6, 0.8, 0.5, 0.7]).

[0064] The modified expert selection vector: first, multiply the initial expert selection vector by the individual difference evaluation coefficient to obtain the modified vector ([0.12, 0.24, 0.05, 0.28]), and then normalize it by the softmax function to obtain the target expert selection vector [0.12, 0.24, 0.05, 0.28], denoted as ([p_1, p_2, p_3, p_4]).

[0065] Determine the target expert network: sample or select the top k experts according to the target expert selection vector. Assuming that the top2 strategy is used, the two expert networks with the highest selection probability are selected as the "target expert network". In the example, expert networks B and D become the target.

[0066] Through the above implementation, the original expert selection vector reflects the priority of the expert network after considering its uniqueness by integrating the individual difference evaluation coefficient. The modified target expert selection vector combines the preliminary decision of the gating network and the differentiated evaluation of the expert network, providing more intelligent guidance for task allocation and improving the accuracy of selecting expert networks.

[0067] In an optional implementation, after determining the individual difference evaluation coefficient matching the current expert network, it further includes:

[0068] S1, determine the network similarity between any two expert networks in the target expert network group as the inter-group similarity set;

[0069] S2, determine the second average value according to the network similarity in the inter-group similarity set;

[0070] S3, determine the inter-group difference evaluation coefficient matching the target expert network group according to the second average value.

[0071] In the above steps S1-S3, optionally, the above inter-group similarity set is a set of similarities between all possible pairs of expert networks in the target expert network group. Assuming that the target expert network group consists of four expert networks A, B, C, and D, the similarities between the six pairs of expert networks (AB), (AC), (AD), (BC), (BD), and (CD) need to be calculated, and these similarity values are collected into a set, i.e., the inter-group similarity set.

[0072] After obtaining the set of inter-group similarities, the average of all similarity values in the set is calculated as the "second average value". For the set of inter-group similarities {0.7, 0.4, 0.6, 0.5, 0.8, 0.3}, the second average value = (0.7 + 0.4 + 0.6 + 0.5 + 0.8 + 0.3) / 6 = 0.55.

[0073] The inter-group difference evaluation coefficient matching the target expert network group is determined according to the second average value, for example, inter-group difference evaluation coefficient = 1 - second average value.

[0074] An ideal difference level can also be set, for example, 0.4 (i.e., the average inter-group similarity is expected to be lower than 0.4 to improve the difference), and the inter-group difference evaluation coefficient is calculated based on the difference between the ideal difference level and the second average value. If the second average value is higher than the ideal difference level, it indicates that the difference between the expert networks is insufficient; if it is lower, it indicates that the difference is good, for example, inter-group difference evaluation coefficient = 1 - (second average value - ideal difference value), which is not specifically limited here.

[0075] By calculating the second average value of the set of inter-group similarities, the network difference within the target expert network group can be evaluated. Then, the inter-group difference evaluation coefficient is used to quantify the difference level of the entire expert network group. In practical applications, the inter-group difference evaluation coefficient can be further integrated into the learning goal of the policy network to improve the overall performance of the expert network group and ensure that tasks can be more efficiently and accurately allocated to the most suitable expert network for processing.

[0076] In an optional implementation, determining at least one target expert network from the plurality of expert networks further comprises:

[0077] S1, obtaining a first decision factor according to the product of the individual difference evaluation coefficient matching the expert network and a first weight value;

[0078] S2, obtaining a second decision factor according to the product of the inter-group difference evaluation coefficient and a second weight value;

[0079] S3, correcting an initial expert selection vector matching the corresponding expert network according to the first decision factor and the second decision factor to obtain a plurality of target expert selection vectors, wherein the initial expert selection vector is used to determine the matching degree of the expert network for processing the input data;

[0080] S4, determining at least one target expert network according to the activation processing result of the target expert selection vector.

[0081] In the above step S1, the first decision factor is obtained according to the product of the individual difference evaluation coefficient matching the expert network and the corresponding first weight value. Optionally, the first decision factor = individual difference evaluation coefficient * first weight value. , is an individual difference evaluation coefficient, is a first weight value.

[0082] In the step S2, the second decision factor is obtained according to the product of the inter-group difference evaluation coefficient and the corresponding second weight value, and the second decision factor is optionally , is an individual difference evaluation coefficient, is a second weight value.

[0083] In the steps S3-S4, the initial expert selection vector matched with the corresponding expert network is corrected according to the first decision factor and the second decision factor, to obtain a plurality of target expert selection vectors, wherein the initial expert selection vector is used to determine the matching degree of the expert network for processing the input data; and at least one target expert network is determined according to the activation processing result of the target expert selection vector.

[0084] Optionally, The above combined reward item is injected into the topk routing decision, such as z'=z+αb, wherein z is the original topk expert selection Logits, and a is a weight parameter. The corrected Logits z' is brought into the subsequent softmax calculation to perform topk expert selection.

[0085] According to the first decision factor (focusing on the difference of individual experts in the group) and the second decision factor (focusing on the difference between groups), the initial expert selection vector is modified, and the probability that each reference expert network is selected to process a specific input data is adjusted. The initial expert selection vector reflects the preliminary matching degree of each expert network based on the input data, and the corrected target expert selection vector is a further optimization of the preliminary matching degree to reflect a more comprehensive and accurate expert network selection tendency, thereby improving the specific matching degree of the expert network and the input data and avoiding the selection of an overly general expert network with poor processing capability.

[0086] In an optional embodiment, the target expert network group is determined from a plurality of expert network groups included in the initial expert model according to the estimated resource occupation of the input data, and the method further comprises:

[0087] S1, determining the estimated resource occupation of the input data according to the complexity information of the input data, wherein the complexity information is used to indicate the difficulty of processing the input data;

[0088] S2, determining the selection probability matched with the plurality of expert network groups according to the estimated resource occupation;

[0089] S3, determining the expert network group corresponding to the selection probability satisfying the target probability condition as the target expert network group.

[0090] In step S1, the estimated resource occupation of the input data is determined according to the complexity information of the input data, wherein the complexity information is used to indicate the difficulty of processing the input data.

[0091] Optionally, the complexity information includes features of the input data, such as vocabulary richness, sentence length, structural complexity, image resolution, pixel density, color complexity, etc., to measure the difficulty of processing the data.

[0092] For example, for text data, the analysis determines that the complexity information is 150 words, the word frequency information indicates that the proportion of professional terms is high, and the grammar structure is complex. According to the complexity information, the estimated resource occupation is medium to high.

[0093] In step S2, the selection probability of each expert network group is determined according to the estimated resource occupation. In the MoE system, there are multiple expert network groups such as lightweight group (processing simple tasks), balanced group (processing medium complexity tasks), and high performance group (processing complex tasks). According to the estimated resource occupation obtained in step S1, the probability of each expert network group being selected is determined. For example, for medium to high resource occupation, the selection probability of the balanced group and the high performance group may be higher, and the selection probability of the lightweight group is lower.

[0094] As an optional implementation, the selection probabilities of the three expert network groups (lightweight group, balanced group, and high performance group) are set to (p_L), (p_M), and (p_H), respectively. For medium to high estimated resource occupation, (p_L=0.1), (p_M=0.4), and (p_H=0.5) are set, i.e., the high performance group is most likely to be selected to process the data, followed by the balanced group, and the selection probability of the lightweight group is the lowest.

[0095] In step S3, the expert network group corresponding to the selection probability that meets the target probability condition is determined as the target expert network group. Optionally, the target probability condition can be to set a probability threshold, or to select the N groups with the highest selection probability. If the target probability condition is "the group with the highest selection probability", the high performance group (p_H=0.5) will be determined as the only "target expert network group". If the target condition is "the group with a selection probability greater than 0.3", both the high performance group and the balanced group will be confirmed as the target expert network group.

[0096] Through the above steps, the MoE system can intelligently determine the target expert network group based on the estimated resource occupation and complexity information of the input data, ensuring that the finally selected expert network group not only meets the resource demand, but also considers the current running state of the system, avoiding the situation of overload or resource idling.

[0097] In an optional embodiment, the expert network group corresponding to the selection probability satisfying the target probability condition is determined as the target expert network group, and the method further comprises at least one of the following:

[0098] In a first mode, the maximum selection probability among the selection probabilities respectively matched with the plurality of expert network groups is determined, and the expert network group corresponding to the maximum selection probability is determined as the target expert network group.

[0099] As an optional embodiment, after the selection probability of each expert network group is calculated, the expert network group with the maximum selection probability value is selected as the target expert network group. For example, the selection probabilities of the expert network groups calculated are 0.2 for the lightweight group, 0.6 for the balanced group, and 0.3 for the high-performance group. The balanced group has the maximum selection probability of 0.6 and becomes the target expert network group.

[0100] In a second mode, the queue length of the task waiting queue respectively matched with the plurality of expert network groups is determined, and the expert network group corresponding to the selection probability greater than the first probability threshold value and the queue length less than the second quantity threshold value is determined as the target expert network group.

[0101] As an optional embodiment, the target expert network group is determined based on the selection probability of the expert network group and the current task waiting queue length of the expert network group. For example, the system sets the first probability threshold value as 0.4 and the second quantity threshold value as 100 requests. Assuming that the selection probability and the queue length of the three expert network groups are as follows: the lightweight group has a selection probability of 0.3 and a queue length of 150; the balanced group has a selection probability of 0.5 and a queue length of 80; and the high-performance group has a selection probability of 0.6 and a queue length of 120. According to the set threshold values, the selection probabilities of the balanced group and the high-performance group are higher than 0.4 (the first probability threshold value), and the queue length of the balanced group is lower than the second quantity threshold value of 100. Therefore, the balanced group is determined as the target expert network group.

[0102] In a third mode, the average resource utilization respectively matched with the plurality of expert network groups is determined, and the expert network group corresponding to the selection probability greater than the second probability threshold value and the average resource utilization greater than the third resource utilization threshold value is determined as the target expert network group.

[0103] As an optional embodiment, the selection probability of the expert network group and the average resource utilization are considered simultaneously to avoid selecting a network group that is too busy. For example, the second probability threshold value is set as 0.4 and the third resource utilization threshold value is set as 70%. The resource utilization and the selection probability of the three expert network groups are as follows: the lightweight group has a selection probability of 0.2 and an average resource utilization of 75%; the balanced group has a selection probability of 0.5 and an average resource utilization of 65%; and the high-performance group has a selection probability of 0.6 and an average resource utilization of 80%.

[0104] Only the high-performance group and the balanced group have a selection probability greater than 0.4. Considering the average resource utilization, the resource utilization of the high-performance group is higher than that of the balanced group, but because the utilization of the high-performance group is also greater than 70%, it may mean that it is approaching its processing limit. The balanced group not only has a selection probability that meets the demand, but also has a resource utilization lower than the threshold, indicating that it has the capacity to handle more tasks, and therefore, the balanced group becomes the target expert network group.

[0105] Method four, determine the historical processing data type matched with each of the plurality of expert network groups, and determine the expert network group with a selection probability greater than the third probability threshold and the same historical processing data type as the data type of the input data as the target expert network group.

[0106] As an optional implementation, assuming that the third probability threshold is 0.4, the current input data is of the mathematical modeling type, and the expert network groups with historical processing data types matching the input data type are: the lightweight group: selection probability 0.2, historical processing data type is text summary; the balanced group: selection probability 0.5, historical processing data type is mathematical modeling; the high-performance group: selection probability 0.6, historical processing data type is code generation. The selection probability of the balanced group and the high-performance group is higher than 0.4, but only the historical processing data type of the balanced group is consistent with the input data type (both are mathematical modeling), so the balanced group is selected as the "target expert network group".

[0107] Through the above-mentioned embodiments described in the present application, multiple decision-making bases (queue length, resource utilization, data type) are introduced to optimize the selection process, ensuring that the system not only selects according to the estimated demand of the data, but also considers the real-time state and professional field of the expert network group, to achieve more efficient and accurate task allocation. By comprehensively considering the selection probability and various real-time states or historical data types, the system can more intelligently determine the target expert network group, effectively avoiding the bottleneck problem caused by excessive concentration of resources in a certain group, while ensuring that the data processing task can be efficiently completed by the most matched expert network group.

[0108] In an optional implementation, before determining the estimated resource occupation of the input data according to the complexity information of the input data, the method further comprises:

[0109] S1, performing vectorization processing on the input data to obtain a vector sequence corresponding to the input data;

[0110] S2, determining semantic density information of the input data according to the frequency of each word vector appearing in the vector sequence;

[0111] S3, determining the complexity information of the input data according to the semantic density information.

[0112] In the step S1, the input data is vectorized to obtain a vector sequence corresponding to the input data. Optionally, each word or sentence fragment in the input data is converted into a word vector using a pre-trained word embedding model. Assuming that the input data is an English text, the text is first segmented into a series of words and punctuation marks, and then a pre-trained embedding model is used to obtain a vector representation of each word. Finally, these word vectors are arranged into a vector sequence in the order of the original text.

[0113] In the step S2, the semantic density information of the input data is determined according to the frequency of each word vector appearing in the vector sequence. Optionally, the semantic density of the text can be analyzed by counting the number of occurrences or frequency of each word vector, and the semantic density information reflects the intensity of information carried in the text, which is usually measured by information entropy (Entropy) or TF-IDF (Term Frequency-Inverse Document Frequency). TF-IDF combines the information of term frequency and inverse document frequency, so that words that frequently appear in a document but rarely appear in the entire document set have a higher weight. Assuming that the vector sequence formed in the foregoing example, the information entropy of all word vectors is calculated to obtain the semantic density information.

[0114] In the step S3, the complexity information of the input data is determined according to the semantic density information. Optionally, if the information entropy is high, it is considered that the input data has high complexity; otherwise, if the information entropy is low, it is considered that the input data has low complexity.

[0115] Through the above embodiments described in the present application, the complexity information of the input data can be accurately determined by gradually processing the original text of the input data through word embedding, information entropy calculation and complexity level mapping, ensuring that data with high complexity can be assigned to an expert network group that is more suitable for processing complex tasks, thereby improving the utilization efficiency of the entire system resources and the accuracy of task processing.

[0116] In an optional embodiment, determining the complexity information of the input data further comprises:

[0117] S1, determining the nesting depth information of the input data according to the number of nested statements in the input data, wherein the nested statements contain multiple layers of text structure;

[0118] S2, determining the syntax depth information of the input data according to the number of target statements in the input data, wherein the text length of the target statement is greater than a target threshold;

[0119] S3, determining the semantic coherence information of the input data according to the number of logical conjunction words in the input data;

[0120] S4, determining complexity information according to semantic density information, nesting depth information, syntax depth information and semantic coherence information.

[0121] In step S1, nesting depth information of the input data is determined according to the number of nested statements in the input data, wherein the nested statements contain multiple layers of text structures.

[0122] Optionally, the input text data is analyzed to count the number of layers of nested statements, for example, if a sentence contains multiple parentheses or quotation marks nesting, the maximum nesting level is counted. If the input data is programming code, the nesting depth of code blocks, such as the number of function, loop or conditional statement nesting, is counted.

[0123] In step S2, syntax depth information of the input data is determined according to the number of target statements in the input data, wherein the text length of the target statements is greater than a target threshold.

[0124] Optionally, all target statements meeting the condition are identified and counted, and then the syntax depth is measured based on the number of target statements. For example, if the threshold is set to 20 words, the number of statements in the text that exceed 20 words is counted to determine the syntax depth information.

[0125] In step S3, semantic coherence information of the input data is determined according to the number of logical conjunctions in the input data.

[0126] Optionally, semantic coherence information is determined by analyzing the number and distribution of logical conjunctions (such as and, but, if, because, etc.) in the text. The more logical conjunctions, the stronger the logical association of the text, and the higher the semantic coherence.

[0127] In step S4, complexity information is determined according to semantic density information, nesting depth information, syntax depth information and semantic coherence information.

[0128] Optionally, a weight can be set for each indicator, and then the weighted values are summarized to determine the complexity information. Assuming that the weight of semantic density information is 0.4, the weight of nesting depth information is 0.2, the weight of syntax depth information is 0.2, and the weight of semantic coherence information is 0.2, the values obtained are 0.7 (semantic density), 1 (nesting depth), 0.6 (syntax depth), and 0.8 (semantic coherence), respectively.

[0129] Then the complexity information of the input data is calculated as follows: complexity information = 0.4 x 0.7 + 0.2 x 1 + 0.2 x 0.6 + 0.2 x 0.8 = 0.28 + 0.2 + 0.12 + 0.16 = 0.76.

[0130] Through the above-mentioned embodiments described in the present application, comprehensive consideration of information such as semantic density, nesting depth, grammatical depth, and semantic coherence can more comprehensively evaluate the complexity of input data, better understand and process text content, and provide more refined basis for subsequent resource allocation, ensuring that the system can reasonably schedule resources according to the complexity of the input data to achieve the best processing effect.

[0131] In an optional embodiment, before adjusting the model parameters in the initial expert model, the following steps are included:

[0132] S1, determine the utilization information of each target expert network processing input data, wherein the utilization information is used to indicate the activity level of the computing resources when the target expert network processes the input data;

[0133] S2, determine the task backlog level of each target expert network;

[0134] S3, determine the task discard rate information matched with the target expert network group;

[0135] S4, determine the load state description information according to the utilization information of each target expert network, the task backlog level, and the task discard rate information matched with the target expert network group.

[0136] In the above step S1, the utilization information of each target expert network processing input data is determined, wherein the utilization information is used to indicate the activity level of the computing resources when the target expert network processes the input data.

[0137] Optionally, the time utilization, computing unit usage, or memory usage of the target expert network when processing the input data is monitored to calculate the utilization information, for example, an expert network uses an average of 70% of the computing resources to process data in a unit of time, and the utilization information is 70%. Assuming that target expert networks A, B, and C are processing input data, the utilization information of network A is 60%, network B is 80%, and network C is 40% obtained by monitoring.

[0138] In the above step S2, the task backlog level of each target expert network is determined.

[0139] Optionally, the task backlog degree can be obtained by calculating the ratio of the number of unprocessed tasks in each target expert network task queue to the processing capacity. For example, the task queue of expert network A has 1200 tasks, and the processing capacity is 200 tasks per minute, so the backlog degree is 6; the task queue of expert network B has 800 tasks, and the processing capacity is 150 tasks per minute, so the backlog degree is 5.33; the task queue of expert network C has 300 tasks, and the processing capacity is 50 tasks per minute, so the backlog degree is 6.

[0140] In the above step S3, the task discard rate information matched with the target expert network group is determined. The above task discard rate information can be the proportion of discarded tasks that each expert network fails to successfully process in the past period of time.

[0141] In the above step S4, the load state description information is determined according to the utilization rate information of each target expert network, the task backlog degree, and the task discard rate information matched with the target expert network group. For example, the load state description information can be quantitatively obtained by splicing the above information or weighted sum of the above information, without specific limitation here.

[0142] As an optional embodiment, the utilization rate information in step S1 is the expert utilization rate, which quantifies the busy degree of expert computing resources and reflects the bottleneck of expert processing capacity, and the calculation formula is as follows:

[0143] , wherein is the sampling period (obtained by fixed window), is the activity level of the expert in the sampling period.

[0144] The task backlog degree in step S2 can be determined according to the queue depth, that is, the queue depth reflects the backlog degree of the tasks to be processed by the expert and predicts the potential delay, and the calculation formula is as follows:

[0145] , wherein represents the number of requests arrived in the sampling period, represents the number of processed requests in the sampling period.

[0146] The task discard rate information in the above step S3 is the discard rate, which measures the severity of insufficient service capacity of the expert group. The higher the value, the more likely the current expert group is overloaded, and the proportion of tasks allocated to it should be reduced, and the calculation formula is as follows:

[0147] , wherein , g represents the target expert group (which can be a lightweight group, a balanced group, or a high-performance group), and t is the number of expert groups. The load description information is obtained by comprehensively considering the above information.

[0148] By analyzing the utilization information, task backlog level and task discard rate of the target expert network, detailed load state description information can be obtained, which provides intuitive data support for subsequent model parameter adjustment, enabling the system to more intelligently cope with resource allocation and performance optimization, ensuring that the expert network group under the MoE architecture can efficiently and stably process various input data.

[0149] In an optional implementation, the model parameters in the initial expert model are adjusted according to the respective load state description information of each target expert network, including:

[0150] S1, the complexity information, utilization information, task backlog level, task discard rate information and inter-group difference evaluation coefficient of the input data are vectorized and spliced to obtain a state vector, wherein the inter-group difference evaluation coefficient is determined according to the network similarity between any two expert networks in the target expert network group;

[0151] S2, based on the state vector of the decision network in the initial expert model, the model parameters are adjusted.

[0152] In the above steps S1-S2, assuming that the complexity information of the target expert network A is 0.76, the utilization information is 0.6, the backlog level is 6.0, and the discard rate information is 0.05; the inter-group difference evaluation coefficient is 0.85. Then the state vector (S_A) is (0.76, 0.6, 6.0, 0.05, 0.85).

[0153] Using the state vector (S) as input, a policy network (such as Actor-Critic architecture) in reinforcement learning is used to adjust the model parameters of the decision network. The policy network calculates the best decision or parameter adjustment scheme according to the given state vector (S) to optimize the resource allocation and processing efficiency of the expert network.

[0154] Specifically, the state vector generated in step S1 is input into the Actor network in the decision network. The Actor network calculates a set of actions or parameter adjustment probability distributions based on the state vector, and selects the most appropriate parameter adjustment strategy. The Critic network estimates the expected value or reward after taking the action based on the state vector (S), which is used to evaluate the rationality of the parameter adjustment scheme. According to the action selected by the Actor network and the value estimate provided by the Critic network, the parameters of the decision network are adjusted to optimize the resource allocation strategy of the network.

[0155] An optional implementation is described as follows: the state vectors of the target expert networks A, B, and C are (0.76, 0.6, 6.0, 0.05, 0.85), (0.68, 0.8, 4.5, 0.03, 0.82), and (0.54, 0.4, 7.2, 0.10, 0.75) respectively. The state vector of each expert network is input into the Actor network, and the state vector is provided to the Critic network as the basis for value estimation.

[0156] Action selection and value estimation: the Actor network calculates a parameter adjustment probability distribution based on the state vector, for example, increasing the processing capacity of expert network A by 20%, and the Critic network estimates the expected value after taking this action as -0.02 (a smaller negative value indicates a decrease in expected rewards); for expert network B, the Actor network suggests reducing its processing capacity by 10%, and the Critic network estimates the expected value as 0.04 (a positive value indicates an increase in expected rewards).

[0157] Parameter adjustment: based on the value estimation and action selection strategy, the parameters of the decision network are adjusted, for example, by increasing the model complexity or adjusting the network architecture of expert network A to improve its processing capacity; at the same time, by reducing the model complexity of expert network B, the overload situation is alleviated. Specifically, the network parameters can be adjusted using an optimizer (such as Adam) to make predictions of future actions more accurate, while optimizing the load distribution and resource utilization of the expert network.

[0158] By vectorizing and concatenating the complexity of input data, the utilization of expert networks, the degree of task backlog, the task discard rate, and the inter-group difference evaluation coefficient into a state vector, a comprehensive and intuitive description of the environment state is provided for the decision network; based on the state vector, the policy network and the value network of reinforcement learning can intelligently adjust the parameters of the expert model, optimize the allocation of network resources, and improve the accuracy of expert network selection.

[0159] In an optional implementation, after adjusting the model parameters in the initial expert model according to the respective load state description information of at least one target expert network, the method comprises:

[0160] S1, obtaining actual evaluation information of at least one expert network processing input data at the current time step, wherein the evaluation information is used to indicate the satisfaction of the execution behavior;

[0161] S2, determining predicted evaluation information of processing the input data at the next time step based on a reward prediction function;

[0162] S3, calculating a loss value according to the actual evaluation information and the predicted evaluation information;

[0163] S4, repeat the above loss value calculation process until the minimum loss condition is met.

[0164] In the above step S1, the actual evaluation information can be the actual value (ActualValue) in the TD-error, reflecting the true reward or satisfaction evaluation of the expert network after processing the input data at the current time step.

[0165] Optionally, at the end of the current time step, an actual reward value is calculated according to the quality, speed and resource consumption of the expert network processing the input data. For example, the expert network A processes the data with an accuracy of 95%, a processing time of 2 seconds, and a moderate resource consumption, so the actual evaluation information (reward) can be +10.

[0166] In the above step S2, the predicted evaluation information can be the predicted value (PredictedValue) in the TD-error, which is the reward that the Critic network predicts the agent can obtain in the future time step based on the current state. Optionally, through the Critic network, the reward that the agent (expert network) can obtain when processing the same type of input data in the next time step is predicted based on the current state (including the load state description information). For example, the Critic network predicts that the predicted evaluation information (reward) of the expert network A when processing similar data in the next time step is +9 based on the current state.

[0167] In the above step S3, the TD-error (time difference error) between the actual evaluation information and the predicted evaluation information can be calculated to measure the accuracy of the prediction. The error is taken as the loss value to drive the update of the network parameters. For example, if the actual evaluation information of the expert network A is +10 and the predicted evaluation information is +9, then the TD-error is (10-9=+1). The loss value calculation can be the square of the TD-error to avoid the positive and negative errors canceling each other out.

[0168] In the above step S4, the above steps are iteratively performed to update the parameters of the Actor and Critic networks until the TD-error (loss value) reaches the minimum loss condition, indicating that the predicted evaluation information is very close to the actual evaluation information. For example, after several iterations, the TD-error (loss value) stabilizes at about 0.1, which meets the minimum loss condition, and it can be considered that the parameters of the expert model have been adjusted to a relatively ideal configuration.

[0169] By quantifying the deviation between the actual execution result and the predicted result, and driving the parameter optimization process of the expert network, at the end of each time step, by calculating the TD-error between the actual evaluation information and the predicted evaluation information, not only the accuracy of the prediction can be evaluated, but also the learning of the parameters can be guided, so that the expert network can gradually optimize its ability to process specific types of input data until the minimum loss condition is reached, achieving efficient, fast and resource reasonable input data processing.

[0170] The following complete embodiment illustrates the present application as shown in Figure 3

[0171] S302, input data, including preprocessing and feature extraction; specifically, the user inputs the original data (text / code / multimodal) into the tokenizer for vectorization. The semantic density and structural complexity of the vectorized feature vector are calculated.

[0172] The semantic density is calculated using information entropy. Information entropy H(x) is used to quantify the information density of random variable X: , where is the probability of the input sequence, and n is the size of the input vocabulary.

[0173] The structural complexity can be estimated by the logical connection word density, punctuation nesting depth, and long sentence penalty of the text data according to the actual scene. For image data of multiple modalities, the structural complexity can also be represented by object occlusion and dependency hierarchy.

[0174] S304, dynamic complexity evaluator. Dynamic hierarchical reinforcement learning, specifically, including policy network parameter initialization: using the proximal policy optimization (PPO) algorithm, an Actor-Critic dual network structure is constructed.

[0175] Actor network: input features, output hierarchical classification of input features, network structure can be 3-layer LSTM (Long Short-Term Memory), a special type of recurrent neural network (RNN), the number of LSTM units in each layer can be customized according to the actual task complexity and data size, the hidden layer uses Tanh (hyperbolic tangent) as the activation function, the output content is specifically the probability distribution of the input in each expert group, the network structure can be configured according to actual needs. Here, the expert group is divided into S304-1, S306-1 for simple tasks, lightweight expert group, S304-2, S306-2 for medium tasks, balanced expert group, S304-3, S306-3 for complex tasks, high-performance expert group, respectively processing tasks of different difficulty.

[0176] ​Critic network: input state vector, output state value estimate, network structure can be 3 layers of full connection, hidden layer dimension can be customized to adapt to data size and task complexity, hidden layer uses "rectified linear unit (ReLU) as activation function.

[0177] Initialize the experience replay buffer and use the Prioritized Experience Replay mechanism to adjust the sampling weight according to the TD-error (time difference error).

[0178] TD-error (time difference error) is a measure used in reinforcement learning to measure the difference between predicted value and actual value, used to update the estimate of advantage function or action value function. In reinforcement learning, the agent learns how to choose actions to maximize cumulative rewards by interacting with the environment. The advantage function estimates the expected cumulative reward that the agent can obtain in a certain state, and the TD-error is used to measure the accuracy of this estimate and update it accordingly. The result of each time step interaction is recorded as an experience tuple for storage, usually represented as (s, a, R, s'), the current state (s), the action taken (a), the reward obtained (R), and the next state (s'), i.e. the state of the environment after executing the current action a.

[0179] For example, the advantage function corresponding to the above (s, a, R, s') is: A = R + γV(s') - V(s), where V(s) is the initial state value, γV(s') is the estimated final state value, and R is the cumulative reward observed during the above-mentioned intermediate behavior process. It can be seen that it only measures the degree of error of the evaluator in a given state s, if a higher cumulative reward (R is larger), the advantage estimate will be positive, then it is more likely to take action in this state. Conversely, if a higher reward is expected but a smaller reward is obtained, the advantage estimate will be negative, which will reduce the likelihood of taking action in this step.

[0180] After completing the feature extraction of the input data, the features are spliced with the calculated semantic density, structural complexity, and expert load feedback into a state vector, which is input into the reinforcement learning strategy network (such as the PPO algorithm driven Actor network), and the probability distribution of token classification is output. According to the action probability distribution, the input data is assigned to the target expert group cluster. Record the change of expert load and calculate the immediate reward.

[0181] Use the Experience Replay mechanism to store state-action-reward samples; periodically update the strategy network parameters to minimize the prediction error of the value function (Critic network) while maximizing cumulative rewards.

[0182] For example, first calculate the Actor loss from the experience replay buffer sampling batch data, the formula is as follows:

[0183] wherein, , is the probability distribution of a under state S, which measures the possibility ratio of the updated policy to the previous policy for performing the operation, that is, the difference in the possibility of the same behavior before and after the policy execution (Q(s, a) is the possibility of the policy execution under the historical condition). A is the advantage function, and ε is the clipping range, which can be defined as 0.2. In order to avoid excessive fluctuations in the policy, the above E represents the mean square error loss function.

[0184] The formula for calculating the Critic loss is as follows: The above E represents the mean square error loss function.

[0185] Jointly optimize the Actor and Critic losses, and the formula is as follows:

[0186] wherein, c1 and c2 are hyperparameters, respectively measuring the importance of policy evaluation and exploration accuracy, and H is an entropy function to encourage exploration, to encourage exploration of the entropy of the policy output distribution. The standard entropy Finally, use the Adam optimizer to update the network parameters.

[0187] S308, top K routing in the group. The differentiated routing in the expert group is used to solve the problem of homogeneous experts in traditional MoE routing, and the similarity of experts in the group is dynamically punished to force the experts to learn differentiated features. That is, a differentiated reward signal is injected into the decision-making of the experts in the group, so that similar experts are at a disadvantage in competition. The input is m experts in the expert group G and the input features allocated to this group; the output is the selection probability of the topk experts and the similarity matrix of the experts in the current group. Different heterogeneous experts can be designed in different groups, including lightweight groups for handling simple tasks: using lightweight small networks (micro CNN or Transformer, etc.), depth separable convolution, etc. to reduce the calculation cost and achieve high inference speed; balanced expert groups for handling medium tasks: setting general networks (CNN + Transformer hybrid structure, etc., such as BERT, etc.), combined with residual connections, etc., to balance efficiency and expression ability; high-performance expert groups for handling complex tasks: can use deeper model structures (enhanced Transformer, etc., such as GPT models), support long context, and ensure the accuracy of inference.

[0188] The specific operation is: first, calculate the original expert Logits: z, and then construct an m x m expert similarity matrix as follows:

[0189] where, For each expert extract its core parameter matrix and concatenate to get, i, j represent different expert networks, the operator in the denominator means the L2 norm of the vector. Use the above matrix to calculate the differentiated reward factor:

[0190] Expert-level reward (punish high-similarity individuals): .

[0191] Group-level reward (promote overall diversity): .

[0192] Combined reward term: where β is the balance weight parameter, m is the number of experts in the group, is the expert similarity index, this index will also be fed back to the reinforcement learning driven dynamic hierarchical guidance reinforcement learning training.

[0193] Inject the above combined reward term into the topk routing decision: z' = z + ab. Where z is the original topk expert selection logits, a is the weight parameter. The modified logits z' is brought into the subsequent softmax calculation to select topk experts.

[0194] S310, expert computing and gradient updating. In the selected expert group, the MOE model performs the regular expert computing process: according to the input data, the topk experts are selected by routing decision, the topk selected experts are calculated in parallel, the weighted fusion experts are calculated, and the corresponding reverse gradient is updated.

[0195] S312, load balancing monitor, used for load monitoring and dynamic feedback. Real-time perception of expert group load state, dynamic adjustment of routing strategy, to avoid overload collapse.

[0196] The expert level mainly focuses on the utilization rate of a single expert and the queue depth; the group level mainly focuses on the discard rate of the expert group. Including the calculation of the following index data: expert utilization rate: quantifies the busy degree of expert computing resources, reflects the bottleneck of expert processing capacity, the calculation formula is as follows:

[0197] where is the sampling period (obtained by fixed window), is the activity level of the expert in the sampling period.

[0198] Queue depth: reflects the backlog of expert pending tasks, predicts potential delay, calculation formula as follows:

[0199] where Indicates the number of requests arriving during the sampling period. Indicates the number of requests processed during the sampling period.

[0200] Abandonment rate: This measures the severity of the expert group's insufficient service capacity. A higher value indicates that the current expert group is more likely to be overloaded, and the proportion of tasks assigned to it should be reduced. The calculation formula is as follows:

[0201] ,in , g represents the target expert group (lightweight group, balanced group, high performance group), and t is the number of expert groups.

[0202] Then, the expert load feedback indicators and the expert similarity indicators can be After normalization, the historical load feedback vector is concatenated and passed to the complexity evaluator driven by reinforcement learning, such as the input features obtained by calculating semantic density and structural complexity. , similarity index with expert load feedback and expert Splice into a state vector to form the final state vector, such as: .

[0203] The above state vector is input into the Actor network, and the action probability distribution π(a|s) is output. After softmax sampling, the final action index or weight distribution is obtained.

[0204] Based on action a, the dynamic complexity estimator distributes task input data to different levels of classification, and then to different groups of experts for processing. If the target expert queue is full, the task is downgraded to the next-best expert group. After task processing is completed, the execution results, i.e., the historical load feedback obtained above, are recorded. Other metrics, such as accuracy, can also be incorporated into the reward function as reward and penalty criteria to adjust agent behavior.

[0205] S314, reinforcement learning policy update, can use the proximal policy optimization (PPO) algorithm. Further return to optimize S304, dynamic complex evaluator.

[0206] Specifically, the Proximal Policy Optimization (PPO) algorithm is a gradient-based algorithm for policy optimization. It consists of an actor (actor) and a critic (critic). The actor outputs the probability distribution of the next action at time t, while the critic estimates the expected cumulative reward for that state. The algorithm aims to learn a policy that maximizes the cumulative reward gained based on experience during training.

[0207] It should be noted that in the process of interaction between the reinforcement learning agent and the environment, the interaction result of each time step is recorded as an experience tuple for storage, which is usually represented as (s, a, R, s'), the current state (s), the action taken (a), the reward obtained (R), and the next state (s'), i.e. the state of the environment after executing the current action a.

[0208] The reward function calculation mainly includes a historical load reward item and an intra-group difference reward item:

[0209] The historical load item is: .

[0210] The inter-group difference reward item is: .

[0211] A model performance reward item can also be added: , (where is a weight coefficient used to adjust the proportion of each index; is the model accuracy obtained by calculating the cross-entropy loss function in the expert calculation and gradient update steps).

[0212] The final reward function is .

[0213] Dynamic deployment, deploy the above algorithm to the real scene, monitor the expert load fluctuation index for real-time fine-tuning training. Strategy fine-tuning can be set: monitor the expert load fluctuation index, if the intra-group queue length variance exceeds the threshold or the number of consecutive overload of a single expert is greater than or equal to 3, online fine-tuning is triggered: freeze the Critic network, only update the Actor network parameters (reduce the learning rate), continue for a given number of training steps, and readjust the hierarchical division of the input data by the reinforcement learning module.

[0214] In summary, the present application aims at a dynamic complexity clustering mechanism for input data: based on reinforcement learning, the input task is dynamically divided into layers, and the input data is divided into simple, medium and complex tasks according to the feature index and load condition, and different levels of computing resources are allocated in the downstream.

[0215] Load-triggered elastic degradation system: build a load-aware mechanism to quantify the computing pressure, resource occupation and task processing efficiency of expert nodes; when the load of the target expert group exceeds the threshold, some tasks are dynamically degraded to the low-level expert group.

[0216] Heterogeneous expert group setting: set up heterogeneous expert groups of different structures to process tasks of different difficulties to reduce training cost.

[0217] Multi-objective optimization design: Differentiate the group experts as the target, add the combination reward of the group expert difference to the expert group calculation, guide the selection of topk experts; and combine the input task complexity characteristics, the load situation of the expert group network (expert utilization rate, queue depth, discard rate, etc.) to design the multi-objective reward function of reinforcement learning.

[0218] According to another aspect of the embodiments of the present application, a training device for implementing the training method of the expert model is also provided. As shown in the figure, the device comprises: Figure 4

[0219] The first determination unit 402 determines the target expert network group from the plurality of expert network groups included in the initial expert model according to the estimated resource occupation of the input data, wherein the expert network group comprises a plurality of expert networks for processing the input data, and the estimated resource occupation is used to indicate the computing resources required by the expert network to process the input data;

[0220] The second determination unit 404 sequentially takes the plurality of expert networks in the target expert network group as the current expert network, and determines the individual difference evaluation coefficient matched with the current expert network, wherein the individual difference evaluation coefficient is determined according to the respective network similarity between the current expert network and the plurality of reference expert networks in the target expert network group, and the reference expert network is the expert network in the target expert network group except the current expert network;

[0221] The third determination unit 406 determines at least one target expert network from the plurality of expert networks according to the respective individual difference evaluation coefficient of the plurality of expert networks in the target expert network group;

[0222] The training unit 408 adjusts the model parameters in the initial expert model according to the respective load state description information of the at least one target expert network.

[0223] Optionally, the second determination unit 404 comprises: a similarity calculation module, configured to calculate the network similarity between the current expert network and the plurality of reference expert networks respectively; and determine the individual difference evaluation coefficient matched with the current expert network according to the first average value of the plurality of network similarities, wherein the individual difference evaluation coefficient has a negative correlation with the first average value.

[0224] ​Optionally, the third determining unit 406 is further configured to correct the initial expert selection vector matched with each of the individual difference evaluation coefficients of the plurality of expert network groups to obtain a plurality of target expert selection vectors, wherein the initial expert selection vector is used to determine the matching degree of the expert network for processing the input data; and determine at least one target expert network from the plurality of expert networks according to the activation processing result of the plurality of target expert selection vectors.

[0225] Optionally, the similarity calculation module is further configured to determine the network similarity between any two expert networks in the target expert network group as a group similarity set; determine a second average value according to the network similarity in the group similarity set; and determine the group difference evaluation coefficient matched with the target expert network group according to the second average value.

[0226] Optionally, the third determining unit 406 is further configured to obtain a first decision factor according to the product of the individual difference evaluation coefficient matched with the expert network and the first weight value; obtain a second decision factor according to the product of the group difference evaluation coefficient and the second weight value; correct the initial expert selection vector matched with the corresponding expert network according to the first decision factor and the second decision factor to obtain a plurality of target expert selection vectors, wherein the initial expert selection vector is used to determine the matching degree of the expert network for processing the input data; and determine at least one target expert network according to the activation processing result of the target expert selection vector.

[0227] Optionally, the third determining unit 406 includes a fourth determining module configured to determine the estimated resource occupation of the input data according to the complexity information of the input data, wherein the complexity information is used to indicate the difficulty of processing the input data; determine the selection probability matched with the plurality of expert network groups according to the estimated resource occupation; and determine the expert network group corresponding to the selection probability satisfying the target probability condition as the target expert network group.

[0228] Optionally, the fourth determining module is further configured to determine a maximum selection probability from the selection probabilities respectively matched with the plurality of expert network groups, determine an expert network group corresponding to the maximum selection probability as a target expert network group; determine a queue length of a task waiting queue respectively matched with the plurality of expert network groups, determine an expert network group corresponding to a selection probability greater than a first probability threshold and a queue length less than a second quantity threshold as the target expert network group; determine an average resource utilization respectively matched with the plurality of expert network groups, determine an expert network group corresponding to a selection probability greater than a second probability threshold and an average resource utilization greater than a third resource utilization threshold as the target expert network group; determine a historical processing data type respectively matched with the plurality of expert network groups, and determine an expert network group corresponding to a selection probability greater than a third probability threshold and a historical processing data type same as a data type of the input data as the target expert network group.

[0229] Optionally, the fourth determining module includes a fifth determining module configured to perform vectorization processing on the input data to obtain a vector sequence corresponding to the input data; determine semantic density information of the input data according to a frequency of each word vector appearing in the vector sequence; and determine complexity information of the input data according to the semantic density information.

[0230] Optionally, the fifth determining module is further configured to determine nested depth information of the input data according to a number of nested statements in the input data, wherein the nested statements include multiple layers of text structures; determine syntax depth information of the input data according to a number of target statements in the input data, wherein a text length of the target statement is greater than a target threshold; determine semantic coherence information of the input data according to a number of logical connection words in the input data; and determine the complexity information according to the semantic density information, the nested depth information, the syntax depth information and the semantic coherence information.

[0231] Optionally, the training unit includes a sixth determining module configured to determine utilization information of the at least one target expert network processing the input data respectively, wherein the utilization information is used to indicate an activity level of a computing resource when the target expert network processes the input data; determine a task backlog level of the at least one target expert network respectively; determine task discard rate information matched with the target expert network group; and determine the load state description information according to the utilization information of the at least one target expert network respectively, the task backlog level of the at least one target expert network respectively, and the task discard rate information matched with the target expert network group.

[0232] Optionally, the sixth determining module is further configured to perform vectorization splicing processing on the complexity information of the input data, the utilization information, the task backlog level, the task discard rate information and the inter-group difference evaluation coefficient to obtain a state vector, wherein the inter-group difference evaluation coefficient is determined according to the network similarity between any two expert networks in the target expert network group; and based on the state vector of the decision network in the input initial expert model, the model parameters are adjusted.

[0233] Optionally, the training unit 408 is further configured to obtain actual evaluation information of processing the input data by at least one expert network at a current time step, wherein the evaluation information is used to indicate the satisfaction degree of the execution behavior; determine predicted evaluation information of processing the input data at a next time step based on a reward prediction function; perform loss value calculation according to the actual evaluation information and the predicted evaluation information; and repeat the loss value calculation process until a minimum loss condition is met.

[0234] The features of the embodiments of the training apparatus of the expert model can be referred to the related descriptions of the embodiments of the training method of the expert model, which will not be repeated here.

[0235] The embodiments of the present application further provide an electronic device including a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to perform the steps in any of the embodiments of the training method of the expert model.

[0236] The electronic device to which the embodiments of the present application apply can be a terminal device or a server. In the embodiments, the electronic device is taken as a mobile phone and a computer as an example. As shown in Figure 5 the electronic device includes a memory 502 and a processor 504, the memory 502 storing a computer program, and the processor 504 being configured to execute the steps in any of the embodiments of the method through the computer program.

[0237] Optionally, in the embodiments, the electronic device can be located in at least one of the network devices in the computer network.

[0238] Optionally, those skilled in the art can understand that Figure 5 the structure shown in the figure is only schematic, and the electronic device can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a mobile Internet device (Mobile Internet Device, MID), a PAD, etc. Figure 4 It does not limit the structure of the electronic device. For example, the electronic device can include more or less components (such as a network interface, etc.) than Figure 5 shown in the figure, or have the same components as Figure 5Different configurations are shown.

[0239] The memory 502 can be used to store software programs and modules, such as program instructions / modules corresponding to the expert model training method and device in the embodiments of the present application. The processor 504 executes various functions by running the software programs and modules stored in the memory 502, that is, implements the expert model training method described above. The memory 502 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 502 can further include a memory remotely arranged with respect to the processor 504, which can be connected to the terminal through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The memory 502 can be used to store signals, data, and other information, but is not limited to this. Figure 5 As shown in the above, the memory 502 can include but is not limited to the first determination unit 402, the second determination unit 404, the third determination unit 406, and the adjustment training unit 408 in the expert model training device. In addition, other module units in the expert model training device can also be included, which will not be described in detail in this example.

[0240] Optionally, the transmission device 506 is used to receive or send data via a network. Specific examples of the above network can include wired networks and wireless networks. In one example, the transmission device 506 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers through a network cable to communicate with the Internet or a local area network. In one example, the transmission device 506 is a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet in a wireless manner.

[0241] In addition, the electronic device further includes a display 508 and a connection bus 510 for connecting various module components in the electronic device.

[0242] In other embodiments, the terminal device or the server can be a node in a distributed system, and the distributed system can be a blockchain system formed by the nodes communicating through a network. The nodes can form a point-to-point network, and any form of computing device, such as a server, a terminal, or an electronic device, can become a node in the blockchain system by joining the point-to-point network.

[0243] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in the training method of the expert model.

[0244] In an example embodiment, the computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media capable of storing computer programs.

[0245] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in the training method of the expert model.

[0246] The embodiment of the present application further provides another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the training method of the expert model.

[0247] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0248] The above provides a detailed description of the training method and device of the expert model, the storage medium and the electronic device. The principles and implementation modes of the present application are described by applying specific examples. The above description of the examples is only applicable to help understand the method and its core idea. It should be noted that, for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A method for training an expert model, characterized in that: include: Determining a target expert network group from a plurality of expert network groups included in the initial expert model according to estimated resource usage of the input data, wherein the expert network group includes a plurality of expert networks for processing the input data, and the estimated resource usage indicates computing resources required by the expert networks for processing the input data; Taking the multiple expert networks in the target expert network group as the current expert network in turn, determining the individual difference evaluation coefficients matching the current expert network, wherein the individual difference evaluation coefficients are determined based on the network similarities between the current expert network and multiple reference expert networks in the target expert network group, where the reference expert networks are the expert networks in the target expert network group other than the current expert network; determining at least one target expert network from a plurality of the expert networks in the target expert network group according to the respective individual difference evaluation coefficients of the plurality of the expert networks; The input data is processed according to at least one target expert network, and model parameters in the initial expert model are adjusted according to load state description information corresponding to each of the at least one target expert network.

2. The method according to claim 1, characterized in that Taking the plurality of expert networks in the target expert network group as current expert networks in turn, and determining individual difference evaluation coefficients matching the current expert networks, including: respectively calculating the network similarities between the current expert network and a plurality of reference expert networks; The individual difference evaluation coefficient that matches the current expert network is determined according to a first average value of a plurality of the network similarities, wherein the individual difference evaluation coefficient has a negative correlation with the first average value.

3. The method according to claim 2, characterized in that Determining at least one target expert network from the plurality of expert networks according to the respective individual difference evaluation coefficients of the plurality of expert networks in the target expert network group includes: According to the individual difference evaluation coefficients of the plurality of expert networks in the target expert network group, respectively modifying the initial expert selection vectors that match them to obtain a plurality of target expert selection vectors, wherein the initial expert selection vectors are used to determine the matching degree of the expert network for processing the input data; At least one target expert network is determined from the plurality of expert networks according to activation processing results of the plurality of target expert selection vectors.

4. The method according to claim 2, characterized in that After determining the individual difference evaluation coefficient that matches the current expert network, the method further includes: Determine the network similarity between any two expert networks in the target expert network group as an inter-group similarity set; determining a second average value based on the network similarities in the inter-group similarity set; An inter-group difference evaluation coefficient matching the target expert network group is determined based on the second average value.

5. The method according to claim 4, characterized in that Determining at least one target expert network from a plurality of expert networks further includes: Obtaining a first decision factor according to the product of the individual difference evaluation coefficient matched with the expert network and the first weight value; Obtaining a second decision factor according to the product of the inter-group difference evaluation coefficient and the second weight value; Modifying an initial expert selection vector that matches the corresponding expert network according to the first decision factor and the second decision factor to obtain a plurality of target expert selection vectors, wherein the initial expert selection vector is used to determine a matching degree of the expert network for processing the input data; At least one target expert network is determined according to the activation processing result of the target expert selection vector.

6. The method according to claim 1, characterized in that A target expert network group is determined from multiple expert network groups included in the initial expert model based on the estimated resource usage of the input data, and further includes: Determining the estimated resource occupancy of the input data according to complexity information of the input data, wherein the complexity information is used to indicate the difficulty of processing the input data; Determining selection probabilities of matching with the plurality of expert network groups respectively according to the estimated resource occupancy; The expert network group corresponding to the selection probability that meets the target probability condition is determined as the target expert network group.

7. The method according to claim 6, characterized in that The expert network group corresponding to the selection probability that meets the target probability condition is determined as the target expert network group, further comprising at least one of the following: determining a maximum selection probability among the selection probabilities respectively matching the plurality of expert network groups, and determining the expert network group corresponding to the maximum selection probability as the target expert network group; Determining the queue lengths of task waiting queues respectively matching the plurality of expert network groups, and determining the expert network group corresponding to the selection probability being greater than a first probability threshold and the queue length being less than a second quantity threshold as the target expert network group; Determining average resource utilization rates that match the plurality of expert network groups respectively, and determining the expert network group corresponding to the expert network group having the selection probability greater than the second probability threshold and the average resource utilization rate greater than the third resource utilization threshold as the target expert network group; Determine the historical processing data types that match the multiple expert network groups respectively, and determine the expert network group whose selection probability is greater than a third probability threshold and whose historical processing data type is the same as the data type of the input data as the target expert network group.

8. The method according to claim 6, characterized in that Before determining the estimated resource occupancy of the input data according to the complexity information of the input data, the method further includes: Performing vectorization processing on the input data to obtain a vector sequence corresponding to the input data; Determining semantic density information of the input data based on the frequency of each word vector appearing in the vector sequence; The complexity information of the input data is determined according to the semantic density information.

9. The method according to claim 8, characterized in that Determining the complexity information of the input data further includes: Determining nesting depth information of the input data according to the number of nested sentences in the input data, wherein the nested sentences contain a multi-layer text structure; determining grammatical depth information of the input data based on a number of target sentences in the input data, wherein a text length of the target sentences is greater than a target threshold; determining semantic coherence information of the input data based on the number of logical connectives in the input data; The complexity information is determined according to the semantic density information, the nesting depth information, the grammatical depth information and the semantic coherence information.

10. The method according to claim 8, characterized in that Before adjusting the model parameters in the initial expert model, the following steps are included: determining utilization information of at least one target expert network for processing the input data, wherein the utilization information is used to indicate an activity level of the computing resources when the target expert network processes the input data; determining a task backlog level of each of at least one of the target expert networks; Determining task discard rate information matching the target expert network group; The load state description information is determined according to the utilization information of each of at least one of the target expert networks, the task backlog level, and the task discard rate information matching the target expert network group.

11. The method according to claim 10, characterized in that Adjusting model parameters in the initial expert model according to the load state description information includes: Vectorizing and concatenating the complexity information, utilization information, task backlog, task abandonment rate, and inter-group difference evaluation coefficient of the input data to obtain a state vector, wherein the inter-group difference evaluation coefficient is determined based on the network similarity between any two expert networks in the target expert network group; The model parameters are adjusted based on the state vector input into the decision network in the initial expert model.

12. The method according to claim 1, characterized in that After adjusting the model parameters in the initial expert model according to the load state description information, the method includes: Obtaining actual evaluation information of at least one of the expert networks processing the input data at a current time step, wherein the evaluation information is used to indicate satisfaction with the execution behavior; Determining predicted evaluation information for processing the input data at a next time step based on a reward prediction function; Calculating a loss value based on the actual evaluation information and the predicted evaluation information; Repeat the above loss value calculation process until the minimum loss condition is met.

13. A training device for an expert model, characterized in that: include: a first determining unit, configured to determine a target expert network group from a plurality of expert network groups included in the initial expert model according to estimated resource occupancy of the input data, wherein the expert network group includes a plurality of expert networks for processing the input data, and the estimated resource occupancy indicates computing resources required by the expert networks for processing the input data; a second determining unit, taking the plurality of expert networks in the target expert network group as current expert networks in sequence, and determining individual difference evaluation coefficients matching the current expert network, wherein the individual difference evaluation coefficients are determined based on respective network similarities between the current expert network and a plurality of reference expert networks in the target expert network group, the reference expert networks being the expert networks in the target expert network group excluding the current expert network; a third determining unit, configured to determine at least one target expert network from a plurality of the expert networks in the target expert network group according to the respective individual difference evaluation coefficients of the plurality of the expert networks; The adjustment training unit processes the input data according to at least one target expert network and adjusts the model parameters in the initial expert model according to the load state description information corresponding to each of the at least one target expert network.

14. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the expert model training method according to any one of claims 1 to 12 when executing the computer program.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the expert model training method according to any one of claims 1 to 12 are implemented.

Images