Hybrid expert model training method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a training method and device of a hybrid expert model, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a training sample set of a hybrid expert model associated with large model input; the training sample set comprises input data of the large model and input analysis label information corresponding to the input data; inputting a target sample in the training sample set into the hybrid expert model to obtain activation probability information of each expert model in the hybrid expert model, a target output result of the hybrid expert model and a sub-output result of the loading expert model; according to the model parameter, the target sample, the target output result and the sub-output result of each expert model, obtaining expert distinguishing reward information; and performing iterative training on the hybrid expert model according to the activation probability information, the expert distinguishing reward information, the target sample and the target output result to obtain a trained hybrid expert model. By adopting the method, the robustness of the hybrid expert model can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of machine learning, and in particular to a training method, apparatus, computer equipment, computer-readable storage medium, and computer program product for a hybrid expert model. Background Art

[0002] In recent years, with the rapid development of deep learning technology, large-scale neural networks have made significant progress in natural language processing, computer vision and other fields. However, as the scale of the model continues to expand, the computational overhead of training and inference has increased dramatically, which has put higher requirements on hardware resources and training efficiency. Mixture of Experts (MoE), as a sparse activation mechanism, effectively reduces the computational cost by dynamically activating some expert models to process input data while maintaining the high performance of the model, and has received widespread attention.

[0003] However, in the existing training process, the regulation mechanism of the gated network is relatively simple, which may cause multiple experts to learn similar feature processing methods and parameter configurations during training. This homogenization phenomenon causes the trained expert model to produce the same or similar output responses to similar inputs, thereby reducing the actual effective capacity of the model and resulting in low robustness of the hybrid expert model. Summary of the invention

[0004] Based on this, it is necessary to provide a training method, apparatus, computer equipment, computer-readable storage medium and computer program product for a hybrid expert model that can improve the robustness of the hybrid expert model in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for training a hybrid expert model, comprising:

[0006] Acquire a training sample set of a hybrid expert model associated with a large model input; the training sample set includes input data of the large model and input parsing label information corresponding to the input data;

[0007] Inputting the target sample in the training sample set into the hybrid expert model, obtaining activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model and the sub-output result of the loaded expert model; the loaded expert model represents a preset number of expert models in the hybrid expert model;

[0008] Obtaining expert distinction reward information according to the model parameters of each expert model, the target sample, the target output result, and the sub-output result;

[0009] The hybrid expert model is iteratively trained according to the activation probability information, the expert distinction reward information, the target sample and the target output result to obtain a trained hybrid expert model.

[0010] In one embodiment, obtaining expert distinction reward information according to the model parameters of each expert model, the target sample, the target output result and the sub-output result includes:

[0011] Determining model diversity reward information according to the model parameters of each expert model;

[0012] Determining output diversity reward information according to the sub-output results;

[0013] Determining input-output difference reward information according to the target sample and the target output result;

[0014] The expert differentiation reward information is determined according to the model diversity reward information, the output diversity reward information and the input-output difference reward information.

[0015] In one embodiment, determining the model diversity reward information according to the model parameters of each expert model includes:

[0016] Determining parameter difference information between two expert models according to the model parameters of each expert model;

[0017] The model diversity reward information is determined according to the parameter difference information.

[0018] In one embodiment, determining output diversity reward information according to the sub-output results includes:

[0019] Determining output difference information between two of the sub-output results according to the sub-output results;

[0020] The output diversity reward information is determined according to the output difference information.

[0021] In one embodiment, determining input-output difference reward information according to the target sample and the target output result includes:

[0022] Determining sample difference information and output result difference information between each pair of the target samples according to the target samples and the target output results;

[0023] Determining discrimination information between each pair of the target samples according to the sample difference information and the output result difference information;

[0024] The input-output difference reward information is determined according to the discrimination information.

[0025] In one of the embodiments, the hybrid expert model includes a gated network model and a plurality of expert models;

[0026] The iterative training of the hybrid expert model according to the activation probability information, the expert distinguishing reward information, the target sample and the target output result includes:

[0027] In each iterative training, a gradient product term of the parameter update gradient is determined according to a preset baseline function and the expert distinguishing reward information;

[0028] According to the activation probability information and the gradient product term, the parameters of the gated network model are updated, and according to the target sample, the target output result and the gradient product term, the parameters of the target expert model are updated; the target expert model is an expert model determined from the multiple expert models according to the activation probability information of this iterative training.

[0029] In one embodiment, determining a gradient product term of the parameter update gradient according to the preset baseline function and the expert distinguishing reward information comprises:

[0030] Obtain historical expert distinction reward information corresponding to the iterative training before the current iterative training;

[0031] Determine a moving average of reward information based on the historical expert differentiated reward information and the expert differentiated reward information;

[0032] The difference between the expert distinguishing reward information and the moving average of the reward information is used as the gradient product term.

[0033] In a second aspect, the present application also provides a training device for a hybrid expert model, comprising:

[0034] A sample acquisition module, used to acquire a training sample set of a hybrid expert model associated with a large model input; the training sample set includes input data of the large model and input parsing label information corresponding to the input data;

[0035] A model input module, used to input the target sample in the training sample set into the hybrid expert model, and obtain the activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model and the sub-output result of the loaded expert model; the loaded expert model represents a preset number of expert models in the hybrid expert model;

[0036] A reward acquisition module, used to obtain expert distinction reward information according to the model parameters of each expert model, the target sample, the target output result and the sub-output result;

[0037] The model training module is used to iteratively train the hybrid expert model according to the activation probability information, the expert distinction reward information, the target sample and the target output result to obtain a trained hybrid expert model.

[0038] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0039] Acquire a training sample set of a hybrid expert model associated with a large model input; the training sample set includes input data of the large model and input parsing label information corresponding to the input data;

[0040] Inputting the target sample in the training sample set into the hybrid expert model, obtaining activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model and the sub-output result of the loaded expert model; the loaded expert model represents a preset number of expert models in the hybrid expert model;

[0041] Obtaining expert distinction reward information according to the model parameters of each expert model, the target sample, the target output result, and the sub-output result;

[0042] The hybrid expert model is iteratively trained according to the activation probability information, the expert distinction reward information, the target sample and the target output result to obtain a trained hybrid expert model.

[0043] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0044] Acquire a training sample set of a hybrid expert model associated with a large model input; the training sample set includes input data of the large model and input parsing label information corresponding to the input data;

[0045] Inputting the target sample in the training sample set into the hybrid expert model, obtaining activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model and the sub-output result of the loaded expert model; the loaded expert model represents a preset number of expert models in the hybrid expert model;

[0046] Obtaining expert distinction reward information according to the model parameters of each expert model, the target sample, the target output result, and the sub-output result;

[0047] The hybrid expert model is iteratively trained according to the activation probability information, the expert distinction reward information, the target sample and the target output result to obtain a trained hybrid expert model.

[0048] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0049] Acquire a training sample set of a hybrid expert model associated with a large model input; the training sample set includes input data of the large model and input parsing label information corresponding to the input data;

[0050] Inputting the target sample in the training sample set into the hybrid expert model, obtaining activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model and the sub-output result of the loaded expert model; the loaded expert model represents a preset number of expert models in the hybrid expert model;

[0051] Obtaining expert distinction reward information according to the model parameters of each expert model, the target sample, the target output result, and the sub-output result;

[0052] The hybrid expert model is iteratively trained according to the activation probability information, the expert distinction reward information, the target sample and the target output result to obtain a trained hybrid expert model.

[0053] The training method, apparatus, computer device, computer-readable storage medium and computer program product of the hybrid expert model are as follows: first, a training sample set of the hybrid expert model associated with the input of the large model is obtained, wherein the training sample set includes the input data of the large model and the input parsed label information corresponding to the input data. The input data and the corresponding parsed label information help the model to accurately understand the relationship between the input and the expected output, provide sufficient supervision signals for subsequent training, and help improve the training effect and the adaptability of the model to specific tasks; then, the target sample in the training sample set is input into the hybrid expert model to obtain the activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model and the sub-output result of the loaded expert model, wherein the loaded expert model represents a preset number of expert models in the hybrid expert model. By introducing the sub-output results of the loaded expert model, the calculation results of the expert model that may not be activated are incorporated into the reward and parameter update process, thereby making full use of the computing resources, so that the outputs of more expert models are also used to participate in the calculation of the reward value, which not only effectively avoids the idleness of computing resources, but also improves the efficiency of the reward by performing the output calculation on multiple expert models. A comprehensive evaluation is performed to enhance the diversity and task adaptability of expert models, so that the model can better utilize the computing power of all expert models during the training process, improve the training efficiency, and enhance the robustness and overall performance of the model. Then, according to the model parameters, target samples, target output results and sub-output results of each expert model, the expert discrimination reward information is obtained. By providing expert discrimination rewards for iterative training, different experts are encouraged to learn different features or tasks, which improves the diversity and robustness of the model. It can effectively avoid multiple experts from learning similar feature processing methods, thereby enhancing the model's adaptability when facing different inputs. Finally, according to the activation probability information, expert discrimination reward information, target samples and target output results, the hybrid expert model is iteratively trained to obtain a trained hybrid expert model. The hybrid expert model is iteratively trained using the activation probability information, expert discrimination reward information, target samples and target output results. The gating network can more effectively learn how to assign tasks to different experts and update the model parameters according to the reward signal, so that the model gradually optimizes the expert selection strategy and improves the overall performance of the model and the adaptability of the expert. In the above method, by introducing expert discrimination reward information, the problem of expert homogeneity that may occur in the mixed expert model is effectively solved. By calculating the expert discrimination reward for each iterative training, the diversity of different experts in feature learning and task processing is encouraged, avoiding similar parameter configurations and outputs between experts, improving the robustness and diversity of the model, enabling the model to better adapt to complex tasks and input changes, and significantly improving the overall performance and training stability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 A schematic diagram of a flow chart of a training method for a hybrid expert model in one embodiment;

[0056] Figure 2 A flowchart of the steps of obtaining expert distinction reward information in one embodiment;

[0057] Figure 3 is a flow chart of a training method of a hybrid expert model in another embodiment;

[0058] Figure 4 is a structural block diagram of a training device for a hybrid expert model in one embodiment;

[0059] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] In one embodiment, Figure 1 As shown, a training method for a hybrid expert model is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various personal computers, laptops, smart phones, and tablet computers. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. In this embodiment, the method includes the following steps:

[0062] Step S101, obtaining a training sample set of a hybrid expert model associated with a large model input.

[0063] The training sample set includes the input data of the large model and the input parsing label information corresponding to the input data.

[0064] Among them, the training sample set is a set consisting of input data and input parsing label information. Input data is the raw data that needs to be processed by the large model, covering various types of task inputs, including but not limited to text data, image data, voice data, video data, etc. Each type of input data usually needs to be equipped with corresponding input parsing label information. This part of data is used to provide the correspondence between the input and the target output or the reference answer to guide the learning process of the model and help the model optimize its parameters during the training process. Input parsing label information is annotated data closely related to the input data. In supervised learning tasks, label information is a key training basis.

[0065] Exemplarily, the terminal collects and organizes input data from data sources. These data usually include text, images, voice, etc. Depending on the task, they may be original user input, annotated image data or text data, etc. The terminal then prepares corresponding parsing label information for these input data. The parsing label is usually the correct answer or output result provided by an expert or artificial intelligence system for the input data. This label information is used to train the model so that it can optimize the parameters of the model by learning the mapping relationship between the input and the label. Finally, the terminal sends the organized data set to the hybrid expert model as a training sample set for subsequent training to support the model's accurate prediction in actual tasks.

[0066] Specifically, ① Input data: natural language text, such as news articles, product reviews, problem descriptions, etc.; corresponding input parsing label information: sentiment labels for sentiment analysis tasks, labels for translation tasks are translated texts in the target language, category labels for text classification tasks (such as technology, sports, entertainment), and answer text labels for question-answering tasks. ② Input data: image data, including static images, video images, or continuous frame image sequences; corresponding input parsing label information: category labels for image / video classification tasks (such as cats, cars), object positions (such as coordinate information) / object category labels for target detection tasks, and action labels for video action recognition tasks (such as running, jumping). ③ Input data: speech signals or audio files; corresponding input parsing label information: text labels for speech-to-text conversion for speech recognition tasks, emotion labels for speech sentiment analysis, and so on.

[0067] Step S102, inputting the target sample in the training sample set into the hybrid expert model, obtaining the activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model and the sub-output result of the loaded expert model.

[0068] Among them, the loaded expert model refers to the preset number of expert models in the hybrid expert model. Specifically, it refers to some experts selected from all expert models in the hybrid expert model according to the preset strategy, and these models are used to participate in the reward calculation. The selection method of the loaded expert model can include the Top-K strategy or the probabilistic selection strategy to ensure the full utilization of resources while improving the comprehensiveness of the reward calculation and the model performance. And it may not be completely consistent with the activated expert model of the hybrid expert model (the activated expert model is the expert model used to obtain the target output result of the hybrid expert model).

[0069] Among them, the activation probability information refers to the activation probability distribution of the expert model selected by the gating network for the current target sample.

[0070] Exemplarily, during each iterative training, the terminal randomly selects target samples from the training sample set and inputs them into the hybrid expert model. The hybrid expert model generates activation probability information for each expert model through a gating network, determines which experts are activated to participate in the forward calculation of the current input, and generates the target output result of the hybrid expert model. At the same time, the terminal further selects a loading expert model from all expert models based on the preset maximum number of loadable experts, and obtains the sub-output result of the loading expert model for subsequent reward value calculation; the maximum number of loadable experts can be determined based on hardware resources. For example, the Top-K strategy can be used to select the loading expert model, then the K expert models with the highest probability are selected as the loading expert models based on the activation probability information; it is also possible to select some models from the activated expert models of the hybrid expert model with probability as the loading expert models.

[0071] Step S103, obtaining expert distinction reward information according to the model parameters, target samples, target output results and sub-output results of each expert model.

[0072] Among them, the expert discrimination reward information is a reward value determined by evaluating the differences in parameter configuration and output characteristics between expert models. The calculation goal of the expert discrimination reward information is to encourage each expert model to learn independent feature representation and task division to enhance the diversity of the model and avoid the problem of expert homogeneity.

[0073] Exemplarily, the terminal calculates the difference between experts by analyzing the parameter information (such as weight distribution, gradient change) of each expert model and its sub-output results on the current target sample. For example, the terminal compares the sub-output results of the loaded expert model and evaluates the output difference between them. At the same time, combined with the target output result, the expert model's ability to distinguish the target sample is analyzed. In addition, by evaluating the parameter differences of the expert model (such as the distance or distribution difference of the parameter value), the independence of the expert model is further quantified.

[0074] Step S104, iteratively training the hybrid expert model according to the activation probability information, the expert distinction reward information, the target sample and the target output result to obtain a trained hybrid expert model.

[0075] Exemplarily, the terminal selects a target sample from the training sample set and inputs it into the hybrid expert model to obtain the target output result, activation probability information, and expert distinguishing reward information. Combined with this information, the terminal iteratively optimizes the hybrid expert model. First, the terminal uses the expert distinguishing reward information and activation probability information to update the parameters of the gated network through the policy gradient optimization algorithm, so that it can more accurately select a suitable expert model and improve the rationality of the activation probability distribution. Secondly, the terminal optimizes the parameters of the activated expert model in combination with the expert distinguishing reward information, adjusts the model gradient direction, and encourages the expert model to form differentiated characteristics and task focus capabilities during the learning process. During the training process, the inactivated expert model indirectly participates in the reward calculation and parameter update through the loading mechanism, further improving the resource utilization and adaptability of the expert model. After multiple iterative training, the terminal can obtain an optimized hybrid expert model with higher task accuracy, better generalization ability, and stronger robustness.

[0076] In the training method of the above hybrid expert model, first, a training sample set of the hybrid expert model associated with the input of the large model is obtained, wherein the training sample set includes the input data of the large model and the input parsed label information corresponding to the input data. The input data and the corresponding parsed label information help the model to accurately understand the relationship between the input and the expected output, provide sufficient supervision signals for subsequent training, and help improve the training effect and the adaptability of the model to specific tasks; then, the target sample in the training sample set is input into the hybrid expert model to obtain the activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model and the sub-output result of the loaded expert model, wherein the loaded expert model represents a preset number of expert models in the hybrid expert model. By introducing the sub-output results of the loaded expert model, the calculation results of the expert model that may not be activated are incorporated into the reward and parameter update process, thereby making full use of the computing resources, so that the outputs of more expert models are also involved in the calculation of the reward value, which not only effectively avoids the idleness of computing resources, but also enhances the expert model by comprehensively evaluating the outputs of multiple expert models. The diversity and task adaptability between them are improved, so that the model can better utilize the computing power of all expert models during the training process, improve the training efficiency, and strengthen the robustness and overall performance of the model. Then, according to the model parameters, target samples, target output results and sub-output results of each expert model, the expert discrimination reward information is obtained. By providing expert discrimination rewards for iterative training, different experts are encouraged to learn different features or tasks, which improves the diversity and robustness of the model. It can effectively avoid multiple experts from learning similar feature processing methods, thereby enhancing the model's adaptability when facing different inputs. Finally, according to the activation probability information, expert discrimination reward information, target samples and target output results, the hybrid expert model is iteratively trained to obtain a trained hybrid expert model. The hybrid expert model is iteratively trained using the activation probability information, expert discrimination reward information, target samples and target output results. The gating network can more effectively learn how to assign tasks to different experts and update the model parameters according to the reward signal, so that the model gradually optimizes the expert selection strategy and improves the overall performance of the model and the adaptability of the experts. In the above method, by introducing expert discrimination reward information, the problem of expert homogeneity that may occur in the mixed expert model is effectively solved. By calculating the expert discrimination reward for each iterative training, the diversity of different experts in feature learning and task processing is encouraged, avoiding similar parameter configurations and outputs between experts, improving the robustness and diversity of the model, enabling the model to better adapt to complex tasks and input changes, and significantly improving the overall performance and training stability of the model.

[0077] In an exemplary embodiment, Figure 2As shown, the above step S103 obtains the expert distinction reward information according to the model parameters, target samples, target output results and sub-output results of each expert model, and can also be implemented by the following steps:

[0078] Step S201, determining model diversity reward information according to the model parameters of each expert model;

[0079] Step S202, determining output diversity reward information according to the sub-output results;

[0080] Step S203, determining input-output difference reward information according to the target sample and the target output result;

[0081] Step S204: determining expert differentiation reward information according to the model diversity reward information, the output diversity reward information and the input-output difference reward information.

[0082] Exemplarily, the terminal first extracts the parameter information of each expert model from the hybrid expert model, including its weight matrix, bias value, and network structure characteristics. The terminal quantifies the distribution difference of the model parameters by calculating the difference between the expert model parameters, such as Euclidean distance, KL divergence, or cosine similarity, and integrates these differences into model diversity reward information, which is used to measure the independence and diversity of the expert model at the parameter level. Subsequently, the terminal calculates the output diversity reward information for the sub-output results of the loaded expert model by comparing the numerical distribution of these output results or the directionality of the feature vectors. For example, the terminal can determine the difference in the response of each expert to the same input by comparing the cosine similarity or distance between the output vectors of the loaded expert model, thereby deriving the degree of diversity of the output characteristics. Next, the terminal analyzes the matching relationship between the input features of the target sample and the target output results. By calculating the correlation between the input sample features (such as image features, text embedding vectors, etc.) and the corresponding target output features, the mapping relationship between the input and output is quantified, thereby generating input-output difference reward information for evaluating the distinguishing ability of the expert model when processing specific tasks. Finally, the terminal performs weighted integration of model diversity reward information, output diversity reward information, and input-output difference reward information. The weight setting can be based on the priority of the task or the training goal of the expert model. The integrated expert differentiation reward information can not only fully reflect the differences between expert models, but also provide multi-dimensional guidance for the optimization of expert models.

[0083] In this embodiment, by introducing a multi-dimensional calculation mechanism of model diversity reward information, output diversity reward information and input-output difference reward information, it is possible to provide more dimensional references for expert model optimization by evaluating the performance of the expert model in parameters, output and input-output matching, thereby improving the adaptability and robustness of the hybrid expert model, significantly improving the independence and diversity between expert models, reducing homogenization, and ensuring its efficient performance in complex tasks.

[0084] In an exemplary embodiment, the above step S201 determines the model diversity reward information according to the model parameters of each expert model, and also includes: determining the parameter difference information between the expert models according to the model parameters of each expert model; and determining the model diversity reward information according to the parameter difference information.

[0085] Exemplarily, the terminal extracts parameter information of each expert model from the hybrid expert model, including its weight matrix and bias value. The terminal calculates the difference information between the expert models by comparing the parameters of the expert models pair by pair. For example, for each pair of expert models, the terminal can calculate the Euclidean distance and cosine similarity of the weight matrix, or measure the difference between the two by the KL divergence (Kullback-Leibler divergence) of the parameter distribution. In addition, for expert models with different hierarchical structures, the terminal can also calculate their difference information based on the statistical characteristics of the parameters between the hierarchies (such as mean and variance). By summarizing and summarizing the parameter difference information between all expert models, the terminal generates a comprehensive parameter difference distribution matrix. Subsequently, the terminal can further calculate the model diversity reward information based on the difference distribution matrix. This process can quantify the distribution independence of different expert models in the overall parameter space by normalizing the difference information.

[0086] In a specific example, for any two expert models and , calculate the difference in its parameters:

[0087]

[0088] in, and Experts and The parameter vector of is the dimension of the parameter and || || is the Euclidean norm.

[0089] Calculate the average of the parameter differences between all N expert models to determine the model diversity reward information:

[0090]

[0091] The reward item of model diversity reward information in expert discrimination reward information is: . is the weight coefficient.

[0092] In this embodiment, by introducing the parameter difference information between expert models and calculating the model diversity reward information based on the difference information, the independence between expert models can be effectively measured and enhanced, and the homogenization phenomenon can be reduced. In addition, this process can also dynamically adjust the distribution of reward values, promote the expert model to form a more diverse parameter configuration during training, and thus improve the overall adaptability and robustness of the hybrid expert model.

[0093] In an exemplary embodiment, the above step S202 determines output diversity reward information according to the sub-output results, and further includes: determining output difference information between two sub-output results according to the sub-output results; and determining output diversity reward information according to the output difference information.

[0094] Exemplarily, the terminal obtains the sub-output results of each expert from the loaded expert model, and the sub-output results can be numerical vectors, classification probability distributions, or other forms of model intermediate results. The terminal calculates output difference information by comparing these sub-output results in pairs. For example, the terminal can use Euclidean distance or cosine similarity to measure its output difference. In addition, for the sub-output results of the probability distribution type, the terminal can calculate KL divergence or Jensen-Shannon divergence to evaluate the difference in distribution between the two sub-outputs. After completing the pairwise comparison and calculating the output difference information between all loaded expert models, the terminal summarizes and integrates these difference information, for example, by calculating the average output difference value to quantify the overall output diversity of the loaded expert model. Subsequently, the terminal generates output diversity reward information based on these difference information. The higher the reward value, the higher the diversity of the output characteristics of the loaded expert model. Finally, the output diversity reward information is used to guide the training process, prompting the expert model to form more independent characteristics when processing tasks, thereby enhancing the robustness and task adaptability of the hybrid expert model.

[0095] In a specific example, for an input sample x t , any two loaded expert models and The sub-output results and , calculate the difference in their output:

[0096]

[0097] For the input sample x t, calculate the average of the output differences between all K loaded expert models:

[0098]

[0099] For T target samples, Further average the data to determine the output diversity reward information:

[0100]

[0101] The reward item of output diversity reward information in expert distinction reward information is: . is the weight coefficient.

[0102] In this embodiment, the calculation of the output diversity reward information can fully explore the differences in the sub-output results of the loaded expert model, avoid the convergence of multiple experts in the output characteristics, further optimize the training direction of the expert model, and enable the hybrid expert model to demonstrate stronger division of labor and cooperation capabilities in complex tasks, while improving the overall performance and generalization ability of the model.

[0103] In an exemplary embodiment, the above-mentioned step S203 determines the input-output difference reward information based on the target samples and the target output results, and also includes: determining the sample difference information and the output result difference information between each target sample based on the target samples and the target output results; determining the discrimination information between each target sample based on the sample difference information and the output result difference information; and determining the input-output difference reward information based on the discrimination information.

[0104] Exemplarily, the terminal extracts the input features and target output results of each target sample. First, the terminal compares the input features of the target samples in pairs, for example, by calculating the Euclidean distance, cosine similarity, etc. of the feature vectors, to generate sample difference information between the target samples to quantify the discrimination of the input samples. Next, the terminal compares the target output results in pairs, for example, by calculating the numerical difference or distribution difference (such as KL divergence) of the output vectors, and calculates the output result difference information to quantify the discrimination of the model output characteristics. Next, the terminal compares the input difference information and the output difference information of each pair of samples, and evaluates the discrimination information between the target samples by calculating their absolute difference. For example, when the input difference information is large but the output difference information is small, the discrimination information of the pair of samples will be reflected as a negative reward signal; and when the input difference information is close to the output difference information, the discrimination information of the pair of samples will be reflected as a positive reward signal, thereby encouraging the selection strategy of the gated network to be more in line with the input-output relationship. Finally, the terminal integrates all the discrimination information of the target sample, for example, by generating input-output difference reward information through weighted averaging or normalization. The reward goal is to enable the gated network to output the same discrimination as the input when processing different inputs. If the input data has a large discrimination, the gated network should output a large discrimination (the output of the gated network can be reflected by the output of the hybrid expert model), and vice versa. This reward information guides the hybrid expert model to more accurately match the relationship between input characteristics and output results by directly optimizing the gated network selection strategy.

[0105] In a specific example, for an input sample and The discrimination degree is :

[0106]

[0107] The corresponding target output result and , the sample label is and , its output discrimination for:

[0108]

[0109] Input Sample and The discrimination information is:

[0110]

[0111] For T target samples, the discrimination information of all sample pairs is further averaged to determine the input-output difference reward information:

[0112]

[0113] The reward item of the input-output difference reward information in the expert distinction reward information is: . is the weight coefficient.

[0114] In this embodiment, by comparing the input difference information and output difference information of the target samples pair by pair, it is ensured that the gated network can maintain the consistency of distinction between input and output when processing the target samples. The output difference of samples with large input differences should also be significant, and the output difference of samples with small input differences should maintain moderate consistency. This can effectively optimize the expert selection strategy of the gated network, improve the task adaptability and differentiation ability of the model, and further enhance the robustness and generalization ability of the model in complex scenarios.

[0115] In an exemplary embodiment, the hybrid expert model includes a gated network model and multiple expert models; the above-mentioned step S104 iteratively trains the hybrid expert model according to the activation probability information, the expert discrimination reward information, the target sample and the target output result, and also includes: in each iterative training, according to the preset baseline function and the expert discrimination reward information, a gradient product term of the parameter update gradient is determined; according to the activation probability information and the gradient product term, the parameters of the gated network model are updated, and according to the target sample, the target output result and the gradient product term, the parameters of the target expert model are updated; the target expert model is an expert model determined from multiple expert models according to the activation probability information of this iterative training.

[0116] Among them, the gradient product term is calculated by combining the preset baseline function and the expert distinguishing reward information. As the core factor in the parameter update process, it can stabilize the gradient change, reduce the variance in training, and improve the convergence efficiency of model training. The target expert model is an expert model dynamically selected in each iteration through the activation probability information, and is trained and optimized for the target sample.

[0117] Exemplarily, in each iterative training, the terminal first calculates the activation probability information of the target sample through the gated network model and dynamically selects the target expert model. Then, the terminal combines the expert distinction reward information R and the historical reward value corresponding to the current target sample to calculate the gradient product term to reflect the relationship between the reward signal and the baseline and its impact on parameter update. Next, the terminal uses the activation probability information and the gradient product term to update the parameters of the gated network so that it can more accurately select a suitable target expert model. For the target expert model, the terminal uses the input and target output results of the target sample to calculate the gradient through the back propagation algorithm, and updates the parameters of the target expert model in combination with the gradient product term.

[0118] In a specific example, the update gradient of the gating network is:

[0119]

[0120] in, For the gating network according to the state s t Select action a t The probability distribution of state s t For the input data sample x t , which can be the original input or the feature vector after being processed by the previous network; action a t is the target expert model selected by the gating network for the current input; G is the gradient product term.

[0121] Update the gating network parameters : , is the learning rate.

[0122] Update target expert network parameters : .in, is the learning rate; The updated gradient corresponding to the loss function of the model output and label information.

[0123] In addition, the target sample for each iterative training is composed of multiple samples, and the target expert model activated by each sample is different. Therefore, each target expert model is updated only based on the target sample mapped to itself.

[0124] In this embodiment, by introducing the gradient product term, the guiding role of the reward signal on the gradient update is effectively enhanced, and the baseline function is used to smooth the reward fluctuations during the training process. The gated network can gradually improve the accuracy in the selection of the target expert model, and the target expert model also gradually optimizes its adaptability to specific tasks during iterative training, thereby significantly improving the robustness and overall performance of the hybrid expert model in complex task scenarios.

[0125] In an exemplary embodiment, the above-mentioned method of determining a gradient product term of the parameter update gradient based on a preset baseline function and expert differentiated reward information also includes: obtaining historical expert differentiated reward information corresponding to the iterative training before the current iterative training; determining a moving average of the reward information based on the historical expert differentiated reward information and the expert differentiated reward information; and using the difference between the expert differentiated reward information and the moving average of the reward information as the gradient product term.

[0126] The historical expert-discriminative reward information refers to the expert-discriminative reward information calculated in several iterations before the current iterative training, which is used to provide a long-term reward signal baseline. The moving average of the reward information is calculated by combining the current reward information and the historical reward information, which is used to smooth the reward signal and reduce the impact of fluctuations on gradient updates.

[0127] For example, in each iteration of training, the terminal first obtains the expert distinction reward information of the current iteration training Then, if there is historical reward information before this iteration, the terminal averages the recent historical reward values ​​and the current reward value to obtain the moving average value b of the reward information. Next, the terminal calculates the difference between the current reward value and the moving average value to generate the gradient product term: In the first iteration of training, due to the lack of historical reward information, the moving average cannot be calculated directly, so the terminal can initialize the moving average to 0.

[0128] in addition, .

[0129] In this embodiment, by introducing the moving average of historical reward information, the impact of the fluctuation of the reward signal on the gradient update is significantly reduced, making the training more stable. In particular, in the first iteration, the problem of lack of historical reward information is solved through a reasonable initialization strategy to ensure the correctness of the gradient product term and the consistency of the training process. At the same time, the moving average mechanism can balance the relationship between the current reward value and the historical reward trend, thereby improving the convergence efficiency and robustness of the model in complex tasks.

[0130] In another exemplary embodiment, Figure 3 As shown, the present application provides a training method for a hybrid expert model, the method comprising:

[0131] Step S301, initializing network parameters.

[0132] Step S302, gated network forward propagation.

[0133] Step S303, forward propagation of the maximum load expert network.

[0134] Step S304, obtaining the target expert network calculation result.

[0135] Step S305, obtaining the maximum load expert network calculation result.

[0136] Step S306, obtaining the hybrid expert model output.

[0137] Step S307, expert differentiation reward information calculation.

[0138] Step S308, update the gradient calculation.

[0139] Step S309: model update.

[0140] For example, the entire hybrid expert model is used as the decision environment of the agent, including the expert network, the gating network and the input data set; the gating network is used as the agent, responsible for selecting the appropriate expert network according to the input state; the feature representation of the input sample is used as the state, which can be the original input or the feature vector after being processed by the front network; the expert network selected by the gating network for the current input is used as the action; the probability distribution of the gating network selecting the action according to the state is used as the strategy; and the expert distinguishing reward information is used as the reward value. In this way, the training optimization of the hybrid expert model is realized.

[0141] In this embodiment, the hybrid expert model is modeled by reinforcement learning, and a special reward mechanism is designed to promote diversity among experts. By penalizing the parameter similarity between experts, experts are encouraged to learn different features and functions, thereby improving the robustness and expressiveness of the model. At the same time, in strategies such as sparse activation or probabilistic activation, only using the activated expert model for calculation will cause some resources of the loaded expert model to be idle and fail to fully utilize the resources. On the basis of the conventional hybrid expert model, this embodiment also incorporates the calculation results of unactivated experts (i.e., the use of the maximum loaded expert network) into the reward, so that the parameter update process fully utilizes the computing resources and improves the utilization rate of computing resources.

[0142] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0143] Based on the same inventive concept, the embodiment of the present application also provides a hybrid expert model training device for implementing the hybrid expert model training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more hybrid expert models provided below can refer to the limitations of the hybrid expert model training method above, and will not be repeated here.

[0144] In an exemplary embodiment, Figure 4 As shown, a training device for a hybrid expert model is provided, comprising: a sample acquisition module 401, a model input module 402, a reward acquisition module 403 and a model training module 404, wherein:

[0145] The sample acquisition module 401 is used to acquire a training sample set of the hybrid expert model associated with the large model input; the training sample set includes the input data of the large model and the input parsing label information corresponding to the input data;

[0146] The model input module 402 is used to input the target sample in the training sample set into the hybrid expert model to obtain the activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model and the sub-output result of the loaded expert model; the loaded expert model represents a preset number of expert models in the hybrid expert model;

[0147] A reward acquisition module 403 is used to obtain expert differentiation reward information according to the model parameters, target samples, target output results and sub-output results of each expert model;

[0148] The model training module 404 is used to iteratively train the hybrid expert model according to the activation probability information, the expert distinction reward information, the target sample and the target output result to obtain a trained hybrid expert model.

[0149] In one embodiment, the above-mentioned reward acquisition module 403 is also used to determine model diversity reward information based on the model parameters of each expert model; determine output diversity reward information based on the sub-output results; determine input-output difference reward information based on the target sample and the target output result; determine expert differentiation reward information based on the model diversity reward information, output diversity reward information and input-output difference reward information.

[0150] In one embodiment, the reward acquisition module 403 is further used to determine parameter difference information between two expert models according to the model parameters of each expert model; and determine model diversity reward information according to the parameter difference information.

[0151] In one embodiment, the reward acquisition module 403 is further used to determine output difference information between two sub-output results according to the sub-output results; and determine output diversity reward information according to the output difference information.

[0152] In one embodiment, the above-mentioned reward acquisition module 403 is also used to determine the sample difference information and output result difference information between each target sample based on the target sample and the target output result; determine the discrimination information between each target sample based on the sample difference information and the output result difference information; and determine the input-output difference reward information based on the discrimination information.

[0153] In one embodiment, the hybrid expert model includes a gated network model and multiple expert models; the above-mentioned model training module 404 is also used to determine a gradient product term of the parameter update gradient according to a preset baseline function and expert distinguishing reward information in each iterative training; update the parameters of the gated network model according to the activation probability information and the gradient product term, and update the parameters of the target expert model according to the target sample, the target output result and the gradient product term; the target expert model is an expert model determined from multiple expert models according to the activation probability information of this iterative training.

[0154] In one embodiment, the above-mentioned model training module 404 is also used to obtain historical expert differentiation reward information corresponding to the iterative training before the current iterative training; determine the moving average of the reward information based on the historical expert differentiation reward information and the expert differentiation reward information; and use the difference between the expert differentiation reward information and the moving average of the reward information as the gradient product term.

[0155] Each module in the training device of the hybrid expert model can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0156] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 5As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a training method for a hybrid expert model is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0157] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0158] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0159] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0160] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0161] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0162] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0163] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0164] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a hybrid expert model, characterized in that: The method comprises: Acquire a training sample set of a hybrid expert model associated with a large model input; the training sample set includes input data of the large model and input parsing label information corresponding to the input data; Inputting the target sample in the training sample set into the hybrid expert model, obtaining activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model and the sub-output result of the loaded expert model; the loaded expert model represents a preset number of expert models in the hybrid expert model; Obtaining expert distinction reward information according to the model parameters of each expert model, the target sample, the target output result, and the sub-output result; The hybrid expert model is iteratively trained according to the activation probability information, the expert distinction reward information, the target sample and the target output result to obtain a trained hybrid expert model.

2. The method according to claim 1, characterized in that The obtaining of expert distinction reward information according to the model parameters of each expert model, the target sample, the target output result and the sub-output result comprises: Determining model diversity reward information according to the model parameters of each expert model; Determining output diversity reward information according to the sub-output results; Determining input-output difference reward information according to the target sample and the target output result; The expert differentiation reward information is determined according to the model diversity reward information, the output diversity reward information and the input-output difference reward information.

3. The method according to claim 2, characterized in that The step of determining model diversity reward information according to the model parameters of each expert model includes: Determining parameter difference information between two expert models according to the model parameters of each expert model; The model diversity reward information is determined according to the parameter difference information.

4. The method according to claim 2, characterized in that: The step of determining output diversity reward information according to the sub-output results includes: Determining output difference information between two of the sub-output results according to the sub-output results; The output diversity reward information is determined according to the output difference information.

5. The method according to claim 2, characterized in that: The step of determining input-output difference reward information according to the target sample and the target output result includes: Determining sample difference information and output result difference information between each pair of the target samples according to the target samples and the target output results; Determining discrimination information between each pair of the target samples according to the sample difference information and the output result difference information; The input-output difference reward information is determined according to the discrimination information.

6. The method according to claim 1, characterized in that The hybrid expert model includes a gated network model and multiple expert models; The iterative training of the hybrid expert model according to the activation probability information, the expert distinguishing reward information, the target sample and the target output result includes: In each iterative training, a gradient product term of the parameter update gradient is determined according to a preset baseline function and the expert distinguishing reward information; According to the activation probability information and the gradient product term, the parameters of the gated network model are updated, and according to the target sample, the target output result and the gradient product term, the parameters of the target expert model are updated; the target expert model is an expert model determined from the multiple expert models according to the activation probability information of this iterative training.

7. The method according to claim 6, characterized in that The step of determining a gradient product term of the parameter update gradient according to the preset baseline function and the expert distinguishing reward information comprises: Obtain historical expert distinction reward information corresponding to the iterative training before the current iterative training; Determine a moving average of reward information based on the historical expert differentiated reward information and the expert differentiated reward information; The difference between the expert distinguishing reward information and the moving average of the reward information is used as the gradient product term.

8. A training device for a hybrid expert model, characterized in that: The device comprises: A sample acquisition module, used to acquire a training sample set of a hybrid expert model associated with a large model input; the training sample set includes input data of the large model and input parsing label information corresponding to the input data; A model input module, used to input the target sample in the training sample set into the hybrid expert model, and obtain the activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model and the sub-output result of the loaded expert model; the loaded expert model represents a preset number of expert models in the hybrid expert model; A reward acquisition module, used to obtain expert distinction reward information according to the model parameters of each expert model, the target sample, the target output result and the sub-output result; The model training module is used to iteratively train the hybrid expert model according to the activation probability information, the expert distinction reward information, the target sample and the target output result to obtain a trained hybrid expert model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images