Data processing method and device based on hybrid expert large model

Abstract

The invention discloses a data processing method and device based on a hybrid expert large model, and relates to the field of hybrid expert large models, the hybrid expert large model is called to process a target question text input by a user to obtain a target answer, and the hybrid expert large model comprises at least one sub-model group, each sub-model group corresponds to different target application fields, and each sub-model group comprises at least one sub-model; the gating weight matrixes correspond to the sub-models respectively, the gating weight matrixes are obtained by processing supervised data corresponding to the sub-models in a pre-trained question and answer model, and dot products of the gating weight matrixes and feature matrixes of question texts in training samples are obtained; the feature representation module is used for representing the similarity between the data features of the supervised data and the data features of the problem text; the weight parameters of the FNN layers are different. According to the method, the sub-model suitable for the question can be selected, and the sub-model is more focused on processing the question of the corresponding application field, so that the accuracy of the target answer is improved.

Description

Technical Field

[0001] The present application relates to the technical fields of large models, large model training, and hybrid expert large models, and more specifically, to a data processing method and device based on a hybrid expert large model. Background Art

[0002] Large language models have attracted much attention due to their powerful natural language processing and understanding capabilities. During the construction and training of large language model networks, the model performance and natural language generation quality of generative large language models are proportional to the parameter magnitude and training data of the large model, that is, a large language model with a larger parameter magnitude and more high-quality data for training can produce a better large language model. However, as the network structure of large language models becomes deeper and the number of parameters increases, the amount of training data and training costs required for large models also increase.

[0003] In order to solve the problem of how to achieve better training results under limited computing resources, Mixed Expert Models (MoEs) are proposed. Mixed expert models do not achieve better results by increasing the parameter magnitude of large models, but by combining multiple large models (experts) with small parameter magnitudes to achieve the effect of large parameter magnitudes, greatly reducing the cost required for training and reasoning. However, the existing large mixed expert models cannot handle problems with strong domain specificity, resulting in inaccurate data processing results of large mixed expert models that cannot meet the user's question-answering needs. Summary of the invention

[0004] In view of the above problems, the present application provides a data processing method and device based on a hybrid expert large model to solve the problem that the data processing results of the hybrid expert large model are not accurate enough. The specific solution is as follows: The first aspect of the present application provides a data processing method based on a hybrid expert large model, comprising: Get the target question text sent by the user; Calling a pre-built hybrid expert model to process the target question text to obtain a target answer, the pre-built hybrid expert model includes: at least one sub-model group, each sub-model group corresponds to a different target application field, and the sub-model group contains at least one sub-model; gating weight matrices corresponding to the sub-models, the gating weight matrices are obtained by processing the supervised data corresponding to the sub-model in a pre-trained question-answering model, and the dot product of the gating weight matrix and the feature matrix of the question text in the training sample is used to characterize the similarity between the data features of the supervised data and the data features of the question text; at least one FNN layer corresponding to each of the sub-models, and the weight parameters of each of the FNN layers are different; The target answer is sent to the user terminal.

[0005] In a possible implementation, the construction process of the pre-built hybrid expert large model includes: Obtain model building request information, the model building request information including: a pre-trained question-answering model, at least one target application field, a sub-model group corresponding to each target application field, the number of sub-models in the sub-model group, and supervised data corresponding to each sub-model, wherein the pre-trained question-answering model is a trained large-scale language model; The supervised data corresponding to each of the sub-models is respectively input into the pre-trained question-answering model for processing, so as to obtain a matrix corresponding to each of the sub-models representing the data features of the supervised data as a gating weight matrix corresponding to each of the sub-models; Constructing an FNN layer to obtain an initial FNN layer corresponding to each of the sub-models, and the initial weight parameters of each of the initial FNN layers are equal; The parameters in the initial FNN layer are adjusted to obtain a target FNN layer, wherein the weight parameters in each target FNN layer are different from each other, and the process of adjusting the parameters in the initial FNN layer at least comprises: adding noise data to the initial weight parameters of each initial FNN layer; A model is constructed based on the pre-trained question-answering model, the gating weight matrix corresponding to each of the sub-models, and the target FNN layer to obtain the hybrid expert large model.

[0006] In a possible implementation, the constructing a model based on the pre-trained question-answering model, the gating weight matrix corresponding to each of the sub-models, and the target FNN layer to obtain the hybrid expert large model includes: Copying the backbone network layer in the pre-trained question-answering model, the backbone network layer at least includes: an attention layer in a transformer decoder module; Constructing a gated network layer based on the gated weight matrix corresponding to each sub-model; Based on the target FNN layer corresponding to each sub-model, a sparse MoE layer is constructed; The backbone network layer, the gated network layer and the sparse MoE layer are combined to obtain a hybrid expert large model.

[0007] In a possible implementation, obtaining supervised data corresponding to each sub-model includes: Obtain supervised data samples corresponding to each target application field; Dividing the supervised data samples corresponding to the target application field to obtain general data in the field and non-general data in the field corresponding to each target application field; Based on the number of sub-models in the sub-model group corresponding to the target application field, the non-common data in this field is divided into a number of parts equal to the number of sub-models, to obtain the non-common data in this field corresponding to each sub-model of the target application field; The general data in the field corresponding to other target application fields except the target application field shall be used as the general data in other fields corresponding to the target application field; Based on the general data in the field and the general data in other fields corresponding to the target application field, and the non-general data in the field corresponding to each sub-model, the supervised data corresponding to each sub-model is obtained.

[0008] In a possible implementation, the supervised data corresponding to each of the sub-models is respectively input into the pre-trained question-answering model for processing, and a matrix representing data features of the supervised data corresponding to each of the sub-models is obtained as a gating weight matrix corresponding to each of the sub-models, including: The supervised data corresponding to each sub-model is respectively input into the pre-trained question-answering model for processing, so as to obtain a first feature matrix corresponding to each sub-model representing data features of the common data in the field, a second feature matrix representing data features of the non-common data in the field, and a third feature matrix representing data features of the common data in other fields; Subtract the third feature matrix from the sum of the first feature matrix and the second feature matrix to obtain a feature matrix corresponding to each of the sub-models; Regularization is performed on the feature matrix corresponding to each of the sub-models to obtain a gating weight matrix corresponding to each of the sub-models.

[0009] In a possible implementation, constructing an FNN layer to obtain an initial FNN layer corresponding to each of the sub-models includes: The FNN layer in the pre-trained question-answering model is copied to obtain an initial FNN layer corresponding to each of the experts, and the initial weight parameters of each of the initial FNN layers are equal.

[0010] In a possible implementation, adding noise data to the initial weight parameters of each initial FNN layer includes: Acquire a preset number of noise distributions, wherein the distribution of each of the noise distributions is different from each other; Noise sampling is performed from the noise distribution to obtain noise samples of the same number as the sub-model, and each of the noise samples is different; Adding the noise samples to the weight parameters of the initial FNN layers respectively to obtain target weight parameters corresponding to each of the initial FNN layers; The weight parameters of the initial FNN layer are updated with the target weight parameters to obtain a target FNN layer.

[0011] In a possible implementation, sampling noise from the noise distribution to obtain noise samples having a number equal to that of the sub-models, and each of the noise samples is different, includes: When the number of the noise distribution is 1, extracting at least one noise segment from the noise distribution as a reference noise; The reference noise is multiplied by different coefficients respectively to obtain noise samples of the same number as the sub-model, and each of the noise samples is different.

[0012] In a possible implementation, the process of adjusting the parameters in the initial FNN layer further includes: Based on the supervised data corresponding to each of the sub-models, the hybrid expert model is fine-tuned in a supervised manner to obtain target FNN layers with different weight parameters; and / or, Obtaining general supervised data common to all application fields; and performing supervised fine-tuning on the hybrid expert large model based on the general supervised data to obtain the target FNN layer with different weight parameters.

[0013] In a possible implementation, the process of the hybrid expert large model processing the target question text includes: The attention layer extracts the features of the target question text to obtain an input feature matrix; Performing dot products on the input feature matrix and each of the gated weight matrices to obtain dot product results corresponding to each of the gated weight matrices; Normalizing the dot product result to obtain the gate selection probability corresponding to each sub-model; Based on the gating selection probability corresponding to each of the sub-models, a sub-model whose corresponding gating selection probability meets a preset condition is determined as a target sub-model, wherein the preset condition at least includes: the gating selection probability reaches a preset threshold, or the gating selection probability is a pre-set number of gating selection probabilities in a probability sequence, and the probability sequence is a sequence of gating selection probabilities arranged in numerical order; The target question text is input into the target FNN layer corresponding to the target sub-model for processing to obtain the target answer.

[0014] The second aspect of the present application provides a data processing device based on a hybrid expert large model, comprising: A text acquisition unit, used to acquire the target question text sent by the user end; A text processing unit, used for calling a pre-built hybrid expert large model to process the target question text to obtain a target answer, wherein the pre-built hybrid expert large model includes: at least one sub-model group, each sub-model group corresponds to a different target application field, and the sub-model group includes at least one sub-model; a gating weight matrix corresponding to each sub-model, wherein the gating weight matrix is ​​obtained by processing the supervised data corresponding to the sub-model in a pre-trained question-answering model, and the dot product of the gating weight matrix and the feature matrix of the question text in the training sample is used to characterize the similarity between the data features of the supervised data and the data features of the question text; at least one FNN layer corresponding to each of the sub-models, and the weight parameters of each of the FNN layers are different; An answer sending unit is used to send the target answer to the user terminal.

[0015] It can be seen from the above technical solution that in the hybrid expert large model pre-constructed in the embodiment of the present application, the data features of the supervised data of each application field are learned in the form of a weight matrix, so that the dot product of the gated weight matrix corresponding to each sub-model (expert) and the feature matrix of the question text can characterize the similarity between the data features of the supervised data corresponding to the sub-model and the data features of the question text. The higher the similarity, the more suitable the corresponding sub-model is for processing the target question text. Therefore, the hybrid expert large model constructed based on the gated weight matrix can improve the accuracy of selecting the sub-model. In addition, the weight parameters of the FNN layer of each expert in the hybrid expert large model are different, which can reduce the ability similarity between each sub-model.

[0016] Based on this, this application uses a pre-built hybrid expert large model to process the target question text, and can accurately select a sub-model suitable for the application field to which the target question text belongs. After the similarity of each sub-model is reduced, each sub-model will be more focused on processing the problems in the corresponding application field, so that the obtained target answer text is more biased towards the application field to which the target question text belongs, thereby improving the accuracy of the target answer text. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0018] Figure 1 A flow chart of a data processing method based on a hybrid expert large model provided in an embodiment of the present application; Figure 2 An example diagram of an application hybrid expert model provided in an embodiment of the present application; Figure 3 A schematic diagram of a process for constructing a hybrid expert large model provided in an embodiment of the present application; Figure 4 A schematic diagram of dividing supervised data provided in an embodiment of the present application; Figure 5 A schematic diagram of constructing a hybrid expert large model based on supervised data provided in an embodiment of the present application; Figure 6 A structural schematic diagram of a data processing device based on a hybrid expert large model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The hybrid expert model usually consists of a gating network and a number of experts. It aims to integrate multiple experts in multiple fields to solve various user problems in a collaborative way. Among them, the "expert" is an independent neural network module with specific structure and parameters. The gating network of the hybrid expert model is similar to a selector. When the user enters the target question text, the hybrid expert model will select multiple experts who are good at solving problems in this field through the gating network for the target question text, and delegate at least one of the experts to perform reasoning calculations on the target question text. The expert's calculation results are weighted and output as the reasoning results of the hybrid expert model to solve the user's problems in a collaborative way.

[0021] In the prior art, there is a problem in which the experts in the hybrid expert model have similar abilities and poor balanced selectivity, which results in the hybrid expert model being unable to accurately select experts suitable for solving user input problems, making it impossible for experts to give accurate answers to user input questions.

[0022] To solve the above problems, the present application provides a data processing method based on a hybrid expert large model. Figure 1 , a flow chart of a data processing method based on a hybrid expert large model provided in an embodiment of the present application specifically includes the following steps: Step S110, obtaining the target question text sent by the user terminal; Step S120, calling the pre-built hybrid expert large model to process the target question text to obtain the target answer; Step S130, sending the target answer to the user end.

[0023] Reference Figure 2 The example diagram of applying the hybrid expert big model provided in the embodiment of the present application exemplifies the data processing process of the hybrid expert big model, wherein the diagram shows the application page of the hybrid expert big model opened by the user on the user end. The user can enter the target question text to ask the hybrid expert big model in the text input box, i.e., the box of "Please enter the question...", such as "What are the types of databases?"

[0024] Furthermore, the hybrid expert model will select experts in the same application field as the target question text, process the target question text, and obtain a target answer for the target question text. In an embodiment of the present application, the pre-constructed hybrid expert model includes: at least one sub-model group, each sub-model group corresponds to a different target application field, and the sub-model group contains at least one sub-model; a gating weight matrix corresponding to each sub-model, the gating weight matrix is ​​obtained by processing the supervised data corresponding to the sub-model in the pre-trained question-answering model, and the dot product of the gating weight matrix and the feature matrix of the question text in the training sample is used to characterize the similarity between the data features of the supervised data and the data features of the question text; at least one FNN layer corresponding to each sub-model, and the weight parameters of each FNN layer are different.

[0025] Based on this, the hybrid expert model has at least one sub-model corresponding to each application field, and a gating weight matrix corresponding to each sub-model, and the sub-model is composed of at least one FNN layer. Among them, a sub-model represents an expert, and the gating weight matrix corresponding to each sub-model constitutes the gating network (gating network layer) of the hybrid expert model.

[0026] After receiving the target question text, the gated network layer selects the corresponding sub-model to process the target question text based on the dot product result of the feature matrix of the target question text and each gated weight matrix, reflecting the similarity between the data features of the target question text and the data features corresponding to each gated weight matrix. Referring to the process of building the hybrid expert model below, it can be seen that the gated weight matrix is ​​the feature learning result of the supervised data used by experts in the application field. Therefore, the higher the similarity reflected by the dot product result, the higher the similarity between the features of the target question text and the features of the supervised data of the corresponding expert, and the more suitable the expert (sub-model) is for solving the problem of the target question text.

[0027] The target sub-model selected as above that is most suitable for solving the problem of the target question text processes the target question text. In addition, considering that each sub-model is constructed by the same FNN layer, there may be a situation where the similarity of each sub-model is too high, resulting in similar output results when different sub-models process the same problem. The high similarity and similar capabilities of each sub-model will significantly affect the overall performance of the large mixed expert model (MoE). Even after further supervised fine-tuning, this similarity will still limit the potential of the model. When the gated network layer selects a specific number of sub-models to process the input task, the selected sub-models often give highly similar prediction results. Since these prediction results are relatively small, the loss values ​​of each sub-model tend to be similar when calculating the loss with the sample labels of the supervised data. This phenomenon further leads to the fact that the gradient update direction of each sub-model is almost the same during the back-propagation process of model training, and each sub-model cannot effectively generate diversified parameter adjustments. As a result, before and after fine-tuning, the sub-models still have high similarities and similar capabilities, and the division of labor and complementarity among multiple experts still cannot be effectively exerted, resulting in a significant reduction in the model's learning results and overall performance.

[0028] Therefore, in the hybrid large model of the embodiment of the present application, the weight parameters of the FNN layer of each sub-model are different, which increases the difference in weights between the sub-models and solves the problem of similar sub-model capabilities, so that each sub-model can focus on processing the problems in the corresponding application field, such as Figure 2 For example, the selected sub-model focuses more on answering questions about databases in the computer field, and the output is such as "Database types may include: relational databases, non-relational databases...", so that the answers obtained by the final hybrid expert model are more accurate and the model answering effect is better.

[0029] In summary, in the hybrid expert large model pre-constructed in the embodiment of the present application, the data features of the supervised data of each application field are learned in the form of a weight matrix, so that the dot product of the gated weight matrix corresponding to each sub-model (expert) and the feature matrix of the question text can characterize the similarity between the data features of the supervised data corresponding to the sub-model and the data features of the question text. The higher the similarity, the more suitable the corresponding sub-model is for processing the target question text. Therefore, the hybrid expert large model constructed based on the gated weight matrix can improve the accuracy of selecting the sub-model. In addition, the weight parameters of the FNN layer of each expert in the hybrid expert large model are different, which can reduce the ability similarity between each sub-model.

[0030] Based on this, this application uses a pre-built hybrid expert large model to process the target question text, and can accurately select a sub-model suitable for the application field to which the target question text belongs. After the similarity of each sub-model is reduced, each sub-model will be more focused on processing the problems in the corresponding application field, so that the obtained target answer text is more biased towards the application field to which the target question text belongs, thereby improving the accuracy of the target answer text.

[0031] Next, other possible implementations of the data processing method based on the hybrid expert large model provided in the embodiment of the present application are described.

[0032] Reference Figure 3 , a flow chart of constructing a hybrid expert big model provided in an embodiment of the present application, which illustrates the construction process of the hybrid expert big model pre-constructed in an embodiment of the present application, and the specific process includes: Step S210, obtaining model building request information, the model building request information includes: a pre-trained question-answering model, at least one target application field, a sub-model group corresponding to each target application field, the number of sub-models in the sub-model group, and supervised data corresponding to each sub-model.

[0033] Among them, the pre-trained question-answering model is a large-scale language model that has been trained, such as a basic model based on deep neural networks and self-supervised learning technology, which is trained on a large-scale, widely sourced data set. Specifically, it can be a GPT series (GPT-3, GPT-4, etc.), qwen-7b, llama-7b and other models. Usually, the pre-trained question-answering model needs to learn the characteristics of each application field and be able to answer questions in each application field. The purpose of obtaining the pre-trained question-answering model in the embodiment of the present application is to establish a large hybrid expert model based on the training results of this model, reduce the amount of training for building the large hybrid expert model, and improve the efficiency of model building.

[0034] The data processing method based on the hybrid expert big model can be applied to terminals, servers, and other devices with model training, data storage, and calling functions. Users can send model building requests for the hybrid expert big model to the device, for example, to build a hybrid expert big model for solving the two fields of law and medicine.

[0035] Furthermore, the device obtains model building request information corresponding to the model building request from the user end, database, etc. For example, the medical field and the legal field are two target application fields, respectively, and the model building request information includes: the number of sub-models that need to be built in the medical field and the legal field, respectively, and the supervised data used for training each sub-model.

[0036] It can be understood that the number of sub-models corresponding to each field can be customized by the user. Each sub-model represents an expert who solves the application field. The more experts there are, the more subdivided the problems in the target application field can be. For example, the medical field includes sub-fields such as cardiology, cardiac surgery, orthopedics, and gynecology. If the number of sub-models in the medical field is large enough, each sub-model can be trained towards a sub-field to reduce the similarity of sub-models in the same field.

[0037] The supervised data used for training each sub-model can be obtained by collecting relevant data of the target application field corresponding to the sub-model, or obtained from the user end and provided by the user. Among them, the supervised data corresponding to each sub-model must belong to the data of the target application field corresponding to the sub-model, and in order to distinguish each sub-model, the supervised data between each sub-model in the same application field also needs to be different.

[0038] Based on this, in an embodiment of the present application, the process of obtaining supervised data corresponding to each sub-model may include: obtaining supervised data samples corresponding to each target application field; dividing the supervised data samples corresponding to the target application field to obtain general data in the field and non-general data in the field corresponding to each target application field; based on the number of sub-models in the sub-model group corresponding to the target application field, dividing the non-general data in the field into a number equal to the number of sub-models to obtain the non-general data in the field corresponding to each sub-model in the target application field; using the general data in the field corresponding to other target application fields except the target application field as the general data in other fields corresponding to the target application field; based on the general data in the field and the general data in other fields corresponding to the target application field, as well as the non-general data in the field corresponding to each sub-model, obtain the supervised data corresponding to each sub-model.

[0039] Optionally, the number of experts (the number of sub-models) to process each application field is determined based on the specific application field of the hybrid expert model and the total number of experts used to construct the hybrid expert model. All the obtained supervised data samples are divided according to different application fields to obtain supervised data samples corresponding to each application field. Furthermore, the supervised data samples corresponding to each application field are divided into fields, for example, common prompt words and key characters in the application field are extracted as common data in this field, and non-common prompt words and key characters in the application field, such as prompt words for a specific field in the application field, are extracted as non-common data in this field.

[0040] Considering the need to learn the characteristics of supervised data samples from other application fields, we can exclude the problems belonging to other application fields and not process them, thereby improving the accuracy of the gating network selection sub-model. For a target application field, the corresponding supervised data samples include at least: general data in this field, non-general data in this field, and general data in other fields.

[0041] In addition, the embodiment of the present application distributes the non-common data in this field equally to each sub-model in this field, and improves the difference of each sub-model in the same field and the gating weight matrix by learning the non-common data.

[0042] Reference Figure 4 , the schematic diagram of dividing supervised data provided in the embodiment of the present application illustrates the above-mentioned supervised data division process. Figure 4 In the data, a total of 180 supervised scenario data (supervised data) were obtained. According to the application field described in each supervised data, they were divided into 60 medical scenario data in the medical field and 120 legal scenario data in the legal field.

[0043] Furthermore, general medical data are extracted from the medical scenario data, for example, 20 general medical data such as patient medical records, X-ray shooting principles, and medical equipment data, and 40 non-general medical data, such as more specific disease problems in each department, specifically, difficult and complicated diseases in the ENT department, gastroenterology problems, cardiovascular diseases, maternal and child health care, etc. In this example, the general data for other fields in the medical field are general legal data.

[0044] In this example, Figure 4 An example of a hybrid expert model consisting of four experts is provided, with two experts corresponding to each application field. Therefore, the 40 non-generic medical data and 80 non-generic legal data are divided into two equal parts and used for model construction of each expert sub-model.

[0045] In step S220, the supervised data corresponding to each sub-model is input into the pre-trained question-answering model for processing, and a matrix representing the data features of the supervised data corresponding to each sub-model is obtained as the gating weight matrix corresponding to each sub-model.

[0046] Using the supervised data corresponding to each sub-model, the gated weight matrix corresponding to each sub-model in the gated network layer is initialized. First, the supervised data corresponding to each sub-model is processed by the pre-trained question-answering model. The processing process can be understood as extracting the data features in the supervised data, allowing the gated network layer to learn different types of data representations in advance, and using the data representation as the gated weight matrix. When the gated network layer is applied, when a certain gated weight matrix encounters similar features, the dot product result with the matrix of the feature can maximize the similar features. Using this principle, the feature matrix of the target question text is dot-producted with each gated weight matrix respectively. The gated weight matrix corresponding to the dot product result that can maximize the similarity is the target gated matrix, and the sub-model corresponding to the target gated matrix is ​​the target sub-model that solves the target question text.

[0047] In one possible implementation, the process of determining the gating weight matrix in step S220 may include: inputting the supervised data corresponding to each sub-model into the pre-trained question-answering model for processing, to obtain a first feature matrix corresponding to each sub-model that characterizes the data features of common data in this field, a second feature matrix that characterizes the data features of non-common data in this field, and a third feature matrix that characterizes the data features of common data in other fields; subtracting the third feature matrix from the sum of the first feature matrix and the second feature matrix to obtain the feature matrix corresponding to each sub-model; and performing regularization processing on the feature matrix corresponding to each sub-model to obtain the gating weight matrix corresponding to each sub-model.

[0048] The attention layer in the transformer decoder module of the pre-trained question-answering model is used to process the common data in the field, the non-common data in the field, and the common data in other fields in the supervised data of the sub-model to obtain the corresponding "hidden states", namely the first feature matrix, the second feature matrix, and the third feature matrix. It can be understood that most of the supervised data are text materials, and the "hidden state" is the feature of the model data state of the text materials extracted by the attention layer.

[0049] Furthermore, the hidden states of "general data in this field" and "non-general data in this field" are added together, so that the sub-model corresponding to the gated network learns the general features of the target application field to which it belongs, as well as the features of the specific non-general problems that the sub-model should solve. Furthermore, the hidden state of "general data in other fields" is subtracted to learn the data representation of other application fields that do not belong to the target application field, so that when receiving the target problem text belonging to other target application fields, the similarity with the target text problem is reduced to avoid being selected to process the target problem text. The result of subtracting the hidden state of "general data in other fields" is L2 regularized to obtain the final matrix result as the gated weight matrix.

[0050] Reference Figure 5 , a schematic diagram of constructing a hybrid expert large model based on supervised data provided in an embodiment of the present application, wherein Figure 4 The supervised data division results are as follows: for the medical field, there are 2 experts in the medical field, namely E1 and E2, and the corresponding gating weight matrices in the gating network are M1 and M2; for the legal field, there are also 2 experts in the legal field, namely E3 and E4, and the corresponding gating weight matrices in the gating network are M3 and M4.

[0051] Taking the initialization of the gated weight matrix in the medical field as an example, through the above-mentioned supervised data division method, 40 non-generic medical data are equally divided according to the number of experts, then each expert obtains 20 non-generic medical data, and the supervised data corresponding to each medical expert include: 20 generic medical data, 20 non-generic medical data and 40 generic legal data, where the amount of data for each item can be customized according to the model requirements. For example, in order to enhance the domain of the medical model, all data in the legal field are learned to avoid assigning legal issues to medical experts; and in order to enhance the generalization ability of the medical model and broaden the field of questions that the medical model can answer, the number of generic data in the legal field is reduced.

[0052] Assign 20 pieces of general medical data, a set of 20 pieces of non-general medical data, and 5 pieces of "general data from other scenarios" from the legal field to M1. Perform the same operation on M2 using another set of non-general medical domain data to initialize the weights of matrices M1 and M2. Figure 5 As shown, matrices M3 and M4 are initialized in the same way. Finally, the gating weight matrix corresponding to each expert is obtained.

[0053] For the data-driven gated network layer constructed based on the above method, when a question belonging to the medical field is input, the gated network layer can select experts E1 and E2 who can handle the medical field to the greatest extent to handle the question. This is because the data belonging to the medical field is learned by matrices M1 and M2 through hidden state addition, and the data that does not belong to the medical field is also learned by hidden state subtraction. When the problem is similar to the representation of matrices M1 and M2, the gated network layer can maximize the dot product result of M1 and M2 matrices and the problem, and select E1 and E2 to handle the user's problem through the dot product result of M1 and M2.

[0054] Based on this, the embodiment of the present application ensures that the matrices in the same field are initialized with specific non-universal data for weights through reasonable supervised data splitting for different application fields. When inputting problems related to the field, because the data features are similar, the dot product of the matrices in the same field can be larger than the dot product of the matrices in other fields, and finally experts in the same field are selected to handle the problem.

[0055] Therefore, the gated network layer constructed by the above-mentioned gating weight matrix can understand the fields and problems that each expert sub-model is adapted to. The gated network layer can effectively delegate problems in different fields to different experts for processing, ensuring that the experts who are most suitable for solving the problems can be selected.

[0056] Step S230, constructing the FNN layer to obtain the initial FNN layer corresponding to each sub-model, and the initial weight parameters of each initial FNN layer are equal.

[0057] Step S240, adjusting the parameters in the initial FNN layer to obtain a target FNN layer, wherein the weight parameters in each target FNN layer are different from each other.

[0058] The embodiment of the present application obtains the initial FNN layer corresponding to each sub-model by copying the FNN layer in the same pre-trained question-answering model, and the initial weight parameters of each initial FNN layer are equal. It can be understood that since it is a copied FNN layer, the parameters of each FNN layer are the same, and the similarity between each sub-model is extremely high, resulting in similar results for different experts to handle the same problem.

[0059] To solve the above problem, the embodiment of the present application adjusts each initial FNN layer to change the parameters of each initial FNN layer and increase the difference between each FNN layer, wherein the modified parameters include at least: weight parameters.

[0060] In a possible implementation, the weight parameters of each FNN layer are adjusted by adding noise to the weight parameters, so that the weight parameters of each target FNN after adjustment are different from each other. The process of adding noise may include: obtaining a preset number of noise distributions, and the distribution of each noise distribution is different from each other; sampling noise from the noise distribution to obtain a number of noise samples equal to the number of sub-models, and each noise sample is different; adding the noise samples to the weight parameters of the initial FNN layer respectively to obtain the target weight parameters corresponding to each initial FNN layer; updating the weight parameters of the initial FNN layer with the target weight parameters to obtain the target FNN layer.

[0061] In the embodiment of the present application, the preset number may be the same as the number of sub-model groups. For example, the hybrid expert large model needs to construct expert groups (i.e., sub-model groups) in four fields, and then obtains 4 different noise distributions, as follows: Noise distribution 1: normal distribution with mean 0.0 and standard deviation 0.1; Noise distribution 2: normal distribution with mean 1.0 and standard deviation 0.2; Noise distribution 3: normal distribution with mean 2.0 and standard deviation 0.3; Noise distribution 4: normal distribution with mean 3.0 and standard deviation 0.4; In this case, different noises can be obtained by sampling from different noise distributions and added to the weight parameters of the initial FNN layers of different sub-models respectively, so as to achieve parameter adjustment of the initial FNN layer.

[0062] In another possible implementation, when the preset number is 1, that is, when the number of noise distributions is 1, at least one section of noise is extracted from the noise distribution as reference noise; the reference noise is multiplied with different coefficients respectively to obtain noise samples of the same number as the sub-model, and each noise sample is different.

[0063] For example, the hybrid expert model obtains 1 noise distribution, namely, noise distribution 1: a normal distribution with a mean of 0.0 and a standard deviation of 1. A noise tensor tensor is extracted from the noise distribution as the reference noise. The reference noise is multiplied by a coefficient noise_scale, where the coefficient noise_scale is a defined variable that represents the scale of the noise. The reference noise is scaled by multiplying it with the reference noise. Different coefficients result in different scaled noises. Therefore, by multiplying the coefficient noise_scale with the reference noise, the same number of noise samples as the sub-model are obtained, and each noise sample is different from each other.

[0064] Alternatively, different segments of noise tensors are extracted from the noise distribution 1. For example, the noise tensors equal to the total number of sub-models N are extracted, that is, there are N reference noises, and the N reference noises are multiplied by N different coefficients noise_scale to obtain N different noise samples. Each noise sample is added to the weight parameter of the initial FNN layer of the corresponding sub-model to obtain the target FNN layer.

[0065] In addition, when adding noise, the noise can be adjusted according to the application field to which each expert belongs. Optionally, different experts in the same field are assigned noise values ​​with different but similar values, and experts in different fields are assigned noise values ​​with greater discrimination. For example, the noise added to experts (sub-models) in the same field is sampled from the same noise distribution, so that the noise added to the FNN layer of experts in the same field is similar but different, and the noise difference between experts in different fields is large. Based on this, it is possible to ensure that the weights of experts in different fields are different, and the weights of experts in the same field are similar, but still have some degree of difference. Ultimately, the model can handle various field problems more effectively, thereby improving the performance of its hybrid expert model.

[0066] In another possible implementation, the adjustment process of adjusting the parameters in the initial FNN layer also includes: based on the supervised data corresponding to each sub-model, performing supervised fine-tuning on the hybrid expert large model to obtain a target FNN layer with different weight parameters; and / or, obtaining general supervised data that is common to all application fields; based on the general supervised data, performing supervised fine-tuning on the hybrid expert large model to obtain a target FNN layer with different weight parameters.

[0067] Through supervised SFT fine-tuning, the ability of each expert in the hybrid expert model to handle domain knowledge problems is improved. A series of knowledge questions and answers of various types in the field and multi-round dialogue data (i.e., "supervised data") are used to supervised fine-tune the hybrid expert model. Through supervised fine-tuning, various parameters in the hybrid expert large model are adjusted, and the parameters of the FNN layer are also fine-tuned, so that the sub-model can learn the problem characteristics and answer content characteristics of the refined field, making it more domain-specific.

[0068] Optionally, based on the above fine-tuning method, all supervised data are collected and shuffled, and the entire hybrid expert model is fine-tuned as a whole. The supervised data is processed based on the fine-tuned hybrid expert model to obtain the output answer, and based on the accurate answer corresponding to the supervised data, the loss of the output answer is calculated, and the parameters of the hybrid expert model are updated based on the loss value until the loss value meets the training requirements, and the parameter update of the hybrid expert model is stopped.

[0069] In practical applications, the gated network layer may delegate the wrong expert to process the target question text, and the delegated expert may not be able to process the target question text well, making it difficult for the hybrid expert model to provide high-quality responses to users.

[0070] Therefore, in order to further improve the generalization ability of the hybrid expert model and the effectiveness of each expert model in dealing with unknown domain problems, the hybrid expert model is fine-tuned in a supervised manner based on general scenario data to increase the fault tolerance of the gating network and the generalization ability of each expert model. Among them, the general supervised data used for supervised fine-tuning will no longer be data in a specific field, but some general knowledge question-answering scenario data, such as today's weather, Dragon Boat Festival date and other common sense questions.

[0071] In addition, compared with the supervised fine-tuning process of the sub-model or the hybrid expert model mentioned above, this process will use a smaller number of training steps and learning rate, so that the hybrid expert model will not significantly update the weights of each expert sub-model and the gated network layer during the training process at this stage. The purpose is to increase the generalization ability of each expert model while retaining the ability of each expert model to handle tasks in different scenarios and fields.

[0072] Optionally, in this process, a data polling strategy can also be adopted. For a batch of data, the data can be allocated to different expert sub-models for training through a polling strategy, so that each data is assigned to a different expert by the gated network layer to ensure that each expert can be trained and each expert has the ability to solve common scenario tasks, thereby improving the generalization ability of the hybrid expert large model.

[0073] Step S250, construct a model based on the pre-trained question-answering model, the gating weight matrix corresponding to each sub-model, and the target FNN layer to obtain a hybrid expert large model.

[0074] In the embodiment of the present application, the hybrid expert model can be constructed based on the pre-trained question-answering model used to construct the hybrid expert model, such as the embedding layer, attention layer and other model layers in the basic model. After obtaining the gating weight matrix for constructing the gated network layer through step S220, the hybrid expert model basically has the ability to select experts. Further, a sub-model corresponding to each expert is constructed in combination with the target FNN layer for processing the problem.

[0075] In one possible implementation, a model is constructed based on a pre-trained question-answering model, a gating weight matrix corresponding to each sub-model, and a target FNN layer to obtain a hybrid expert large model, including: copying the backbone network layer in the pre-trained question-answering model, the backbone network layer at least including: the attention layer in the transformer decoder module; constructing a gated network layer based on the gating weight matrix corresponding to each sub-model; constructing a sparse MoE layer based on the target FNN layer corresponding to each sub-model; combining the gated network layer and the sparse MoE layer to obtain a hybrid expert large model.

[0076] The backbone network layer can also include an input_embedding layer, an output layer, etc. The attention layer is used to extract the features of the target question text as the input of the gated network layer, and perform dot product operations with each gated weight matrix. Based on the dot product operation results, the target sub-model is selected and called from the sparse MoE layer to process the target question text.

[0077] Specifically, the attention layer extracts the features of the target question text to obtain an input feature matrix; the input feature matrix is ​​dot-producted with each gating weight matrix to obtain the dot product results corresponding to each gating weight matrix; the dot product results are normalized to obtain the gating selection probability corresponding to each sub-model; based on the gating selection probability corresponding to each sub-model, the sub-model whose corresponding gating selection probability meets the preset conditions is determined as the target sub-model, and the preset conditions at least include: the gating selection probability reaches a preset threshold, or the gating selection probability is the first preset number of gating selection probabilities in the probability sequence, and the probability sequence is a sequence of gating selection probabilities arranged in numerical order; the target question text is input into the target FNN layer corresponding to the target sub-model for processing to obtain the target answer.

[0078] The gated network layer has gated weight matrices (M1, M2, ..., Mn) corresponding to n experts (E1, E2, ...En). When selecting an expert sub-model, the gated network layer uses the calculation result of the attention layer in the decoder layer on the target question text as the input of the gated network layer. The calculation result is a matrix that represents the data features of the target question text. The matrix is ​​sequentially dot-producted with the gated weight matrices M1, M2, ..., Mn to obtain the dot product result, and each dot product result is normalized by softmax to obtain the probability of the gated network selecting each expert sub-model.

[0079] The sub-model with the highest probability or the probability exceeding the threshold is selected as the target sub-model, or the probability corresponding to each expert sub-model is sorted, and a preset number of sub-models are selected in sequence as the target sub-model. The target question text is input into the target sub-model for processing to obtain the target answer.

[0080] In summary, the above-mentioned hybrid expert big model construction process has many advantages over the existing hybrid expert big model construction method. For example, in the existing hybrid expert model construction method, the hybrid expert model can be constructed by building from scratch or by selecting multiple different types of big models from a preset big model library for combination.

[0081] Regardless of the above methods, the parameter weights of the gated network layer are set by random initialization, and do not have the ability to effectively select and delegate experts in specific fields to handle user-specific fields or application scenario problems. This results in poor balance of the gated network layer of the constructed hybrid expert model and cannot be directly used to solve specific field problems. Further training is required to optimize the parameters of the gated network layer, which has high training costs, and due to the imbalanced selection of the gated network layer during the training process, each expert cannot be fully trained, which affects the accuracy of the final hybrid expert model in handling problems.

[0082] The gated network layer constructed according to the gated weight matrix in the embodiment of the present application understands, learns, and captures the characteristics of the input data of each application field in the form of a weight matrix. When the gated network layer is applied, the gated weight matrix corresponding to the sub-model (expert) in each field is dot-producted with the feature matrix of the question text, which can characterize the similarity between the data characteristics of the supervised data in the field corresponding to the sub-model and the data characteristics of the question text. The higher the similarity, the more the question text belongs to the knowledge problem in the field, and the corresponding sub-model is more suitable for processing the question text. That is, the gated network layer constructed by the embodiment of the present application understands the field to which each expert sub-model is adapted and the field problems it is good at handling, and can effectively delegate problems in different fields to different experts for processing, ensuring that the most suitable experts to solve the problem can be selected. In addition, due to the rationality of the construction of the gated network layer, when the large hybrid expert model is trained as a whole, the balanced selectivity of the gated network layer can be ensured.

[0083] Secondly, the construction process of the hybrid expert large model built from scratch is similar to training a generative large model from scratch, but the training difficulty is greater. In order to effectively parameterize each part of the model, it is necessary to use a large amount of corpus for multiple steps of training, and the balanced selectivity of the gated network layer needs to be ensured during the training process. Therefore, there are problems such as high training difficulty, many training steps, and low model construction efficiency. The embodiment of the present application reuses the FNN layer in the trained question-answering model to construct the sparse MoE layer of the hybrid expert large model, which greatly reduces the pre-training requirements of the hybrid expert large model, reduces the steps of building the model, and improves the efficiency of model construction. At the same time, different noises are added to each FNN layer, so that the parameters of each expert in the sparse MoE layer are differentiated, which solves the problem of similar capabilities of individual experts, so that the trained hybrid expert large model can better play the advantages of multi-expert collaboration and better handle complex and diverse input data.

[0084] In addition, the embodiment of the present application introduces a smaller amount of general scenario data and further trains the mixed expert large model at a lower learning rate, thereby improving the generalization ability of the MoE model without losing the expert ability.

[0085] In summary, the accuracy of the hybrid expert large model in processing user input questions can be improved by improving the balanced selectivity, generalization ability and expert diversity of the hybrid expert large model.

[0086] The following describes a data processing device based on a hybrid expert big model provided in an embodiment of the present application. The data processing device based on a hybrid expert big model described below and the data processing method based on a hybrid expert big model described above can refer to each other.

[0087] First, combine Figure 6 , the data processing device based on the hybrid expert large model is introduced, such as Figure 6 As shown, the data processing device based on the hybrid expert large model may include: The text acquisition unit 100 is used to acquire the target question text sent by the user terminal; The text processing unit 200 is used to call a pre-built hybrid expert model to process the target question text to obtain a target answer. The pre-built hybrid expert model includes: at least one sub-model group, each sub-model group corresponds to a different target application field, and the sub-model group contains at least one sub-model; a gating weight matrix corresponding to each sub-model, the gating weight matrix is ​​obtained by processing the supervised data corresponding to the sub-model in a pre-trained question-answering model, and the dot product of the gating weight matrix and the feature matrix of the question text in the training sample is used to characterize the similarity between the data features of the supervised data and the data features of the question text; at least one FNN layer, and the weight parameters of each FNN layer are different; The answer sending unit 300 is used to send the target answer to the user terminal.

[0088] In summary, in the hybrid expert large model pre-constructed in the embodiment of the present application, the data features of the supervised data of each application field are learned in the form of a weight matrix, so that the dot product of the gated weight matrix corresponding to each sub-model (expert) and the feature matrix of the question text can characterize the similarity between the data features of the supervised data corresponding to the sub-model and the data features of the question text. The higher the similarity, the more suitable the corresponding sub-model is for processing the target question text. Therefore, the hybrid expert large model constructed based on the gated weight matrix can improve the accuracy of selecting the sub-model. In addition, the weight parameters of the FNN layer of each expert in the hybrid expert large model are different, which can reduce the ability similarity between each sub-model.

[0089] Based on this, this application uses a pre-built hybrid expert large model to process the target question text, and can accurately select a sub-model suitable for the application field to which the target question text belongs. After the similarity of each sub-model is reduced, each sub-model will be more focused on processing the problems in the corresponding application field, so that the obtained target answer text is more biased towards the application field to which the target question text belongs, thereby improving the accuracy of the target answer text.

[0090] In a possible implementation, the functional unit for implementing the construction process of the pre-built hybrid expert large model includes: A request information acquisition unit, used to acquire model building request information, wherein the model building request information includes: a pre-trained question-answering model, at least one target application field, a sub-model group corresponding to each target application field, the number of sub-models in the sub-model group, and supervised data corresponding to each sub-model, wherein the pre-trained question-answering model is a trained large-scale language model; A weight matrix determination unit, used to input the supervised data corresponding to each of the sub-models into the pre-trained question-answering model for processing, and obtain a matrix corresponding to each of the sub-models representing the data features of the supervised data as a gating weight matrix corresponding to each of the sub-models; An FNN layer construction unit, used for constructing an FNN layer, obtaining an initial FNN layer corresponding to each of the sub-models, and the initial weight parameters of each of the initial FNN layers are equal; A parameter adjustment unit, configured to adjust the parameters in the initial FNN layer to obtain a target FNN layer, wherein the weight parameters in each target FNN layer are different from each other, and the adjustment process at least comprises: adding noise data to the initial weight parameters of each initial FNN layer; A model construction unit is used to construct a model based on the pre-trained question-answering model, the gating weight matrix corresponding to each of the sub-models and the target FNN layer to obtain the hybrid expert large model.

[0091] In a possible implementation, the model building unit includes: A layer copying subunit, used to copy the backbone network layer in the pre-trained question-answering model, wherein the backbone network layer at least includes: an attention layer in a transformer decoder module; A network layer construction subunit, used to construct a gated network layer based on the gated weight matrix corresponding to each sub-model; A MoE layer construction subunit, used for constructing a sparse MoE layer based on the target FNN layer corresponding to each sub-model; The large model construction subunit is used to combine the backbone network layer, the gated network layer and the sparse MoE layer to obtain a hybrid expert large model.

[0092] In a possible implementation, the functional unit in the request information acquisition unit for acquiring the supervised data corresponding to each sub-model includes: A sample acquisition subunit is used to obtain supervised data samples corresponding to each target application field; A data division subunit is used to divide the supervised data samples corresponding to the target application field to obtain common data and non-common data in the field corresponding to each target application field; A data equalizer unit is used to divide the non-common data in this field into a number of parts equal to the number of the sub-models based on the number of sub-models in the sub-model group corresponding to the target application field, so as to obtain the non-common data in this field corresponding to each sub-model of the target application field; The other field data statistics subunit is used to take the general data in this field corresponding to other target application fields except the target application field as the other field general data corresponding to the target application field; The supervised data determination subunit is used to obtain the supervised data corresponding to each sub-model based on the general data in the field and the general data in other fields corresponding to the target application field, and the non-general data in the field corresponding to each sub-model.

[0093] In a possible implementation, the weight matrix determining unit includes: A data processing subunit, used to input the supervised data corresponding to each sub-model into the pre-trained question-answering model for processing, to obtain a first feature matrix corresponding to each sub-model representing data features of the common data in the field, a second feature matrix representing data features of the non-common data in the field, and a third feature matrix representing data features of the common data in other fields; a matrix operation subunit, configured to subtract the third characteristic matrix from the sum of the first characteristic matrix and the second characteristic matrix to obtain a characteristic matrix corresponding to each of the sub-models; The regularization processing subunit is used to perform regularization processing on the feature matrix corresponding to each of the sub-models to obtain a gating weight matrix corresponding to each of the sub-models.

[0094] In one possible implementation, the FNN layer building unit includes: The copying subunit is used to copy the FNN layer in the pre-trained question-answering model to obtain an initial FNN layer corresponding to each of the experts, and the initial weight parameters of each of the initial FNN layers are equal.

[0095] In one possible implementation, the functional unit for implementing adding noise data to the initial weight parameters of each of the initial FNN layers includes: A noise acquisition subunit, used for acquiring a preset number of noise distributions, wherein the distribution of each of the noise distributions is different from each other; A noise sampling subunit, used for sampling noise from the noise distribution to obtain noise samples of the same number as the sub-model, and each of the noise samples is different; A noise adding subunit, used for adding the noise samples to the weight parameters of the initial FNN layer respectively, to obtain a target weight parameter corresponding to each of the initial FNN layers; The parameter updating subunit is used to update the weight parameters of the initial FNN layer with the target weight parameters to obtain the target FNN layer.

[0096] In a possible implementation, the noise sampling subunit includes: a reference noise extraction subunit, configured to extract at least one noise segment from the noise distribution as a reference noise when the number of the noise distribution is 1; The reference noise scaling subunit is used to multiply the reference noise by different coefficients respectively to obtain noise samples of the same number as the sub-model, and each of the noise samples is different.

[0097] In a possible implementation, the functional unit implementing the adjustment process of adjusting the parameters in the FNN layer further includes: A first supervised fine-tuning subunit is used to perform supervised fine-tuning on the hybrid expert large model based on the supervised data corresponding to each of the sub-models to obtain a target FNN layer with different weight parameters; The second supervised fine-tuning subunit is used to obtain universal supervised data that is common to all application fields; based on the universal supervised data, supervised fine-tuning is performed on the hybrid expert large model to obtain the target FNN layer with different weight parameters.

[0098] In a possible implementation, the process of the hybrid expert large model processing the target question text includes: The attention layer extracts the features of the target question text to obtain an input feature matrix; Performing dot products on the input feature matrix and each of the gated weight matrices to obtain dot product results corresponding to each of the gated weight matrices; Normalizing the dot product result to obtain the gate selection probability corresponding to each sub-model; Based on the gating selection probability corresponding to each of the sub-models, a sub-model whose corresponding gating selection probability meets a preset condition is determined as a target sub-model, wherein the preset condition at least includes: the gating selection probability reaches a preset threshold, or the gating selection probability is a pre-set number of gating selection probabilities in a probability sequence, and the probability sequence is a sequence of gating selection probabilities arranged in numerical order; The target question text is input into the target FNN layer corresponding to the target sub-model for processing to obtain the target answer.

[0099] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0100] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0101] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method based on a hybrid expert large model, characterized in that: include: Get the target question text sent by the user; Calling a pre-built hybrid expert model to process the target question text to obtain a target answer, wherein the pre-built hybrid expert model includes: at least one sub-model group, each of the sub-model groups corresponds to a different target application field, and the sub-model group includes at least one sub-model; A gating weight matrix corresponding to each of the sub-models, wherein the gating weight matrix is ​​obtained by processing the supervised data corresponding to the sub-model in a pre-trained question-answering model, and the dot product of the gating weight matrix and the feature matrix of the question text in the training sample is used to characterize the similarity between the data features of the supervised data and the data features of the question text; at least one FNN layer corresponding to each of the sub-models, and the weight parameters of each of the FNN layers are different; The target answer is sent to the user terminal.

2. The data processing method based on the hybrid expert large model according to claim 1 is characterized in that: The construction process of the pre-built hybrid expert large model includes: Obtain model building request information, the model building request information including: a pre-trained question-answering model, at least one target application field, a sub-model group corresponding to each target application field, the number of sub-models in the sub-model group, and supervised data corresponding to each sub-model, wherein the pre-trained question-answering model is a trained large-scale language model; The supervised data corresponding to each of the sub-models is respectively input into the pre-trained question-answering model for processing, so as to obtain a matrix corresponding to each of the sub-models representing the data features of the supervised data as a gating weight matrix corresponding to each of the sub-models; Constructing an FNN layer to obtain an initial FNN layer corresponding to each of the sub-models, and the initial weight parameters of each of the initial FNN layers are equal; The parameters in the initial FNN layer are adjusted to obtain a target FNN layer, wherein the weight parameters in each target FNN layer are different from each other, and the process of adjusting the parameters in the initial FNN layer at least comprises: adding noise data to the initial weight parameters of each initial FNN layer; A model is constructed based on the pre-trained question-answering model, the gating weight matrix corresponding to each of the sub-models, and the target FNN layer to obtain the hybrid expert large model.

3. The data processing method based on the hybrid expert large model according to claim 2 is characterized in that: The model is constructed based on the pre-trained question-answering model, the gating weight matrix corresponding to each of the sub-models, and the target FNN layer to obtain the hybrid expert large model, including: Copy the backbone network layer in the pre-trained question-answering model, wherein the backbone network layer includes at least: an attention layer in a transformer-decoder module; Constructing a gated network layer based on the gated weight matrix corresponding to each sub-model; Based on the target FNN layer corresponding to each sub-model, a sparse MoE layer is constructed; The backbone network layer, the gated network layer and the sparse MoE layer are combined to obtain a hybrid expert large model.

4. The data processing method based on the hybrid expert large model according to claim 2 is characterized in that: Obtain the supervised data corresponding to each sub-model, including: Obtain supervised data samples corresponding to each target application field; Dividing the supervised data samples corresponding to the target application field to obtain general data in the field and non-general data in the field corresponding to each target application field; Based on the number of sub-models in the sub-model group corresponding to the target application field, the non-common data in this field is equally divided into a number of parts equal to the number of the sub-models, to obtain the non-common data in this field corresponding to each of the sub-models in the target application field; The general data in the field corresponding to other target application fields except the target application field shall be used as the general data in other fields corresponding to the target application field; Based on the general data in the field and the general data in other fields corresponding to the target application field, and the non-general data in the field corresponding to each sub-model, the supervised data corresponding to each sub-model is obtained.

5. The data processing method based on the hybrid expert large model according to claim 4 is characterized in that: The supervised data corresponding to each of the sub-models is input into the pre-trained question-answering model for processing, and a matrix representing the data features of the supervised data corresponding to each of the sub-models is obtained as a gating weight matrix corresponding to each of the sub-models, including: The supervised data corresponding to each sub-model is respectively input into the pre-trained question-answering model for processing, so as to obtain a first feature matrix corresponding to each sub-model representing data features of the common data in the field, a second feature matrix representing data features of the non-common data in the field, and a third feature matrix representing data features of the common data in other fields; Subtract the third feature matrix from the sum of the first feature matrix and the second feature matrix to obtain a feature matrix corresponding to each of the sub-models; Regularization is performed on the feature matrix corresponding to each of the sub-models to obtain a gating weight matrix corresponding to each of the sub-models.

6. The data processing method based on the hybrid expert large model according to claim 2 is characterized in that: The constructing of the FNN layer to obtain the initial FNN layer corresponding to each of the sub-models includes: The FNN layer in the pre-trained question-answering model is copied to obtain an initial FNN layer corresponding to each of the experts, and the initial weight parameters of each of the initial FNN layers are equal.

7. The data processing method based on the hybrid expert large model according to claim 2 is characterized in that: The adding noise data to the initial weight parameters of each of the initial FNN layers comprises: Acquire a preset number of noise distributions, wherein the distribution of each of the noise distributions is different from each other; Noise sampling is performed from the noise distribution to obtain noise samples of the same number as the sub-model, and each of the noise samples is different; Adding the noise samples to the weight parameters of the initial FNN layers respectively to obtain target weight parameters corresponding to each of the initial FNN layers; The weight parameters of the initial FNN layer are updated with the target weight parameters to obtain a target FNN layer.

8. The data processing method based on the hybrid expert large model according to claim 7 is characterized in that: The step of sampling noise from the noise distribution to obtain noise samples of the same number as the sub-model, wherein each of the noise samples is different, includes: When the number of the noise distribution is 1, extracting at least one noise segment from the noise distribution as a reference noise; The reference noise is multiplied by different coefficients respectively to obtain noise samples of the same number as the sub-model, and each of the noise samples is different.

9. The data processing method based on the hybrid expert large model according to claim 2 is characterized in that: The process of adjusting the parameters in the initial FNN layer also includes: Based on the supervised data corresponding to each of the sub-models, the hybrid expert model is fine-tuned in a supervised manner to obtain target FNN layers with different weight parameters; and / or, Obtaining general supervised data common to all application fields; and performing supervised fine-tuning on the hybrid expert large model based on the general supervised data to obtain the target FNN layer with different weight parameters.

10. The data processing method based on the hybrid expert large model according to any one of claims 1 to 9, characterized in that: The process of the mixed expert large model processing the target question text includes: The attention layer extracts the features of the target question text to obtain an input feature matrix; Performing dot products on the input feature matrix and each of the gated weight matrices to obtain dot product results corresponding to each of the gated weight matrices; Normalizing the dot product result to obtain the gate selection probability corresponding to each sub-model; Based on the gating selection probability corresponding to each of the sub-models, a sub-model whose corresponding gating selection probability meets a preset condition is determined as a target sub-model, wherein the preset condition at least includes: the gating selection probability reaches a preset threshold, or the gating selection probability is a pre-set number of gating selection probabilities in a probability sequence, and the probability sequence is a sequence of gating selection probabilities arranged in numerical order; The target question text is input into the target FNN layer corresponding to the target sub-model for processing to obtain the target answer.

11. A data processing device based on a hybrid expert large model, characterized in that: include: A text acquisition unit, used to acquire the target question text sent by the user end; A text processing unit, used for calling a pre-built hybrid expert large model to process the target question text to obtain a target answer, wherein the pre-built hybrid expert large model includes: at least one sub-model group, each sub-model group corresponds to a different target application field, and the sub-model group includes at least one sub-model; a gating weight matrix corresponding to each sub-model, wherein the gating weight matrix is ​​obtained by processing the supervised data corresponding to the sub-model in a pre-trained question-answering model, and the dot product of the gating weight matrix and the feature matrix of the question text in the training sample is used to characterize the similarity between the data features of the supervised data and the data features of the question text; at least one FNN layer corresponding to each of the sub-models, and the weight parameters of each of the FNN layers are different; An answer sending unit is used to send the target answer to the user terminal.

Images