Training method and system for hybrid expert model of emergency industry and medium

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a training method and system for a hybrid expert model of an emergency industry and a medium, and the method comprises the steps: adding a decoding layer structure behind a base large model, and introducing the hybrid expert model in a decoding layer; the hybrid expert model comprises a shared expert model and a plurality of independent domain expert models; sequentially selecting data of multiple fields of the emergency industry as training data of corresponding training stages; in each training stage, according to the domain information of the training data, only activating the weights of the corresponding domain expert model and the shared expert model, and keeping the weights of other domain expert models in a frozen state; and performing regularization processing on the output of the shared expert model, and using the sum of the loss function of the field expert model in the activated state and the regularization of the shared expert model as the loss function of the current training stage. The method can improve the utilization efficiency of limited data in the emergency industry, simplify the model structure, and improve the prediction accuracy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically to a training method, system and medium for a hybrid expert model in the emergency industry. Background Art

[0002] In the emergency industry, in order to process multi-domain data, a Mixture of Experts (MOE) model is generally built based on the deepseek-v2 network architecture. The Mixture of Experts model is an advanced machine learning architecture that aims to handle complex and diverse tasks by combining multiple "expert" models. The core idea is to decompose the task to multiple expert models that specialize in handling different subtasks, and a gating network dynamically selects or weights the outputs of these expert models.

[0003] Deepseek-v2 changes the FFN layer of the network to MOE, and changes every layer of the network. Although this architectural adjustment is expected to improve the performance of the model, there are still many problems in practical applications. First of all, Deepseek-v2 needs to apply MOE to each layer, which means that its computing cost and resource requirements are several times that of traditional models. This has high hardware requirements and limits its application in resource-limited environments. Moreover, for the emergency industry where the amount of data is not large enough, it is impossible to provide enough training data to support the effective training of Deepseek-v2, which makes the model prone to overfitting and unable to show its due performance in practical applications. MOE is used in each layer of the Deepseek-v2 network, which increases the complexity of the model's parameter adjustment. A lot of experiments and calculations are required to optimize the model parameters, which puts higher technical and resource requirements on the development team.

[0004] Therefore, how to improve the utilization efficiency of limited data in the emergency industry, simplify the model structure, and improve prediction accuracy has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of this, the present invention provides a training method, system and medium for a hybrid expert model in the emergency industry, which solves the technical problems of complex models, large data requirements and inconvenient use in the emergency industry.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for training a hybrid expert model for an emergency industry, comprising the following steps:

[0008] A decoding layer structure is added after the base large model, and a hybrid expert model is introduced in the decoding layer; the hybrid expert model includes a shared expert model and multiple independent domain expert models;

[0009] Select data from the fields of flood control and drought relief, earthquake relief, production safety, forest and grassland fire prevention and control, and disaster prevention, mitigation and relief in turn as training data for the corresponding training stages;

[0010] In each training stage, according to the domain information of the training data, only the weights of the corresponding domain expert model and the shared expert model are activated, and the weights of other domain expert models remain frozen;

[0011] Regularize the output of the shared expert model, and use the sum of the loss function of the currently activated domain expert model and the regularization of the shared expert model as the loss function of the current training stage;

[0012] The weights of the shared expert model and the current domain expert model are updated through the back-propagation algorithm.

[0013] Furthermore, after selecting the training data in each field, the method further includes: performing data enhancement processing on the training data.

[0014] Furthermore, the loss function is expressed as:

[0015] L=L new-expert +L share-expert

[0016] Among them, L new-expert Represents the loss function of the domain expert model that is activated at the current stage; L share-expert Represents the loss function of the shared expert model at the current stage.

[0017] Furthermore, the loss function of the domain expert model is the cross loss function, which is expressed as:

[0018]

[0019] Among them, M represents the number of training samples in the corresponding field in the current training stage, T represents the sequence length, and y i,t represents the true label of the i-th sample at the t-th position, y i,＜t Represents the input before the t-th position of the i-th sample.

[0020] Furthermore, the loss function of the shared expert model is expressed as:

[0021]

[0022] Among them, λ krepresents the regularization coefficient of the kth task, θ represents the weight matrix of the current task, θ k Represents the weight matrix of task k.

[0023] Furthermore, the regularization coefficient λ k The calculation formula is:

[0024]

[0025] Among them, α is a hyperparameter, F k It is calculated by the Fisher information matrix, and the calculation formula is as follows:

[0026]

[0027] Among them, L old is the loss function of the previous task, and the expected value E is obtained by averaging the data set of the previous task.

[0028] Furthermore, the large model of the base adopts any one of llama, Tongyi Qianwen and Baichuan.

[0029] Furthermore, when new knowledge in a certain field needs to be added, the parameters of the base large model are frozen and do not participate in training. The weights of the corresponding domain expert model and shared expert model are activated through the Attention layer of the base large model.

[0030] In a second aspect, the present invention provides a training system for a hybrid expert model in an emergency industry, comprising:

[0031] The model building module is used to add a decoding layer structure after the base model and introduce a hybrid expert model in the decoding layer; the hybrid expert model includes a shared expert model and multiple independent domain expert models;

[0032] A data acquisition module is used to sequentially select data in the fields of flood control and drought relief, earthquake relief, production safety, forest and grassland fire prevention and extinguishing, and disaster prevention, mitigation and relief as training data for the corresponding training stage;

[0033] The model training phase is used to activate only the weights of the corresponding domain expert model and shared expert model according to the domain information of the training data in each training phase, while keeping the weights of other domain expert models frozen; regularize the output of the shared expert model, and use the sum of the loss function of the currently activated domain expert model and the regularized shared expert model as the loss function of the current training phase; and update the weights of the shared expert model and the current domain expert model through the back-propagation algorithm.

[0034] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the training method for the hybrid expert model for the emergency industry as described above.

[0035] It can be seen from the above technical solutions that, compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The present invention does not apply MOE to each layer, but selectively applies MOE on the base large model according to task requirements and data characteristics. This can not only reduce computing costs, but also simplify the model structure. During the training process, the base large model is frozen, and only the parameters of the MOE in the corresponding field are tuned. Through data enhancement and transfer learning technology, the model can be fine-tuned for the emergency industry with small data volume, thereby improving the utilization efficiency of limited data in the emergency field and reducing the overfitting problem caused by insufficient data.

[0037] 2. For the problem of knowledge forgetting in the model, the present invention adopts the method of regularizing the output of the shared expert model to maintain the previous knowledge. Through these improvements, the present invention solves the problems of large model complexity, large data requirements and inconvenient use while maintaining the MOE model's ability to efficiently handle complex tasks, making it more suitable for the professional knowledge data processing needs in the emergency field.

[0038] 3. The shared expert model is responsible for cross-domain knowledge learning, while the independent domain expert model focuses on knowledge in a specific field, which can improve the generalization ability and accuracy of the model.

[0039] 4. The present invention dynamically activates the corresponding domain expert model based on the domain information of the input data, ensuring that the data in each domain can be processed in a targeted manner and improving the processing efficiency of the model. Through the design of independent domain experts, it is ensured that the knowledge of the newly added domain will not have a negative impact on the original domain, thus realizing the isolated management of knowledge.

[0040] 5. In the present invention, when new domain knowledge is needed, only a new domain expert model needs to be added without making major modifications to the original model structure, thus simplifying the model expansion process. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0042] Figure 1A flowchart of a training method for a hybrid expert model for the emergency industry provided by the present invention.

[0043] Figure 2 A schematic diagram of the structure of the hybrid expert model based on the large base model provided by the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] like Figure 1 As shown, the embodiment of the present invention discloses a training method for a hybrid expert model of the emergency industry, comprising the following steps:

[0046] S1. Add a decoding layer structure after the base large model and introduce a hybrid expert model in the decoding layer; the hybrid expert model includes a shared expert model and multiple independent domain expert models;

[0047] S2, sequentially select data in the fields of flood control and drought relief, earthquake relief, production safety, forest and grassland fire prevention and control, and disaster prevention, mitigation and relief as training data for the corresponding training stage;

[0048] S3. In each training stage, according to the domain information of the training data, only the weights of the corresponding domain expert model and the shared expert model are activated, and the weights of other domain expert models remain frozen;

[0049] Regularize the output of the shared expert model, and use the sum of the loss function of the currently activated domain expert model and the regularization of the shared expert model as the loss function of the current training stage;

[0050] The weights of the shared expert model and the current domain expert model are updated through the back-propagation algorithm.

[0051] Repeat S3 to train other domain expert models in turn, and only update the expert model weights and shared expert model weights in the relevant fields during each training.

[0052] The above steps are further explained below.

[0053] S1. The present invention adopts a structure in which a decoding layer is added after the base large model, and a hybrid expert model structure is introduced in the decoding layer to improve the feedforward neural network layer (FFC layer) of the original Transformer decoding Attention. The specific structure is as follows Figure 1As shown, it includes a shared expert model (Share Expert) and multiple independent domain expert models (Expert).

[0054] The shared expert model is responsible for learning knowledge in all domains, while each independent domain expert model focuses on knowledge in its own domain.

[0055] The large model of the base adopts large models such as llama, Tongyi Qianwen, and Baichuan.

[0056] S2. Select data in the fields of flood control and drought relief, earthquake relief, production safety, forest and grassland fire prevention and control, and disaster prevention, mitigation and relief in turn, perform data enhancement processing, and use them as training data for the corresponding training stage; a specific data enhancement method may be to train the model by randomly masking some words in a sentence and allowing the model to infer the masked words through the context during training.

[0057] Specifically, the present invention has produced text question answering and intent classification datasets for the fields of flood control and drought relief, earthquake disaster relief, production safety, forest and grassland fire prevention and control, and disaster prevention, mitigation and relief.

[0058] In the field of forest drills, disaster prevention and fire fighting, the sample text questions and answers are:

[0059] Input: Which of the following is not included in the commonly used forest fire fighting tactics?

[0060] A. Single point breakthrough, long-term assault tactics

[0061] B. Surround on all sides and assault on all fronts

[0062] C. One-shot, full-line control tactics

[0063] D. Night assault tactics

[0064] Output: D. Night assault tactics

[0065] In the field of production safety, examples of text questions and answers are:

[0066] Input: What types of administrative compulsory measures for production safety are included?

[0067] A. Seizure

[0068] B. Seizure

[0069] C. Order to correct

[0070] D. Forced suspension of production and business

[0071] Output: A. Seizure, B. Detention

[0072] Intent classification is to classify the query sentences into categories such as flood control and drought relief, earthquake relief, production safety, forest and grassland fire prevention and control, and disaster prevention, mitigation and relief. For example:

[0073] Example 1:

[0074] Input: In industrial and trade enterprises, setting up employee dormitories in factory buildings without approval will be considered a major accident hazard?

[0075] Output: safe production.

[0076] Example 2:

[0077] Input: Among the basic principles of explosion prevention, can strict control of fire sources reduce the damage after an explosion?

[0078] Output: disaster prevention, mitigation and relief.

[0079] S3. In each training stage, according to the domain information of the training data, only the weights of the corresponding domain expert model and the shared expert model are activated, and the weights of other domain expert models remain frozen.

[0080] For example:

[0081] When the data of domain 0 is input into the model, only the weight of Expert0 and the weight of the shared expert are activated after Attention.

[0082] When the data from domain 1 is input into the model, only the weights of Expert1 and Share Expert are activated after Attention.

[0083] When the data of domain 2 is newly added into the model, only the weights of Expert2 and Share Expert are activated after Attention.

[0084] With this method, when new knowledge in a certain field is added, only the expert model of the corresponding field is needed to understand the new field knowledge without affecting the original field. This method has a plug-and-play effect and effectively improves the scalability and flexibility of the model.

[0085] After model training and fine-tuning, the model may forget some previously learned knowledge. In order to solve this problem and prevent catastrophic forgetting of previously learned knowledge when new corpus samples appear, the present invention freezes other expert models when training the model and only activates the shared expert model and the newly added domain expert model.

[0086] Specifically, when receiving a new corpus sample, first, all domain expert models except the shared expert model and the newly added domain expert model are frozen to ensure that the parameters of these models do not change during the new training process, thereby retaining the knowledge they learned in previous tasks.

[0087] Afterwards, only the shared expert model and the newly added expert model are trained to adapt them to the new data distribution, so that the model can effectively utilize the new data when processing new tasks.

[0088] The present invention also uses regularization technology to constrain the shared expert output to ensure that it does not deviate too much from previous knowledge and maintains the previously pre-trained knowledge, thereby maintaining the stability and consistency of the model.

[0089] Through this training method, the present invention can not only adapt to new data distribution while maintaining old knowledge, but also show good balance and adaptability in new tasks. When processing new data, the shared expert model can retain old knowledge and effectively fit the new distribution, ensuring the stable performance and generalization ability of the model in different tasks.

[0090] Specifically, when a new domain expert model is added, the previous domain expert model is frozen, and only the shared expert model and the newly added domain expert model are activated. The present invention adopts the regularization method of elastic weight integration (EWC) to avoid the knowledge learned by the new task from having too much influence on the previously learned task. Specifically, when the newly added expert is the Nth expert, the loss function is the sum of the loss function of the newly added domain expert model and the regularization output of the shared expert model. The specific formula is as follows:

[0091] L=L new-expert +L share-expert

[0092] Among them, L new-expert Represents the loss function of the domain expert model that is activated at the current stage; L share-expert Represents the loss function of the shared expert model at the current stage.

[0093] The loss function of the domain expert model is the cross loss function, which is expressed as:

[0094]

[0095] Among them, M represents the number of training samples in the corresponding field in the current training stage, T represents the sequence length, and y i,t represents the true label of the i-th sample at the t-th position, y i,＜t Represents the input before the t-th position of the i-th sample.

[0096] The loss function of the shared expert model is expressed as:

[0097]

[0098] Among them, λ k represents the regularization coefficient of the kth task, where the kth task refers to the kth expert in the N-1 domain expert models; θ represents the weight matrix of the current task, θ k Represents the weight matrix of task k.

[0099] Regularization coefficient λ k The calculation formula is:

[0100]

[0101] Among them, α is a hyperparameter, F k It is calculated by the Fisher information matrix, and the calculation formula is as follows:

[0102]

[0103] Among them, L old is the loss function of the previous task, and the previous task refers to the N-1 expert tasks before the current Nth expert model. For example, expert models for flood control and drought relief, earthquake relief, safe production, and forest and grassland fire prevention and control are trained respectively, and the current expert model for disaster prevention, mitigation and relief needs to be trained. The model training for flood control and drought relief, earthquake relief, safe production, and forest and grassland fire prevention and control is the previous task; the expected value E is obtained by summing and averaging the data sets of the previous tasks.

[0104] The present invention aims at the fields of flood control and drought relief, earthquake relief, production safety, forest and grassland fire prevention and control, disaster prevention, mitigation and relief in the emergency industry, constructs corresponding domain expert models and shared expert models, constitutes a hybrid expert model, and trains the hybrid expert model in a fine-tuning manner. The base large model adopts the currently popular large models such as llama, Qianwen, Baichuan, etc. The parameters of the base large model are frozen and do not participate in the training. The parameters in the MOE decoding layer are randomly initialized. During the training process, each time the expert model in a different field is trained, only the weight of the domain-specific expert in the field is activated and adjusted, and the weight of the shared expert is also fine-tuned according to the knowledge in the field.

[0105] The shared expert model contains common knowledge in all fields and remains activated during each training process. In this way, it can be ensured that the shared experts can continuously consolidate and update their knowledge during the training process in different fields. In order to avoid the shared experts' previous knowledge being forgotten, the above regularization function is combined with the loss function as the final loss function.

[0106] Through this training method, the weight adjustment of shared experts and experts in specific fields in the neural network model can be effectively managed and optimized, efficient cross-domain learning and knowledge sharing can be achieved, and an effective solution can be provided for the training of neural network models for complex tasks.

[0107] Next, the performance of the method of the present invention is further verified.

[0108] The present invention selects qwen2-1.5b-instruct, Llama2-Chinese-7b, and baichuan2-13b-chat-v1 as the base large model, and adds the decoding layer of the MOE (Mixture of Experts) mixed expert model on the basis of it. Through the aforementioned training process, expert models specifically for flood control and drought relief, earthquake relief, production safety, forest and grassland fire prevention and control, and disaster prevention, mitigation and relief are generated.

[0109] In order to ensure the practical application effect of the model, a large number of verification experiments and application tests were carried out to ensure that the performance of the model in various fields reached the practical standard. Through these efforts, the hybrid expert model not only has a significant improvement in performance, but also shows extremely high reliability and effectiveness in practical applications. Data sets for text question answering and intent classification were produced for specific fields. Table 1 shows the test results of text generation and intent classification for specific fields. Test data: Question answering data for each field, the data for each field ranges from 2k to 5k.

[0110] Table 1

[0111]

[0112] The present invention uses a mixture of experts (MoE) model to enhance the performance of the base large model. By introducing the MoE model, it can be observed that the accuracy on specific tasks has been significantly improved. Specifically, compared with the single base model, the introduction of the mixed expert model brings about a 1% improvement in accuracy.

[0113] In the present invention, domain expert models are developed for knowledge in the fields of flood control and drought relief, earthquake relief, production safety, forest and grassland fire prevention and control, and disaster prevention, mitigation and relief. For different knowledge fields, the number of domain expert models can be adjusted, and models have been customized for knowledge in different fields. For the base large model, the present invention tested qwen2-1.5b-instruct, Llama2-Chinese-7b, and baichuan2-13b-chat-v1 respectively. The base large model can use any large language model for fine-tuning development.

[0114] The present invention adopts the elastic weight integration (EWC) regularization method as the regularization function for the shared expert. Other regularization functions may be considered to suppress the old knowledge of the shared expert.

[0115] In other embodiments, the present invention further provides a training system for a hybrid expert model of the emergency industry, which is applicable to the method described above, including:

[0116] The model building module is used to add a decoding layer structure after the base model and introduce a hybrid expert model in the decoding layer; the hybrid expert model includes a shared expert model and multiple independent domain expert models;

[0117] A data acquisition module is used to sequentially select data in the fields of flood control and drought relief, earthquake relief, production safety, forest and grassland fire prevention and extinguishing, and disaster prevention, mitigation and relief as training data for the corresponding training stage;

[0118] The model training phase is used to activate only the weights of the corresponding domain expert model and shared expert model according to the domain information of the training data in each training phase, while keeping the weights of other domain expert models frozen; regularize the output of the shared expert model, and use the sum of the loss function of the currently activated domain expert model and the regularized shared expert model as the loss function of the current training phase; and update the weights of the shared expert model and the current domain expert model through the back-propagation algorithm.

[0119] In other embodiments, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the training method of the hybrid expert model for the emergency industry as described above are implemented.

[0120] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0121] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for a hybrid expert model in the emergency industry, characterized in that: The following steps are involved: A decoding layer structure is added after the base large model, and a hybrid expert model is introduced in the decoding layer; the hybrid expert model includes a shared expert model and multiple independent domain expert models; Select data from the fields of flood control and drought relief, earthquake relief, production safety, forest and grassland fire prevention and control, and disaster prevention, mitigation and relief in turn as training data for the corresponding training stages; In each training stage, according to the domain information of the training data, only the weights of the corresponding domain expert model and the shared expert model are activated, and the weights of other domain expert models remain frozen; Regularize the output of the shared expert model, and use the sum of the loss function of the currently activated domain expert model and the regularization of the shared expert model as the loss function of the current training stage; The weights of the shared expert model and the current domain expert model are updated through the back-propagation algorithm.

2. The training method of the hybrid expert model for the emergency industry according to claim 1 is characterized in that: After selecting the training data in each field, it also includes: performing data enhancement processing on the training data.

3. The training method of the hybrid expert model for the emergency industry according to claim 1 is characterized in that: The expression of the loss function is: L=L new-expert +L share-expert Among them, L new-expert Represents the loss function of the domain expert model that is activated at the current stage; L share-expert Represents the loss function of the shared expert model at the current stage.

4. The training method of the hybrid expert model for the emergency industry according to claim 3 is characterized in that: The loss function of the domain expert model is the cross loss function, which is expressed as: Among them, M represents the number of training samples in the corresponding field in the current training stage, T represents the sequence length, and y i,t represents the true label of the i-th sample at the t-th position, y i,＜t Represents the input before the t-th position of the i-th sample.

5. The training method of the hybrid expert model for the emergency industry according to claim 3 is characterized in that: The loss function of the shared expert model is expressed as: Among them, λ k represents the regularization coefficient of the kth task, θ represents the weight matrix of the current task, θ k Represents the weight matrix of task k.

6. The training method of the hybrid expert model for the emergency industry according to claim 5 is characterized in that: Regularization coefficient λ k The calculation formula is: Among them, α is a hyperparameter, F k It is calculated by the Fisher information matrix, and the calculation formula is as follows: Among them, L old is the loss function of the previous task, and the expected value E is obtained by averaging the data set of the previous task.

7. The training method of the hybrid expert model for the emergency industry according to claim 1, characterized in that: The large model of the base can be any one of llama, Tongyi Qianwen and Baichuan.

8. The training method of the hybrid expert model for the emergency industry according to claim 1, characterized in that: When new knowledge in a certain field needs to be added, the parameters of the base large model are frozen and do not participate in training. The weights of the corresponding domain expert model and shared expert model are activated through the Attention layer of the base large model.

9. A training system for a hybrid expert model in the emergency industry, characterized in that: It is applicable to the method according to any one of claims 1 to 8, comprising: The model building module is used to add a decoding layer structure after the base model and introduce a hybrid expert model in the decoding layer; the hybrid expert model includes a shared expert model and multiple independent domain expert models; A data acquisition module is used to sequentially select data in the fields of flood control and drought relief, earthquake relief, production safety, forest and grassland fire prevention and extinguishing, and disaster prevention, mitigation and relief as training data for the corresponding training stage; The model training phase is used to activate only the weights of the corresponding domain expert model and shared expert model according to the domain information of the training data in each training phase, while keeping the weights of other domain expert models frozen; regularize the output of the shared expert model, and use the sum of the loss function of the currently activated domain expert model and the regularized shared expert model as the loss function of the current training phase; and update the weights of the shared expert model and the current domain expert model through the back-propagation algorithm.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the training method for the hybrid expert model of the emergency industry as described in any one of claims 1 to 8 are implemented.

Images