Model construction method and device based on model fusion, readable medium and equipment

Abstract

The invention discloses a model construction method and device based on model fusion, a readable medium and equipment. The method comprises the steps of obtaining a plurality of pre-training models according to a target task; according to the pre-training model, based on a performance evaluation index of the target task, allocating a weight to the pre-training model; performing hierarchical fusion processing on the plurality of pre-training models according to the weight of each pre-training model to obtain a fusion model; the hierarchical fusion processing comprises the step of performing fusion processing on the plurality of pre-training models for a plurality of data processing layers; and performing model training according to the fusion model to obtain a target model which is used for executing the target task. According to the technical scheme, the obtained target model is the model obtained by fusing the multiple pre-training models, the target model can integrate the advantages of the pre-training models, higher adaptability and robustness are shown in a multi-task environment, and the accuracy and generalization ability of the model are improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a model building method, device, readable medium and equipment based on model fusion. Background Art

[0002] It is a relatively common technical operation to process natural language tasks through machine learning models. In order to improve processing efficiency, some institutions have publicly released pre-trained language models that have been trained on a large amount of data. Users can quickly select a suitable pre-trained language model based on their own target tasks. In some cases, users may select multiple pre-trained language models, and finally select a suitable target model as the final model based on the performance of each pre-trained language model on the validation set. This model selection method only considers a single best model, so that the final model can only guarantee a relatively good effect on the task processing result in one aspect, and the overall performance still needs to be improved. Summary of the invention

[0003] The purpose of this application is to provide a model construction method, device, readable medium and equipment based on model fusion, so as to fully integrate the advantages of multiple pre-trained models and improve the accuracy and generalization ability of the target model.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a model building method based on model fusion is provided, comprising:

[0006] Obtain multiple pre-trained models based on the target task;

[0007] Assigning a weight to the pre-trained model according to a performance evaluation index of the pre-trained model based on the target task;

[0008] Performing layered fusion processing on the multiple pre-trained models according to the weights of the pre-trained models to obtain a fusion model; the layered fusion processing includes performing fusion processing on the multiple pre-trained models for multiple data processing layers respectively;

[0009] Model training is performed according to the fusion model to obtain a target model, and the target model is used to execute the target task.

[0010] According to one aspect of an embodiment of the present application, a model building device based on model fusion is provided, comprising:

[0011] A pre-trained model acquisition module is used to acquire multiple pre-trained models according to the target task;

[0012] A weight assignment module, used to assign weights to the pre-trained model according to a performance evaluation index of the pre-trained model based on the target task;

[0013] A hierarchical fusion module, used for performing hierarchical fusion processing on the multiple pre-trained models according to the weights of each pre-trained model to obtain a fusion model; the hierarchical fusion processing includes performing fusion processing on the multiple pre-trained models for multiple data processing layers respectively;

[0014] The training module is used to perform model training according to the fusion model to obtain a target model, and the target model is used to perform the target task.

[0015] In one embodiment of the present application, the training module is specifically used for:

[0016] The multiple pre-trained models are respectively used as multiple teacher models, and the fusion model is used as a student model to perform knowledge distillation to obtain the target model.

[0017] In one embodiment of the present application, the training module is specifically used for:

[0018] Using the multiple pre-trained models as multiple teacher models respectively and the fusion model as a student model, performing knowledge distillation to obtain a distillation model;

[0019] The model parameters of the distillation model are fine-tuned to obtain the target model.

[0020] In one embodiment of the present application, the weight allocation module is specifically used to:

[0021] Calculate the sum of the indicators of the multiple pre-training models based on the performance evaluation indicators of the target task respectively;

[0022] The weight of the pre-trained model is generated according to the ratio of the performance evaluation index of the target task based on the pre-trained model to the sum of the indexes.

[0023] In one embodiment of the present application, the weight allocation module is specifically used to:

[0024] Obtaining computing resource consumption of the pre-trained model;

[0025] The weight of the pre-trained model is obtained by multiplying the ratio of the performance evaluation index of the pre-trained model based on the target task to the sum of the indexes by the inverse of the computing resource consumption.

[0026] In one embodiment of the present application, the weight allocation module is specifically used to:

[0027] According to the weight of each performance evaluation indicator of the target task, calculate the weighted sum of the first indicators of the multiple pre-training models based on the multiple performance evaluation indicators of the target task respectively;

[0028] Calculate the weighted sum of the second indicator of the pre-trained model based on the multiple performance evaluation indicators of the target task according to the weight of each performance evaluation indicator of the target task;

[0029] The weight of the pre-trained model is generated according to the ratio of the weighted sum of the first indicators to the weighted sum of the second indicators.

[0030] In one embodiment of the present application, the pre-training model acquisition module is specifically used to:

[0031] Obtain multiple pre-trained models with the same model architecture according to the target task;

[0032] The input data formats and output layer structures of the plurality of pre-trained models are normalized.

[0033] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the model building method based on model fusion in the above technical solution is implemented.

[0034] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor executes the executable instructions so that the electronic device executes a model building method based on model fusion as in the above technical solution.

[0035] According to one aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the model building method based on model fusion in the above technical solution.

[0036] In the technical solution provided in the embodiment of the present application, firstly, multiple pre-trained models are obtained according to the target task; then, weights are assigned to the pre-trained models according to the performance evaluation indicators of the pre-trained models based on the target task; next, the multiple pre-trained models are layered and fused according to the weights of each pre-trained model to obtain a fusion model; the layered fusion processing includes fusing the multiple pre-trained models for multiple data processing layers respectively; finally, model training is performed according to the fusion model to obtain a target model, which is used to perform the target task. Finally, the target model obtained is a model obtained by fusing multiple pre-trained models. The target model can integrate the strengths of each pre-trained model and show stronger adaptability and robustness in a multi-task environment, thereby improving the accuracy and generalization ability of the model.

[0037] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 The exemplary system architecture block diagram applying the technical solution of the present application is schematically shown.

[0040] Figure 2 A flow chart of a model building method based on model fusion provided by an embodiment of the present application is schematically shown.

[0041] Figure 3 A flow chart of a model building method based on model fusion provided by an embodiment of the present application is schematically shown.

[0042] Figure 4 A flow chart of a model building method based on model fusion provided by an embodiment of the present application is schematically shown.

[0043] Figure 5 The structural block diagram of the model building device based on model fusion provided in an embodiment of the present application is schematically shown.

[0044] Figure 6 The structure block diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0045] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0046] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0047] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0048] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0049] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0050] It is understandable that in the specific implementation of this application, related data such as customer information (such as transaction information, reconciliation data) is involved. When the above embodiments of this application are applied to specific products or technologies, customer permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0051] Figure 1 The exemplary system architecture block diagram applying the technical solution of the present application is schematically shown.

[0052] like Figure 1 As shown, the system architecture 100 may include a terminal device 110, a network 120 and a server 130. The terminal device 110 may include a smart phone, a tablet computer, a laptop computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, etc. The server 130 may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The network 120 may be a communication medium of various connection types that can provide a communication link between the terminal device 110 and the server 130, for example, a wired communication link or a wireless communication link.

[0053] According to the implementation requirements, the system architecture in the embodiment of the present application can have any number of terminal devices, networks and servers. For example, the server 130 can be a server group composed of multiple server devices. In addition, the technical solution provided in the embodiment of the present application can be applied to the terminal device 110, can also be applied to the server 130, or can be implemented by the terminal device 110 and the server 130 together, and the present application does not make any special restrictions on this.

[0054] The model building method based on model fusion provided by this application is described in detail below in conjunction with specific implementation methods.

[0055] Figure 3 A flowchart of a model building method based on model fusion provided by an embodiment of the present application is schematically shown. The method can be implemented by a terminal device or a server, such as Figure 1 The terminal device 110 or the server 130 shown. Figure 3 As shown, the model building method based on model fusion provided in this embodiment includes steps 310 to 340, which are specifically as follows:

[0056] Step 210: Acquire multiple pre-trained models according to the target task.

[0057] Specifically, the target task refers to the task that the user needs the model to perform, such as classifying text, recognizing the meaning of speech, sentiment analysis, answering questions, etc. A pre-trained model refers to a machine learning or deep learning model that has been trained on a large amount of data and has learned a wide range of features, such as the GPT model, BERT model, etc. Different pre-trained models can be used to handle different tasks. Therefore, when obtaining a pre-trained model, it is necessary to refer to the target task and obtain multiple pre-trained models that can achieve the target task.

[0058] In one embodiment of the present application, when multiple pre-trained models are obtained according to a target task, multiple pre-trained models with the same model architecture are first obtained according to the target task; then the input data format and output layer structure of the multiple pre-trained models are normalized. That is, multiple pre-trained models that can achieve the target task and have the same model architecture are obtained, and the input and output of the multiple pre-trained models are normalized to facilitate subsequent model fusion. For example, the selected pre-trained models are all Transformer-based models. Unifying the model architecture of multiple pre-trained models can simplify the subsequent fusion process and avoid compatibility issues caused by architectural differences.

[0059] The normalization of the input data format includes unifying the word segmentation method and special character processing method of each pre-trained model for the input data. The normalization of the output layer structure is to make the output data of each pre-trained model have consistent dimensions and shapes. Here, the normalization of the output layer structure can be performed by interpolation, cropping or supplementary layers.

[0060] In one embodiment of the present application, in order to reduce the waste of computing resources and improve the fusion effect, when selecting a pre-trained model, it is also necessary to evaluate the parameter scale and computing efficiency of the model to ensure that the selected model can complete training and reasoning under reasonable resource conditions. For example, pre-trained models with too large parameter scale or too slow computing efficiency will not be considered.

[0061] Step 220: assign weights to the pre-trained model according to the performance evaluation index of the pre-trained model based on the target task.

[0062] Specifically, the performance evaluation indicators of the pre-trained model based on the target task are the evaluation data of the pre-trained model executing the target task, such as accuracy (Acc), precision (Precision), loss value (loss), recall rate (Recall), F1 score, resource utilization rate, etc., among which the loss value directly reflects the degree of difference between the model prediction result and the actual target; the F1 score is the harmonic mean of the precision and recall rate; resource utilization rate refers to the computing resources occupied by the pre-trained model for training or recommendation, such as CPU / GPU occupancy, content occupancy, etc.

[0063] When assigning weights, you can first determine the performance evaluation indicator that is important for this target task, and then assign a larger weight to the pre-trained model that performs better on this performance evaluation indicator. For example, in classification tasks, accuracy is usually an important indicator, so a higher weight is assigned to the pre-trained model with a higher accuracy; for another example, in information extraction or sequence labeling tasks, the F1 score may be more valuable, so a higher weight can be assigned to the pre-trained model with a higher F1 score.

[0064] In one embodiment of the present application, when assigning weights, the sum of the performance evaluation indicators of the target task based on multiple pre-trained models can be calculated first; then the weight of the pre-trained model is generated according to the ratio of the performance evaluation indicator of the target task based on the pre-trained model to the sum of the indicators. For example, the weight ω of the i-th pre-trained model i It can be expressed as the following formula:

[0065]

[0066] Among them, Perf i represents the performance evaluation index of the i-th pre-trained model for the target task. For example, if the performance evaluation index is accuracy, then Perf i represents the accuracy of the i-th pre-trained model for the target task; n represents the total number of pre-trained models; It represents the sum of the performance evaluation indicators of multiple pre-trained models based on the target task. For example, assuming that the target task focuses on accuracy, there are three pre-trained models, and the accuracy of each pre-trained model for the target task is 0.5, 0.3, and 0.8 respectively. Then the sum of the indicators corresponding to these three pre-trained models is 0.5+0.3+0.8=1.6, and the weights of the three pre-trained models are

[0067] In one embodiment of the present application, when allocating weights, the consumption of computing resources by the pre-trained model can be further considered, and the weight allocation process includes: calculating the sum of the performance evaluation indicators of multiple pre-trained models based on the target task; obtaining the computing resource consumption of the pre-trained model; multiplying the ratio of the performance evaluation indicator of the pre-trained model based on the target task to the sum of the indicators by the inverse of the computing resource consumption to obtain the weight of the pre-trained model. Large-scale models usually have higher performance, but in resource-constrained environments, their weights need to be appropriately reduced to reduce the computational burden. For example, the weight ω of the i-th pre-trained model i It can be expressed as the following formula:

[0068]

[0069] Among them, Perf i represents the performance evaluation index of the i-th pre-trained model for the target task, Cost i Represents the computing resource consumption of the i-th pre-trained model for the target task, such as parameter size or floating-point operations per second. By normalizing resource consumption, a balance can be achieved between performance and resources.

[0070] By combining performance evaluation indicators and computing resource consumption for weight allocation, the reliance on multi-device parallelism is greatly reduced, so that complex model fusion tasks can be efficiently completed even in an environment with limited hardware resources. The entire model fusion process can be efficiently completed on a single GPU, thereby improving the applicability and operability of the technology.

[0071] In one embodiment of the present application, a pre-trained model for a target task can usually be evaluated by multiple performance evaluation indicators, and the importance of each performance evaluation indicator relative to the target task may also be different. For example, in a classification task, accuracy may be the primary consideration indicator; while in an unbalanced classification task, the F1 score or recall rate may be more important. Therefore, the weight allocation process may include: calculating the weighted sum of first indicators of multiple pre-trained models based on multiple performance evaluation indicators of the target task according to the weights of the target task for each performance evaluation indicator; calculating the weighted sum of second indicators of the pre-trained model based on multiple performance evaluation indicators of the target task according to the weights of the target task for each performance evaluation indicator; generating the weight of the pre-trained model according to the ratio of the weighted sum of the first indicator to the weighted sum of the second indicator. Exemplarily, the weight ω of the i-th pre-trained model i It can be expressed as the following formula:

[0072]

[0073] Among them, Pref i,k represents the kth performance evaluation index when the i-th pre-trained model is evaluated for the target task, λ k represents the weight of the target task for the kth performance evaluation indicator, n represents the total number of pre-trained models, and m represents the total number of performance evaluation indicators. It represents the weighted sum of the first indicator, Then represents the weighted sum of the second indicator. In this embodiment, the weights of the target task for each performance evaluation indicator are introduced, and the weights of each pre-training model can be dynamically adjusted.

[0074] Step 230, performing hierarchical fusion processing on the multiple pre-trained models according to the weights of the respective pre-trained models to obtain a fusion model; the hierarchical fusion processing includes performing fusion processing on the multiple pre-trained models for multiple data processing layers respectively.

[0075] Specifically, from the perspective of model structure, a pre-trained model usually includes multiple data processing layers, such as input layer, hidden layer, output layer, etc. Layered fusion processing means that multiple pre-trained models are fused for each data processing layer. For example, for the input layer, the input layers of multiple pre-trained models are weighted fused to obtain the input layer of the fused model; for the hidden layer, the hidden layers of multiple pre-trained models are weighted fused to obtain the hidden layer of the fused model; for the output layer, the output layers of multiple pre-trained models are weighted fused to obtain the output layer of the fused model.

[0076] In one embodiment of the present application, in the process of layered fusion, the functions of each data processing layer can be further considered, and the best fusion can be selected. For example, the input layer is mainly responsible for converting raw data into feature vectors, so the model with the best text encoding effect for a specific language or field can be selected. The hidden layer focuses on capturing the deep semantic information of the text, so a model with strong semantic modeling ability can be integrated. The output layer is closely related to the specific task, and the model that performs best on the target task (target task) can be selected.

[0077] Layered fusion not only retains the knowledge advantages of each model at different levels, but also significantly reduces redundant parameters and improves computational efficiency. Furthermore, in order to achieve coordination between layers, a weight adjustment mechanism between layers can be introduced to ensure the performance optimization of the overall model.

[0078] Step 240: Perform model training based on the fusion model to obtain a target model, which is used to perform the target task.

[0079] Specifically, model training is performed based on the fusion model, including calculating the loss function and updating the model parameters based on the loss function. The final trained model is the target model used to perform the target task. In this way, the final target model is not limited to selecting a single best model, but assigns weights to each model based on performance indicators on the validation set (such as accuracy, F1 score, etc.). By integrating the advantages of multiple models, the generalization ability and prediction accuracy of the model are significantly improved, providing reliable technical support for the efficient solution of complex tasks.

[0080] In one embodiment of the present application, the data set for model training should cover a variety of natural language processing tasks, such as text classification, sentiment analysis, and named entity recognition, etc. The diversity of these tasks helps to improve the model's adaptability to different task scenarios.

[0081] In the technical solution provided in the embodiment of the present application, firstly, multiple pre-trained models are obtained according to the target task; then, weights are assigned to the pre-trained models according to the performance evaluation indicators of the pre-trained models based on the target task; next, the multiple pre-trained models are layered and fused according to the weights of each pre-trained model to obtain a fusion model; the layered fusion processing includes fusing the multiple pre-trained models for multiple data processing layers respectively; finally, model training is performed according to the fusion model to obtain a target model, which is used to perform the target task. Finally, the target model obtained is a model obtained by fusing multiple pre-trained models. The target model can integrate the strengths of each pre-trained model and show stronger adaptability and robustness in a multi-task environment, thereby improving the accuracy and generalization ability of the model.

[0082] Figure 3 The flowchart of a model building method based on model fusion provided by an embodiment of the present application is schematically shown. This embodiment is a further refinement of the above embodiment. Figure 3 As shown, the model building method based on model fusion provided in the embodiment of the present application includes steps 310 to 340, which are as follows:

[0083] Step 310: Acquire multiple pre-trained models according to the target task.

[0084] Step 320: assign weights to the pre-trained model according to the performance evaluation index of the pre-trained model based on the target task.

[0085] Step 330: performing hierarchical fusion processing on the multiple pre-trained models according to the weights of the respective pre-trained models to obtain a fusion model; the hierarchical fusion processing includes performing fusion processing on the multiple pre-trained models at multiple data processing layers respectively.

[0086] For steps 310 to 330, reference may be made to the related descriptions in the aforementioned steps 210 to 230, which will not be repeated here.

[0087] Step 340: Use multiple pre-trained models as multiple teacher models and the fusion model as the student model to perform knowledge distillation to obtain a target model.

[0088] Specifically, after obtaining the fusion model, the model is further trained through knowledge distillation. During the knowledge distillation process, the pre-trained model is used as the teacher model and the fusion model is used as the student model. The training is completed by learning the prediction distribution (soft label) of the teacher model. In this way, the knowledge of multiple teacher models can be transferred to the student model, thereby improving the performance of the fusion model.

[0089] In one embodiment of the present application, in the process of using multiple pre-trained models as teacher models to perform knowledge distillation on the fusion model, the multiple pre-trained models can be sorted first to obtain a model order, and then the pre-trained models in the model order are used as teacher models in sequence to perform sequential knowledge distillation. That is, the current pre-trained model is obtained according to the model order; the current pre-trained model is used as the teacher model, and the fusion model is used as the student model to perform knowledge distillation to obtain a distilled model. If the current pre-trained model is not the last pre-trained model in the model order, the distilled model is used as the fusion model, and the step of obtaining the current pre-trained model according to the model order is returned; if the current pre-trained model is the last pre-trained model in the model order, the distilled model is used as the target model.

[0090] In one embodiment of the present application, multiple pre-trained models can also be used to perform knowledge distillation simultaneously. During this process, the average value of the output data of each pre-trained model can be used as the learning target of the fusion model, or the output data of each pre-trained model can be weighted combined, and the weighted combination value can be used as the learning model of the fusion model.

[0091] In the technical solution provided in the embodiment of the present application, after obtaining the fusion model, multiple pre-trained models are used as multiple teacher models respectively, and the fusion model is used as the student model to perform knowledge distillation to obtain the target model, so that the fusion model can learn its implicit patterns and rules from multiple pre-trained models, which not only improves the performance of the fusion model, but also significantly enhances its generalization ability. In addition, knowledge distillation can also effectively reduce the model size, making the fusion model more suitable for deployment in resource-constrained environments.

[0092] In one embodiment of the present application, after performing knowledge distillation using multiple pre-trained models as multiple teacher models and a fusion model as a student model to obtain a distillation model, the model parameters of the distillation model can be further fine-tuned to obtain a target model. Fine-tuning is a small-scale training process for a specific task or data set based on the distillation model. Through fine-tuning, the fusion model can be better adapted to the target task and the accuracy and generalization ability of the target model can be improved. During the fine-tuning process, hyperparameters such as the learning rate, batch size, and regularization parameters may need to be adjusted to find the optimal training configuration. At the same time, it is also necessary to monitor the training process of the model, such as changes in the loss function, performance indicators on the validation set, etc., to ensure that the model can stably converge to the optimal solution.

[0093] In one embodiment of the present application, after obtaining the target model, an independent test set can also be used to evaluate the performance of the target model. The test set should be independent of the training set and the validation set to ensure the objectivity and accuracy of the evaluation results. By comparing the performance evaluation indicators (accuracy, F1 score, recall rate, etc.) of the target model and the single pre-trained model on the test set, the effectiveness of the target model can be verified.

[0094] Figure 4 The flowchart of a model building method based on model fusion provided by an embodiment of the present application is schematically shown. This embodiment is a further refinement of the above embodiment. Figure 4 As shown, the model building method based on model fusion provided in the embodiment of the present application includes the following steps:

[0095] Step 1: Start, data preparation phase

[0096] Data preparation is the first stage of model fusion. At this stage, it is necessary to collect and preprocess datasets suitable for training and validation. To ensure that the fused model has wide applicability, the dataset should cover a variety of natural language processing tasks, such as text classification, sentiment analysis, and named entity recognition. The diversity of these tasks helps improve the model's adaptability to different task scenarios.

[0097] At the same time, the validation dataset plays a vital role in model fusion. The validation dataset is an important basis for evaluating the performance of each model. It is mainly used to measure the performance of the model on unseen data and ensure the generalization ability of the fusion model. Specifically, in the process of model fusion, the validation dataset is used to quantitatively evaluate the performance of different models, such as key indicators such as accuracy, F1 score, and loss value, and provide a reliable reference for weighted average or other fusion strategies. In addition, the validation dataset can also help technicians discover possible deviations or deficiencies in the model and provide guidance for further optimization of the fusion strategy. For example, when the validation dataset reveals that some models perform poorly on specific tasks, their weights can be appropriately adjusted during fusion to ensure balanced performance of the final fusion model.

[0098] In terms of data preprocessing, it is necessary to perform systematic operations on the training and validation data, including cleaning, deduplication, word segmentation, and labeling. The cleaning step removes noisy data, such as repeated, redundant, or meaningless information; deduplication ensures the independence and efficiency of the data set; word segmentation and labeling provide semantic and structural information for the model. These preprocessing operations not only improve the efficiency and accuracy of model training, but also ensure that the validation data set can accurately reflect the actual performance of the model, laying a solid foundation for subsequent model fusion.

[0099] Step 2: Model selection and preprocessing

[0100] In the model selection and preprocessing stage, it is necessary to screen candidate models from multiple pre-trained large language models. The candidate models in this stage are the multiple pre-trained models obtained based on the target task mentioned above. These models may be built based on different architectures (such as Transformer, RNN, etc.), training data (such as Wikipedia, Common Crawl, etc.) or training strategies (such as autoregression, autoencoding, etc.). Their diversity provides a rich selection space for model fusion. However, in order to ensure the compatibility between models during the fusion process, preprocessing of candidate models (pre-trained models) is essential.

[0101] First, the input format of the candidate models needs to be standardized. For example, different models may have different requirements for the word segmentation method of the input text and the processing of special characters. These differences need to be resolved through a unified preprocessing step. Secondly, the output dimension of the model needs to be unified, which is the basis for the fusion of model parameters. For models with different output layer structures, it may be necessary to make their outputs have consistent dimensions and shapes through interpolation, cropping, or supplementary layers. In addition, in order to reduce the waste of computing resources and improve the fusion effect, it is also necessary to evaluate the parameter scale and computing efficiency of the model to ensure that the selected model can complete training and reasoning under reasonable resource conditions.

[0102] In terms of specific model selection, it is recommended to give priority to selecting candidate models from mature open source model libraries (such as ModelScope and other platforms). These platforms provide a wealth of pre-trained model resources and provide comprehensive tool support, which helps to efficiently complete model selection and preprocessing. In particular, if the goal is to perform model fusion, it is recommended to select models with the same architecture, such as multiple Transformer-based models. This consistency not only simplifies the technical difficulties in the fusion process, but also avoids compatibility issues caused by architectural differences. On the basis of consistency, if model fusion is performed for tasks in a specific field, the final model to be fused can be obtained based on the capabilities of the candidate model in that field.

[0103] Step 3: Parameter fusion

[0104] Parameter fusion is performed by weighted averaging, which is a model parameter fusion method based on the performance indicators of the validation set. This method uses indicators such as accuracy and F1 score as the core basis, assigns weights to each candidate model, and then linearly combines the model parameters through weighted calculation to generate a fused model. The allocation of weights depends not only on the performance on the validation set, but can also be dynamically adjusted according to the needs of specific tasks. For example, in classification tasks that emphasize accuracy, models with higher accuracy can be given greater weights; and in tasks that need to balance accuracy and recall, the F1 score can be given priority. In addition, computing resource limitations are also an important consideration when allocating weights. For example, in a resource-constrained environment, the weights of larger models can be reduced first to reduce computing overhead. The specific weight allocation strategy can refer to the relevant description in the aforementioned step 220, which will not be repeated here.

[0105] Step 4: Layered Fusion

[0106] Hierarchical fusion is a hierarchical optimization method for deep neural networks. It achieves finer-grained model fusion by selecting the optimal model structure or parameters at different levels. A deep model is usually composed of an input layer, a hidden layer, and an output layer, and these layers have different emphases in terms of function and objectives. The specific fusion process can be referred to the relevant description in the aforementioned step 230, which will not be repeated here.

[0107] Step 5: Knowledge Distillation

[0108] Knowledge distillation technology is used to realize knowledge transfer in the fusion model. Its core idea is to improve the performance of the fusion model by transferring the knowledge of multiple teacher models to the student model. In this embodiment, multiple pre-trained models are used as teacher models, and the fused model is used as the student model to complete the training by learning the predicted distribution (soft label) of the teacher model.

[0109] Specifically, the teacher model generates a predicted probability distribution for the same batch of data. For example, for input x, the output of the teacher model is a probability vector P T (x). The goal of the student model is to learn its knowledge by minimizing the difference between the predicted distribution and that of the teacher model, usually achieved through the following loss function:

[0110]

[0111] Where C is the number of categories. S (x) is the predicted distribution of the fusion model, P S (x) i represents the predicted probability of the fusion model for category i, P T (x) i It represents the predicted probability of the pre-trained model for category i.

[0112] Step 6: Fine-tune the fusion model

[0113] In the fusion model fine-tuning stage, fine-tuning techniques are used to optimize the parameters of the fused model. Here, the distilled model obtained after knowledge distillation is fine-tuned. During the fine-tuning process, hyperparameters such as learning rate, batch size, and regularization parameters may need to be adjusted to find the optimal training configuration. At the same time, it is also necessary to monitor the model training process, such as changes in the loss function, performance indicators on the validation set, etc., to ensure that the model can stably converge to the optimal solution.

[0114] Step 7: Performance evaluation and optimization

[0115] In the performance evaluation and optimization phase, an independent test set is used to evaluate the performance of the fusion model. The test set should be independent of the training set and validation set to ensure the objectivity and accuracy of the evaluation results. The effectiveness of the fusion model can be verified by comparing the performance indicators (such as accuracy, F1 score, recall rate, etc.) of the fusion model and the individual candidate models on the test set.

[0116] The model construction method based on model fusion proposed in this application can significantly reduce the demand for training resources. Specifically, through the model fusion method, the capabilities of multiple pre-trained models can be efficiently integrated without retraining to generate a fusion model with excellent performance. This method avoids the high resource consumption of training models from scratch and greatly reduces the computing cost. At the same time, the technical solution of this application can achieve comprehensive capture of diverse languages ​​and semantic patterns. With the help of weighted averaging and hierarchical fusion technology, it can fully tap the unique capture capabilities of different models for language and semantic patterns, thereby showing stronger adaptability and robustness in a multi-task environment. The technical solution of this application is also highly scalable and flexible, supports flexible integration of models of multiple architectures or sources, and can be quickly deployed and applied in different task scenarios without a lot of retraining, and has excellent scalability.

[0117] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0118] The following introduces an embodiment of the device of the present application, which can be used to execute the model building method based on model fusion in the above-mentioned embodiment of the present application. Figure 5 The structural block diagram of the model building device based on model fusion provided in the embodiment of the present application is schematically shown. Figure 5As shown, the model building device based on model fusion provided in the embodiment of the present application includes:

[0119] A pre-training model acquisition module 510 is used to acquire multiple pre-training models according to a target task;

[0120] A weight assignment module 520, configured to assign a weight to the pre-trained model according to a performance evaluation index of the pre-trained model based on the target task;

[0121] A hierarchical fusion module 530 is used to perform hierarchical fusion processing on the multiple pre-trained models according to the weights of each pre-trained model to obtain a fusion model; the hierarchical fusion processing includes performing fusion processing on the multiple pre-trained models for multiple data processing layers respectively;

[0122] The training module 540 is used to perform model training according to the fusion model to obtain a target model, and the target model is used to perform the target task.

[0123] In one embodiment of the present application, the training module 540 is specifically used for:

[0124] The multiple pre-trained models are respectively used as multiple teacher models, and the fusion model is used as a student model to perform knowledge distillation to obtain the target model.

[0125] In one embodiment of the present application, the training module 540 is specifically used for:

[0126] Using the multiple pre-trained models as multiple teacher models respectively and the fusion model as a student model, performing knowledge distillation to obtain a distillation model;

[0127] The model parameters of the distillation model are fine-tuned to obtain the target model.

[0128] In one embodiment of the present application, the weight allocation module 520 is specifically used to:

[0129] Calculate the sum of the indicators of the multiple pre-training models based on the performance evaluation indicators of the target task respectively;

[0130] The weight of the pre-trained model is generated according to the ratio of the performance evaluation index of the target task based on the pre-trained model to the sum of the indexes.

[0131] In one embodiment of the present application, the weight allocation module 520 is specifically used to:

[0132] Obtaining computing resource consumption of the pre-trained model;

[0133] The weight of the pre-trained model is obtained by multiplying the ratio of the performance evaluation index of the pre-trained model based on the target task to the sum of the indexes by the inverse of the computing resource consumption.

[0134] In one embodiment of the present application, the weight allocation module 520 is specifically used to:

[0135] According to the weight of each performance evaluation indicator of the target task, calculate the weighted sum of the first indicators of the multiple pre-training models based on the multiple performance evaluation indicators of the target task respectively;

[0136] Calculate the weighted sum of the second indicator of the pre-trained model based on the multiple performance evaluation indicators of the target task according to the weight of each performance evaluation indicator of the target task;

[0137] The weight of the pre-trained model is generated according to the ratio of the weighted sum of the first indicators to the weighted sum of the second indicators.

[0138] In one embodiment of the present application, the pre-trained model acquisition module 510 is specifically used to:

[0139] Obtain multiple pre-trained models with the same model architecture according to the target task;

[0140] The input data formats and output layer structures of the plurality of pre-trained models are normalized.

[0141] The specific details of the model building device based on model fusion provided in each embodiment of the present application have been described in detail in the corresponding method embodiments and will not be repeated here.

[0142] Figure 6 The structure block diagram of a computer system for implementing an electronic device according to an embodiment of the present application is schematically shown.

[0143] It should be noted that Figure 6 The computer system 600 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0144] like Figure 6As shown, the computer system 600 includes a central processing unit 601 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 602 (ROM) or the program loaded from the storage part 608 to the random access memory 603 (RAM). Various programs and data required for system operation are also stored in the random access memory 603. The central processing unit 601, the read-only memory 602 and the random access memory 603 are connected to each other through a bus 604. An input / output interface 605 (Input / Output interface, i.e., I / O interface) is also connected to the bus 604.

[0145] The following components are connected to the input / output interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a local area network card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

[0146] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processor 601, various functions defined in the system of the present application are executed.

[0147] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0148] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0149] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0150] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0151] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0152] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A model building method based on model fusion, characterized in that: include: Obtain multiple pre-trained models based on the target task; Assigning a weight to the pre-trained model according to a performance evaluation index of the pre-trained model based on the target task; Performing layered fusion processing on the multiple pre-trained models according to the weights of the pre-trained models to obtain a fusion model; the layered fusion processing includes performing fusion processing on the multiple pre-trained models for multiple data processing layers respectively; Model training is performed according to the fusion model to obtain a target model, and the target model is used to execute the target task.

2. The model building method based on model fusion according to claim 1, characterized in that: Model training is performed according to the fusion model to obtain a target model, including: The multiple pre-trained models are respectively used as multiple teacher models, and the fusion model is used as a student model to perform knowledge distillation to obtain the target model.

3. The model building method based on model fusion according to claim 2, characterized in that: The plurality of pre-trained models are respectively used as a plurality of teacher models, and the fusion model is used as a student model to perform knowledge distillation to obtain the target model, including: Using the multiple pre-trained models as multiple teacher models respectively and the fusion model as a student model, performing knowledge distillation to obtain a distillation model; The model parameters of the distillation model are fine-tuned to obtain the target model.

4. The model building method based on model fusion according to claim 1, characterized in that: Allocating a weight to the pre-trained model according to a performance evaluation index of the pre-trained model based on the target task includes: Calculate the sum of the indicators of the multiple pre-training models based on the performance evaluation indicators of the target task respectively; The weight of the pre-trained model is generated according to the ratio of the performance evaluation index of the target task based on the pre-trained model to the sum of the indexes.

5. The model building method based on model fusion according to claim 4, characterized in that: Generating a weight of the pre-trained model according to a ratio of a performance evaluation index of the target task based on the pre-trained model to the sum of the indexes, comprising: Obtaining computing resource consumption of the pre-trained model; The weight of the pre-trained model is obtained by multiplying the ratio of the performance evaluation index of the pre-trained model based on the target task to the sum of the indexes by the inverse of the computing resource consumption.

6. The model building method based on model fusion according to claim 1, characterized in that: Allocating a weight to the pre-trained model according to a performance evaluation index of the pre-trained model based on the target task includes: According to the weight of each performance evaluation indicator of the target task, calculate the weighted sum of the first indicators of the multiple pre-training models based on the multiple performance evaluation indicators of the target task respectively; Calculate the weighted sum of the second indicator of the pre-trained model based on the multiple performance evaluation indicators of the target task according to the weight of each performance evaluation indicator of the target task; The weight of the pre-trained model is generated according to the ratio of the weighted sum of the first indicators to the weighted sum of the second indicators.

7. The model building method based on model fusion according to any one of claims 1 to 6, characterized in that: Get multiple pre-trained models based on the target task, including: Obtain multiple pre-trained models with the same model architecture according to the target task; The input data formats and output layer structures of the plurality of pre-trained models are normalized.

8. A model building device based on model fusion, characterized in that: include: A pre-trained model acquisition module is used to acquire multiple pre-trained models according to the target task; A weight assignment module, used to assign weights to the pre-trained model according to a performance evaluation index of the pre-trained model based on the target task; A hierarchical fusion module, used for performing hierarchical fusion processing on the multiple pre-trained models according to the weights of each pre-trained model to obtain a fusion model; the hierarchical fusion processing includes performing fusion processing on the multiple pre-trained models for multiple data processing layers respectively; The training module is used to perform model training according to the fusion model to obtain a target model, and the target model is used to perform the target task.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model building method based on model fusion described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the executable instructions to enable the electronic device to implement the model building method based on model fusion as described in any one of claims 1 to 7.

Images