Evaluation model training method and device, and electronic device

Abstract

The application relates to an evaluation model training method and device and electronic equipment. The evaluation model training method comprises the following steps: obtaining a plurality of participant local models trained based on transverse federated learning; selecting two models from the plurality of participant local models as competition models, and selecting part or all of the models from the remaining participant local models as referee models; evaluating a target problem based on the competition models and the referee models to obtain an evaluation result corresponding to the target problem, wherein the evaluation result is used to evaluate the accuracy of the competition models in obtaining an answer corresponding to the target problem; based on the evaluation result, adding a corresponding target problem and an answer corresponding to the target problem to an evaluation data training set, and training an evaluation model according to the evaluation data training set. The technical scheme of the application can filter out sample data with high saliency and consistency through the evaluation process of multiple models, thereby improving the accuracy of the evaluation model.

Description

Technical Field

[0001] This application relates to the fields of computer and communication technology, and more specifically, to an evaluation model training method, apparatus, and electronic device. Background Technology

[0002] Federated learning is a method of machine learning that involves collaborating different participants. In federated learning, participants do not need to expose their data to other participants, coordinators, or aggregation servers, thus effectively protecting user privacy and ensuring data security.

[0003] In the era of large models, these models possess powerful and diverse functionalities. On one hand, they have fundamental capabilities in natural language understanding and generation (multimodal), and possess basic knowledge and general understanding across various domains. On the other hand, through fine-tuning, they can acquire domain-specific capabilities. Therefore, evaluating the capabilities of large models has become a crucial issue. Specific large models (such as GPT-3 / 4) are often used to evaluate open-ended problems and have proven effective. However, ordinary large models perform poorly in evaluating open-ended problems. In federated large model scenarios, whether for federated evaluation or federated training, evaluating the general or scenario-specific capabilities of individual models or the trained global model is a challenging problem, given that the application scenarios and target tasks of large models are typically open-ended. Summary of the Invention

[0004] The embodiments of this application provide an evaluation model training method, apparatus, and electronic device, which can improve the accuracy of the evaluation model.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to one aspect of the embodiments of this application, a method for evaluating model training is provided, the method comprising:

[0007] Obtain local models of multiple participants trained based on horizontal federated learning;

[0008] Two models are selected from the multiple local models of the participating parties as the competition models, and some or all of the remaining local models of the participating parties are selected as the referee models.

[0009] The target question is evaluated based on the competition model and the referee model to obtain the evaluation result corresponding to the target question. The evaluation result is used to evaluate the accuracy of the competition model's answer to the target question.

[0010] Based on the evaluation results, the corresponding target questions and their corresponding answers are selected and added to the evaluation data training set, and the evaluation model is trained based on the evaluation data training set.

[0011] According to one aspect of the embodiments of this application, an evaluation model training apparatus is provided, the apparatus comprising:

[0012] The acquisition module is used to acquire local models of multiple participants trained based on horizontal federated learning.

[0013] The selection module is used to select two models from the multiple local models of the participants as competition models, and to select some or all of the remaining local models of the participants as referee models.

[0014] An evaluation module is used to evaluate the target question based on the competition model and the referee model, and obtain an evaluation result corresponding to the target question. The evaluation result is used to evaluate the accuracy of the answer of the competition model to the target question.

[0015] The training module is used to select the corresponding target questions and the answers corresponding to the target questions into the evaluation data training set based on the evaluation results, and to train the evaluation model based on the evaluation data training set.

[0016] In some embodiments of this application, based on the above technical solutions, the acquisition module is further configured to: receive model parameters updated locally from different participants; fuse the updated model parameters from different participants to obtain global model parameters; and distribute the global model parameters to each participant so that each participant can train its local model based on the global model parameters as the latest model parameters of its local model, thereby obtaining multiple participant local models.

[0017] In some embodiments of this application, based on the above technical solutions, the competition model includes a first competition model and a second competition model; the evaluation module is further configured to: input the target question into the first competition model and the second competition model respectively; obtain a first answer corresponding to the target question through the first competition model; obtain a second answer corresponding to the target question through the second competition model; and send the target question, the first answer corresponding to the target question, and the second answer corresponding to the target question to the referee model, so that the referee model evaluates the first answer and the second answer to obtain an evaluation result corresponding to the target question.

[0018] In some embodiments of this application, based on the above technical solutions, there are multiple referee models; the evaluation module is further configured to send the target question, the first answer corresponding to the target question, and the second answer corresponding to the target question to multiple referee models to obtain the evaluation results corresponding to each referee model; and to filter the evaluation results from the evaluation results corresponding to each referee model according to the screening index to obtain the evaluation result corresponding to the target question.

[0019] In some embodiments of this application, based on the above technical solutions, the evaluation module is further configured to: filter the evaluation results from the evaluation results corresponding to each referee model according to the preset weight values ​​of each referee model to obtain a first evaluation result; filter the evaluation results from the evaluation results corresponding to each referee model according to the proportion of valid evaluation results to obtain a second evaluation result; filter the evaluation results from the evaluation results corresponding to each referee model according to the win-loss ratio of the model to obtain a third evaluation result; and obtain an evaluation result corresponding to the target problem based on the first evaluation result, the second evaluation result, and the third evaluation result.

[0020] In some embodiments of this application, based on the above technical solutions, the evaluation module is further configured to: select valid evaluation results from the evaluation results corresponding to each referee model to obtain valid evaluation results; calculate the ratio of valid evaluation results based on the number of valid evaluation results and the number of referee models; and take the evaluation result corresponding to the ratio of valid evaluation results exceeding a set first ratio threshold as the second evaluation result.

[0021] In some embodiments of this application, based on the above technical solutions, the evaluation module is further configured to: determine the number of correct evaluation results of the first competition model or the number of correct evaluation results of the second competition model according to the evaluation results corresponding to each referee model; calculate the ratio of the evaluation results of the first competition model to the total evaluation results, or calculate the ratio of the evaluation results of the second competition model to the total evaluation results; and take the evaluation result corresponding to the calculated ratio exceeding the set second ratio threshold as the third evaluation result.

[0022] In some embodiments of this application, based on the above technical solutions, the training module is further configured to: select the corresponding target question and the answer corresponding to the target question as the evaluation data training set according to the evaluation result; and train a personalized evaluation model for each participant based on the evaluation data training set; or train a global evaluation model based on the evaluation data training set and the global model.

[0023] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the evaluation model training method as described in the above embodiments.

[0024] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more computer programs, wherein when the one or more computer programs are executed by the one or more processors, the electronic device enables the evaluation model training method as described in the above embodiments.

[0025] According to one aspect of the embodiments of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of an electronic device reads from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the evaluation model training methods provided in the various alternative embodiments described above.

[0026] In the technical solution provided in the embodiments of this application, multiple participant local models trained based on horizontal federated learning are first obtained. Each participant's local model is then used to generate evaluation results corresponding to the target problem, and these evaluation results are added to the evaluation data training set to train an evaluation model. In this way, by fully utilizing the knowledge learned by each participant from their own data and combining the comprehensive evaluation results from multiple parties, a more accurate evaluation data training set can be obtained. The evaluation model trained on this accurate evaluation dataset is more accurate. That is, by incorporating a multi-model evaluation process, highly significant and consistent sample data can be selected, thereby improving the accuracy of the evaluation model.

[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0028] Figure 1 A flowchart of an evaluation model training method according to an embodiment of this application is shown.

[0029] Figure 2 A diagram of a horizontal federated learning system architecture according to an embodiment of this application is shown.

[0030] Figure 3 A flowchart of an evaluation model training method according to an embodiment of this application is shown.

[0031] Figure 4 A flowchart of an evaluation model training method according to an embodiment of this application is shown.

[0032] Figure 5 A flowchart of an evaluation model training method according to an embodiment of this application is shown.

[0033] Figure 6 A block diagram of an evaluation model training apparatus according to an embodiment of this application is shown.

[0034] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0035] Exemplary embodiments will now be described in a more comprehensive manner with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be construed as limited to these examples; rather, these embodiments are provided so that this application will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art.

[0036] Furthermore, the features, structures, or characteristics described in this application can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to provide a full understanding of the embodiments of this application. However, those skilled in the art will recognize that when implementing the technical solutions of this application, not all the detailed features in the embodiments may be used, one or more specific details may be omitted, or other methods, elements, devices, steps, etc., may be employed.

[0037] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0038] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0039] It should be noted that "multiple" in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0040] It is understood that in the specific implementation of this application, data related to target object information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0041] There are two main methods for evaluating model capabilities in related technical solutions. The first method involves providing a so-called standard answer manually. During evaluation, the model's generated answer is compared with the given standard answer to determine if their characters match one-to-one. However, this method is not very effective for open-ended questions. The second method utilizes external models for evaluation. This requires sending the test question to the external model for assessment. Because this method relies on an external model, meaning data needs to be sent to it, there is a risk of data leakage.

[0042] Based on this, embodiments of this application provide a novel evaluation model training method, which includes: obtaining multiple participant local models trained based on lateral federated learning; selecting two models from the multiple participant local models as competition models, and selecting some models from the remaining participant local models as referee models; evaluating a target problem based on the competition models and referee models to obtain an evaluation result corresponding to the target problem, wherein the evaluation result is used to evaluate the accuracy of the competition model for the answer to the target problem; and based on the evaluation result, selecting the corresponding target problem and the answer corresponding to the target problem to add to the evaluation data training set, and training the evaluation model based on the evaluation data training set.

[0043] In the technical solution of this application, multiple local models of participating parties, trained based on horizontal federated learning, are first obtained. Each participating party's local model is then used to generate evaluation results corresponding to the target problem, and these evaluation results are added to the evaluation data training set to train the evaluation model. In this way, by fully utilizing the knowledge learned by each participating party from its own data and combining the comprehensive evaluation results from multiple parties, a more accurate evaluation data training set can be obtained. The evaluation model trained on this accurate evaluation dataset is more accurate. That is, by incorporating a multi-model evaluation process, highly significant and consistent sample data can be selected, thereby improving the accuracy of the evaluation model.

[0044] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0045] Figure 1A flowchart of an evaluation model training method according to an embodiment of this application is shown. This evaluation model training method can be executed by a server. (Refer to...) Figure 1 As shown, the training method for this evaluation model includes at least S101 to S104, which are described in detail below:

[0046] In S101, local models of multiple participants are obtained based on horizontal federated learning training.

[0047] In some optional embodiments, horizontal federated learning involves extracting the data from participants with similar features but less overlap in user data, and then performing joint machine learning on that portion of the data. For example, two banks located in different regions have user groups from their respective regions with little overlap. However, their businesses are very similar, and a large portion of their recorded user data features are the same. Therefore, horizontal federated learning can be used to help the two banks build a joint model to predict their respective customer behaviors. In this embodiment, by first obtaining multiple local models from participants trained based on horizontal federated learning, it is beneficial to fully utilize the knowledge learned by each participant from their respective data, and combine the comprehensive evaluation results from multiple parties to obtain a more accurate evaluation data training set.

[0048] In S102, two models are selected from the local models of multiple participants as competition models, and some or all of the remaining local models of participants are selected as referee models.

[0049] In some optional embodiments, two local models from multiple participating parties are randomly selected as competition models. Subsequently, based on ranking weights, the model with the higher ranking is selected as the judge model. For example, any two models can be selected from multiple participating parties' local models as competition models, and then K models can be selected as judge models from the remaining models. During the selection of the K models, different weight values ​​are pre-set for each judge model, and the judge models are selected sequentially according to their weight values. Here, 2 + K <= M, where M is the total number of participating parties' local models. The selection of the number of judge models can be made by those skilled in the art according to actual needs and is not limited here.

[0050] In S103, the target problem is evaluated based on the competition model and the referee model to obtain the evaluation result corresponding to the target problem. The evaluation result is used to evaluate the accuracy of the competition model for the corresponding answer to the target problem.

[0051] In some optional embodiments, the target problem can be an additional dataset containing only the problem in the corresponding domain, generated by a large model based on a public validation / test dataset or a public dataset. Based on this additional dataset, additional training problems are generated by the large model participating in the training, thus obtaining the target problem. After obtaining the target problem, evaluation results corresponding to the target problem are obtained through a competition model and a referee model.

[0052] In S104, based on the evaluation results, the corresponding target questions and their corresponding answers are selected and added to the evaluation data training set, and the evaluation model is trained based on the evaluation data training set.

[0053] In some optional embodiments, after obtaining the evaluation results, the question data corresponding to the evaluation results and the answers corresponding to the question data are used as the evaluation data training set. The evaluation data training set is then used as training samples to train the model to obtain the evaluation model. In this way, by fully utilizing the knowledge learned by each participant from their respective data and combining the comprehensive evaluation results from multiple parties, a more accurate evaluation data training set can be obtained. The evaluation model trained on an accurate evaluation dataset is more accurate.

[0054] In the technical solution provided in the embodiments of this application, multiple participant local models trained based on horizontal federated learning are first obtained. Each participant's local model is then used to generate evaluation results corresponding to the target problem, and these evaluation results are added to the evaluation data training set to train an evaluation model. In this way, by fully utilizing the knowledge learned by each participant from their own data and combining the comprehensive evaluation results from multiple parties, a more accurate evaluation data training set can be obtained. The evaluation model trained on this accurate evaluation dataset is more accurate. That is, by incorporating a multi-model evaluation process, highly significant and consistent sample data can be selected, thereby improving the accuracy of the evaluation model.

[0055] In some optional embodiments, in S101, obtaining the local models of multiple participants trained based on lateral federated learning includes the following steps:

[0056] Receive model parameters updated locally from different participants;

[0057] The updated model parameters from different participants are merged to obtain the global model parameters;

[0058] The global model parameters are distributed to each participant so that each participant can train its local model using the global model parameters as the latest model parameters, resulting in multiple local models for each participant.

[0059] Specifically, see Figure 2 , Figure 2A diagram illustrating a horizontal federated learning system architecture according to an embodiment of this application is provided. This system architecture includes multiple participants, such as Participant1, Participant2, ..., ParticipantK, and a coordinator. Specifically, its operation involves the following steps: ① After a participant A updates its model parameters locally, participant A can send the updated model parameters to the coordinator. Participant A can send the model parameter updates to the coordinator using encryption, for example, homomorphic encryption. The model parameters can be parameters of the federated learning model, such as the weight parameters connecting nodes in a neural network; or, the joint model parameters can be gradient information of the federated learning model, such as the gradient information in a neural network gradient descent algorithm. ② The coordinator merges the received model parameter updates from different participants, for example, by calculating a weighted average. ③ The coordinator redistributes the merged model parameter updates (also called global model parameters) to the participating participants. The coordinator can also transmit the global model parameters using encryption. Step 4: Participants can use the received global model parameters as the starting point for local model training or as the latest model parameters to begin training or continue training the model locally. Participants and the coordinator iterate through the above steps until the loss function converges, the model parameters converge, the maximum number of iterations is reached, or the maximum training time is reached, thus completing the entire model training process.

[0060] In this way, by first obtaining the local models of multiple participants trained based on horizontal federated learning, it is beneficial to make full use of the knowledge learned by each participant from their respective data, so as to combine the comprehensive evaluation results of multiple parties and obtain a more accurate evaluation data training set.

[0061] In some optional embodiments, the competition model includes a first competition model and a second competition model;

[0062] See Figure 3 , Figure 3 A flowchart of an evaluation model training method according to an embodiment of this application is shown. In S103, the target problem is evaluated based on the competition model and the referee model to obtain an evaluation result corresponding to the target problem, including:

[0063] In S301, the target question is input into the first competition model and the second competition model respectively. The first competition model is used to obtain the first answer corresponding to the target question, and the second competition model is used to obtain the second answer corresponding to the target question.

[0064] In S302, the target question, the first answer corresponding to the target question, and the second answer corresponding to the target question are sent to the referee model so that the referee model can evaluate the first answer and the second answer to obtain the evaluation result corresponding to the target question.

[0065] To facilitate understanding of the embodiments of this application, an example is given below. For instance, the target problem is Q, the first competition model is M1, and the second competition model is M2. Q is input into M1, yielding A1 as the first answer. Q is input into M2, yielding A2 as the second answer. After obtaining A1 and A2, (Q, A1) and (Q, A2) are sent to the referee model. The referee model evaluates A1 and A2. Since the referee model has an automatic judgment function, by inputting Q, A1, and A2 into the referee model, it can automatically determine which answer, A1 or A2, is more accurate for the target problem Q, thereby obtaining the output result.

[0066] In this way, by inputting the target question into the first competition model and the second competition model respectively, the corresponding first answer and second answer are obtained. Then, the first answer and the second answer are evaluated by the judge model to obtain the evaluation result corresponding to the target question, which is conducive to the accuracy of the evaluation result corresponding to the target question.

[0067] In some optional embodiments, there are multiple referee models. See also Figure 4 , Figure 4 A flowchart of an evaluation model training method according to an embodiment of this application is shown. In S302, the target question, a first answer corresponding to the target question, and a second answer corresponding to the target question are sent to the referee model so that the referee model evaluates the first answer and the second answer to obtain an evaluation result corresponding to the target question, including:

[0068] In S401, the target question, the first answer corresponding to the target question, and the second answer corresponding to the target question are all sent to multiple judge models to obtain the evaluation results corresponding to each judge model.

[0069] When there are multiple referee models, such as K1, K2, and K3, and the target problem is Q, the first competition model is M1, and the second competition model is M2. Q is input into M1, yielding A1 as the first answer; Q is input into M2, yielding A2 as the second answer. After obtaining A1 and A2, (Q, A1) and (Q, A2) are sent to K1 to obtain the evaluation result R1. (Q, A1) and (Q, A2) are sent to K2 to obtain the evaluation result R2. (Q, A1) and (Q, A2) are sent to K3 to obtain the evaluation result R3.

[0070] In S402, the evaluation results are filtered from the evaluation results corresponding to each referee model according to the screening indicators to obtain the evaluation results corresponding to the target problem.

[0071] After obtaining the evaluation results corresponding to each referee model, the evaluation results are filtered according to screening criteria to obtain the evaluation results corresponding to the target problem. Screening criteria may include, for example, the weight values ​​of each referee model, the proportion of valid evaluation results, and the win-loss ratio of the referee model. Of course, other conditions can also be set for the screening criteria, and those skilled in the art can set them according to actual needs; this is not limited here. Taking the above example, after obtaining R1, R2, and R3, R1, R2, and R3 are filtered according to the set screening criteria to select the more accurate evaluation results.

[0072] In this way, by sending the target question, the first answer corresponding to the target question, and the second answer corresponding to the target question to multiple judge models, the evaluation results corresponding to each judge model are obtained. Then, the evaluation results corresponding to each judge model are filtered according to the screening index to obtain the evaluation results corresponding to the target question. This helps to screen out sample data with high significance and consistency, thereby improving the accuracy of the evaluation model.

[0073] In some alternative embodiments, see Figure 5 , Figure 5 A flowchart of an evaluation model training method according to an embodiment of this application is shown. In S402, the evaluation results are filtered from the evaluation results corresponding to each referee model according to a screening criterion to obtain the evaluation results corresponding to the target problem, including:

[0074] In S501, the evaluation results are filtered from the evaluation results corresponding to each referee model according to the preset weight values ​​of each referee model to obtain the first evaluation result.

[0075] When setting the screening criteria, you can set them according to the weight values ​​of each judge model. The weight of the judge model can be determined based on the existing model ranking or contribution, such as the accuracy of validation in public data, model size, data volume, and other indicators.

[0076] In S502, the evaluation results are filtered from the evaluation results corresponding to each referee model according to the proportion of valid evaluation results to obtain the second evaluation result.

[0077] Optionally, when setting screening criteria, the evaluation results can also be screened based on the proportion of valid evaluation results.

[0078] In S503, the evaluation results are filtered from the evaluation results corresponding to each referee model according to the win-loss ratio of the model to obtain the third evaluation result.

[0079] In S504, the evaluation results corresponding to the target problem are obtained based on the first evaluation results, the second evaluation results, and the third evaluation results.

[0080] After obtaining the first, second, and third evaluation results, these results are integrated to obtain the evaluation result corresponding to the target problem. When integrating the first, second, and third evaluation results, for example, the union of these three evaluation results can be taken to obtain the final evaluation result.

[0081] In this way, by setting screening indicators across different dimensions, evaluation results corresponding to different dimensions can be obtained. By aggregating the evaluation results from different dimensions, a more accurate evaluation result corresponding to the target problem can be obtained.

[0082] In some optional embodiments, the evaluation results are filtered from the evaluation results corresponding to each referee model according to the proportion of valid evaluation results to obtain a second evaluation result, including:

[0083] Select the valid evaluation results from the evaluation results corresponding to each referee model to obtain the valid evaluation results;

[0084] The proportion of valid evaluation results is calculated based on the number of valid evaluation results and the number of referee models.

[0085] The evaluation result corresponding to the proportion of valid evaluation results exceeding the set first proportion threshold is taken as the second evaluation result.

[0086] For example, there are 5 judging models, 4 of which are valid results. Taking one judging model K1 as an example, for judging valid results, Q is the target problem, A1 is the first answer to the target problem obtained using the first competition model, and A2 is the second answer to the target problem obtained using the second competition model. Assuming that (Q, A1) and (Q, A2) are sent to K1 in sequence, the evaluation result is R1. However, if (Q, A2) and (Q, A1) are sent to K1 in sequence, the evaluation result is R2. This means that the result changes after the order of the input data changes, and the result is considered valid. Conversely, if (Q, A1) and (Q, A2) are sent to K1 in sequence, the evaluation result is R1. If (Q, A2) and (Q, A1) are sent to K1 in sequence, the evaluation result is also R1. Since the result remains the same regardless of the order, the result is considered invalid. Based on the number of valid results and the number of referee models, the proportion of valid evaluation results is 4 / 5 = 0.8. Assuming the first proportion threshold is set to 70%, since the proportion of valid evaluation results exceeds 70%, these valid evaluation results that exceed the threshold are retained as the second evaluation results.

[0087] In this way, by filtering the evaluation results from the evaluation results corresponding to each judge model according to the proportion of valid evaluation results, a more accurate evaluation result corresponding to the target problem can be obtained.

[0088] In some optional embodiments, the evaluation results are filtered from the evaluation results corresponding to each referee model based on the model's win-loss ratio to obtain a third evaluation result, including:

[0089] The number of correct evaluation results for the first competition model or the number of correct evaluation results for the second competition model are determined based on the evaluation results corresponding to each referee model.

[0090] Calculate the ratio of the evaluation result of the first competition model to the total evaluation result, or calculate the ratio of the evaluation result of the second competition model to the total evaluation result;

[0091] The evaluation result corresponding to the calculated proportion exceeding the set second proportion threshold is taken as the third evaluation result.

[0092] For example, if there are 4 evaluation results, 3 of which indicate that the first competition model is correct (meaning the first competition model wins), and 1 of which indicates that the second competition model is correct, then the win rate is 3 / (3+1) = 0.75. Assuming the second ratio threshold is set to 70%, since the win-loss ratio exceeds 70%, the correct evaluation result is retained as the third evaluation result.

[0093] In this way, by filtering the evaluation results from the various referee models according to the win-loss ratio of each model, a more accurate evaluation result corresponding to the target problem can be obtained.

[0094] In some optional embodiments, based on the evaluation results, the corresponding target questions and their corresponding answers are selected and added to the evaluation data training set, and an evaluation model is trained based on the evaluation data training set, including:

[0095] Based on the evaluation results, select the corresponding target questions and their corresponding answers as the evaluation data training set, and train personalized evaluation models for each participant based on the evaluation data training set; or train a global evaluation model based on the evaluation data training set and the global model.

[0096] For example, suppose the referee model is K1, the target question is Q, the first competition model is M1, and the second competition model is M2. Inputting Q into M1 yields A1 as the first answer, and inputting Q into M2 yields A2 as the second answer. After obtaining A1 and A2, (Q, A1) and (Q, A2) are sent to K1 to obtain the evaluation result R1. After obtaining the evaluation result, the evaluation result R1, the corresponding question Q, and the first answer A1 and second answer A2 for Q in the first competition model are used as training samples, i.e., as the evaluation data training set. The evaluation model is then trained based on this evaluation data training set.

[0097] In this way, the local personalized evaluation models trained by each party can improve the accuracy and reliability of the evaluation results when evaluating the answers. In addition, if a global evaluation model is trained based on the evaluation data training set and the global model, its independence can be achieved by selecting a large model with a different structure or training set than the large model participating in the evaluation, since it is similar to an independent third party. The large model participating in the evaluation can provide a fairer evaluation result.

[0098] The following describes an embodiment of the apparatus described in this application, which can be used to execute the evaluation model training method described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the evaluation model training method described above.

[0099] Figure 6 A block diagram of an evaluation model training apparatus according to an embodiment of this application is shown.

[0100] Reference Figure 6 As shown, an evaluation model training apparatus 600 according to an embodiment of this application includes:

[0101] Module 601 is used to acquire local models of multiple participants trained based on horizontal federated learning.

[0102] The selection module 602 is used to select two models from multiple local models of the participants as competition models, and to select some or all of the remaining local models of the participants as referee models.

[0103] Evaluation module 603 is used to evaluate the target question based on the competition model and the referee model, and obtain the evaluation result corresponding to the target question. The evaluation result is used to evaluate the accuracy of the competition model for the corresponding answer to the target question.

[0104] Training module 604 is used to select the corresponding target questions and their corresponding answers to the evaluation data training set based on the evaluation results, and to train the evaluation model based on the evaluation data training set.

[0105] In some embodiments of this application, based on the above technical solutions, the acquisition module 601 is further configured to: receive model parameters updated locally from different participants; fuse the updated model parameters from different participants to obtain global model parameters; and distribute the global model parameters to each participant so that each participant can train its local model based on the global model parameters as the latest model parameters of its local model, thereby obtaining multiple local models for each participant.

[0106] In some embodiments of this application, based on the above technical solutions, the competition model includes a first competition model and a second competition model; the evaluation module 603 is further configured to input the target question into the first competition model and the second competition model respectively, obtain a first answer corresponding to the target question through the first competition model, obtain a second answer corresponding to the target question through the second competition model; and send the target question, the first answer corresponding to the target question, and the second answer corresponding to the target question to the referee model so that the referee model evaluates the first answer and the second answer to obtain an evaluation result corresponding to the target question.

[0107] In some embodiments of this application, based on the above technical solutions, there are multiple referee models; the evaluation module 603 is further configured to send the target question, the first answer corresponding to the target question, and the second answer corresponding to the target question to multiple referee models to obtain the evaluation results corresponding to each referee model; and to filter the evaluation results from the evaluation results corresponding to each referee model according to the screening indicators to obtain the evaluation results corresponding to the target question.

[0108] In some embodiments of this application, based on the above technical solutions, the evaluation module 603 is further configured to: filter the evaluation results from the evaluation results corresponding to each referee model according to the preset weight values ​​of each referee model to obtain a first evaluation result; filter the evaluation results from the evaluation results corresponding to each referee model according to the proportion of effective evaluation results to obtain a second evaluation result; filter the evaluation results from the evaluation results corresponding to each referee model according to the win-loss ratio of the model to obtain a third evaluation result; and obtain an evaluation result corresponding to the target problem based on the first evaluation result, the second evaluation result, and the third evaluation result.

[0109] In some embodiments of this application, based on the above technical solutions, the evaluation module 603 is further configured to: select valid evaluation results from the evaluation results corresponding to each referee model to obtain valid evaluation results; calculate the proportion of valid evaluation results based on the number of valid evaluation results and the number of referee models; and take the evaluation result corresponding to the proportion of valid evaluation results exceeding a set first proportion threshold as the second evaluation result.

[0110] In some embodiments of this application, based on the above technical solutions, the evaluation module 603 is further configured to: determine the number of correct evaluation results of the first competition model or the number of correct evaluation results of the second competition model according to the evaluation results corresponding to each referee model; calculate the ratio of the evaluation results of the first competition model to the total evaluation results, or calculate the ratio of the evaluation results of the second competition model to the total evaluation results; and take the evaluation result corresponding to the calculated ratio exceeding the set second ratio threshold as the third evaluation result.

[0111] In some embodiments of this application, based on the above technical solutions, the training module 604 is further configured to select the corresponding target question and the answer corresponding to the target question as the evaluation data training set according to the evaluation results, and train a personalized evaluation model for each participant based on the evaluation data training set; or train a global evaluation model based on the evaluation data training set and the global model.

[0112] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0113] It should be noted that, Figure 7 The computer system 700 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0114] like Figure 7As shown, the computer system 700 includes a Central Processing Unit (CPU) 701, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 702 or programs loaded from storage portion 708 into Random Access Memory (RAM) 703. The RAM 703 also stores various programs and data required for system operation. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An Input / Output (I / O) interface 705 is also connected to the bus 704.

[0115] The following components are connected to I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 710 as needed so that computer programs read from it can be installed into storage section 708 as needed.

[0116] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs various functions defined in the system of this application.

[0117] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and a computer program.

[0119] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0120] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more computer programs that, when executed by the electronic device, cause the electronic device to implement the methods described in the above embodiments.

[0121] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0122] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0123] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0124] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An evaluation model training method applied to a server, characterized in that, The method includes: Obtain local models of multiple participants trained based on horizontal federated learning; Two models are selected from the multiple local models of the participating parties as the first competition model and the second competition model, respectively, and some or all of the remaining local models of the participating parties are selected as the referee models. The target question is input into the first competition model and the second competition model respectively. The first competition model is used to obtain the first answer corresponding to the target question, and the second competition model is used to obtain the second answer corresponding to the target question. The target question, the first answer corresponding to the target question, and the second answer corresponding to the target question are all sent to multiple judge models to obtain the evaluation results corresponding to each judge model; The evaluation results are filtered from the evaluation results corresponding to each of the referee models according to the screening criteria to obtain the evaluation results corresponding to the target question. The evaluation results are used to evaluate the accuracy of the competition model's answer to the target question. Based on the evaluation results, the corresponding target questions and their corresponding answers are selected and added to the evaluation data training set, and the evaluation model is trained based on the evaluation data training set.

2. The method of claim 1, wherein, The acquisition of multiple participant local models trained based on lateral federated learning includes: Receive model parameters updated locally from different participants; The updated model parameters from different participants are merged to obtain the global model parameters; The global model parameters are distributed to each participant, so that each participant can train its local model using the global model parameters as the latest model parameters, thereby obtaining multiple participant local models.

3. The method of claim 1, wherein, The evaluation results are filtered from the evaluation results corresponding to each of the various referee models according to the screening criteria to obtain the evaluation results corresponding to the target problem, including: The evaluation results are filtered from the evaluation results corresponding to each of the referee models according to the preset weight values ​​of each referee model to obtain the first evaluation result; The evaluation results are filtered from the evaluation results corresponding to each of the referee models according to the proportion of valid evaluation results to obtain the second evaluation result; The evaluation results are filtered from the evaluation results corresponding to each referee model according to the win-loss ratio of the model to obtain the third evaluation result; Based on the first evaluation result, the second evaluation result, and the third evaluation result, an evaluation result corresponding to the target problem is obtained.

4. The method of claim 3, wherein, The evaluation results are filtered from the evaluation results corresponding to each of the aforementioned referee models based on the proportion of valid evaluation results to obtain the second evaluation result, which includes: Select the valid evaluation results from the evaluation results corresponding to each of the aforementioned referee models to obtain the valid evaluation results; Calculate the proportion of valid evaluation results based on the number of valid evaluation results and the number of referee models; The evaluation result corresponding to the proportion of the effective evaluation result exceeding the set first proportion threshold is taken as the second evaluation result.

5. The method of claim 3, wherein, The evaluation results are filtered from the evaluation results corresponding to each of the aforementioned referee models based on the win-loss ratio of the models, resulting in a third evaluation result, including: The number of correct evaluation results of the first competition model or the number of correct evaluation results of the second competition model are determined based on the evaluation results corresponding to each referee model. Calculate the ratio of the evaluation result of the first competition model to the total evaluation result, or calculate the ratio of the evaluation result of the second competition model to the total evaluation result; The evaluation result corresponding to the calculated proportion exceeding the set second proportion threshold is taken as the third evaluation result. 6.The method of Claim 1, wherein Based on the evaluation results, the corresponding target questions and their corresponding answers are selected and added to the evaluation data training set. An evaluation model is then trained using the evaluation data training set, including: Based on the evaluation results, select the corresponding target questions and their corresponding answers as the evaluation data training set, and train personalized evaluation models for each participant based on the evaluation data training set; or A global evaluation model is obtained by training the evaluation data training set and the global model.

7. An evaluation model training apparatus characterized by comprising: The device includes: The acquisition module is used to acquire local models of multiple participants trained based on horizontal federated learning. The selection module is used to select two models from the multiple local models of the participants as the first competition model and the second competition model, and to select some or all of the remaining local models of the participants as the referee models. An evaluation module is used to input a target question into a first competition model and a second competition model, respectively; obtain a first answer corresponding to the target question through the first competition model; obtain a second answer corresponding to the target question through the second competition model; send the target question, the first answer, and the second answer to multiple referee models to obtain evaluation results for each referee model; filter the evaluation results from the evaluation results of each referee model according to screening indicators to obtain the evaluation result corresponding to the target question, and the evaluation result is used to evaluate the accuracy of the competition model for the answer corresponding to the target question; The training module is used to select the corresponding target question and the answer corresponding to the target question into the evaluation data training set based on the evaluation results, and to train the evaluation model based on the evaluation data training set.

8. An electronic device, comprising: include: One or more processors; A memory for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the evaluation model training method as described in any one of claims 1 to 6.

Images