Model evaluation method and apparatus, device, and medium

Abstract

The embodiments of the present disclosure relate to a model evaluation method and apparatus, a device, and a medium. The method comprises: acquiring a plurality of models to be evaluated and historical evaluation results; on the basis of said plurality of models and the historical evaluation results, generating a plurality of evaluation tasks, wherein each evaluation task defines that two models to be evaluated undergo comparative evaluation; generating a plurality of pieces of evaluation data corresponding to the plurality of evaluation tasks; distributing the plurality of pieces of evaluation data to evaluation users for evaluation, so as to obtain a plurality of single-evaluation results corresponding to the plurality of evaluation tasks; on the basis of initial scores of said plurality of models and the plurality of single-evaluation results, and by using a game-playing level scoring method, determining current scores of said plurality of models; and on the basis of the current scores of said plurality of models, determining a model ranking result.

Description

A model evaluation method, apparatus, equipment and medium

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411027113.0, filed on July 29, 2024, entitled "A Model Evaluation Method, Apparatus, Device and Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of computer technology, and in particular to a model evaluation method, apparatus, device and medium. Background Technology

[0004] With the development of technology, models such as text-based image models and text-based video models can be evaluated on model evaluation platforms. Summary of the Invention

[0005] This disclosure provides a model evaluation method, apparatus, equipment, and medium.

[0006] This disclosure provides a model evaluation method, the method comprising:

[0007] Obtain multiple models to be tested and historical evaluation results;

[0008] Multiple evaluation tasks are generated based on the multiple models to be tested and the historical evaluation results, wherein each evaluation task defines two models to be tested for comparative evaluation;

[0009] Generate multiple evaluation data corresponding to the multiple evaluation tasks;

[0010] The multiple evaluation data are distributed to evaluation users for evaluation, resulting in multiple single evaluation results corresponding to the multiple evaluation tasks;

[0011] The current scores of the multiple models under test are determined using a game-level scoring method based on the initial scores of the multiple models under test and the multiple single evaluation results.

[0012] The model ranking result is determined based on the current scores of the multiple models to be tested.

[0013] This disclosure also provides a model evaluation apparatus, the apparatus comprising:

[0014] The acquisition module is used to acquire multiple models to be tested and historical evaluation results;

[0015] The task module is used to generate multiple evaluation tasks based on the multiple models to be tested and the historical evaluation results, wherein each evaluation task defines two models to be tested for comparative evaluation.

[0016] The data module is used to generate multiple evaluation data corresponding to the multiple evaluation tasks;

[0017] The evaluation module is used to distribute the multiple evaluation data to evaluation users for evaluation, and obtain multiple single evaluation results corresponding to the multiple evaluation tasks;

[0018] The scoring module is used to determine the current score of the multiple models under test based on the initial scores of the multiple models under test and the multiple single evaluation results using a game-level scoring method;

[0019] The sorting module is used to determine the model sorting result based on the current scores of the multiple models to be tested.

[0020] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the model evaluation method provided in this disclosure.

[0021] This disclosure also provides a computer-readable storage medium storing a computer program for executing the model evaluation method provided in this disclosure. Attached Figure Description

[0022] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0023] Figure 1 is a flowchart illustrating a model evaluation method provided in an embodiment of this disclosure;

[0024] Figure 2 is a schematic diagram of an evaluation page provided in an embodiment of this disclosure;

[0025] Figure 3 is a schematic diagram of a model ranking result provided in an embodiment of this disclosure;

[0026] Figure 4 is a schematic diagram of another model sorting result provided by an embodiment of this disclosure;

[0027] Figure 5 is a schematic diagram of a model evaluation process provided in an embodiment of this disclosure;

[0028] Figure 6 is a schematic diagram of another model evaluation process provided in an embodiment of this disclosure;

[0029] Figure 7 is a schematic diagram of a model evaluation device provided in an embodiment of this disclosure;

[0030] Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0031] In the technology of evaluating text-based image models, text-based video models, and other related models on model evaluation platforms, the evaluation tasks of the model evaluation platforms are usually created manually, which is inefficient, has poor usability, and has low reliability in comparing and ranking different models, and needs to be improved.

[0032] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0033] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0034] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0035] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0036] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0037] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0038] In the field of model evaluation, one type of platform requires users to register with their real names and undergo verification before becoming evaluation users. Evaluation users create evaluation tasks manually offline by uploading code and models. The platform then calculates a ranking list of different models based on the accumulated evaluation results from various users, using a win rate score. However, this method requires real-name authentication, has a high user screening threshold, a lengthy task creation process, poor usability, and is unfriendly to users who cannot code. Furthermore, the scoring method uses traditional win rates, resulting in low confidence. Another type of model evaluation platform allows users to anonymously vote for better models. The platform calculates a model ranking list using a more authoritative scoring method based on accumulated user match results. However, this method has limitations in supported languages, lower usability, and also lower confidence.

[0039] To address the aforementioned issues, this disclosure provides a model evaluation method, which will be described below with reference to specific embodiments.

[0040] Figure 1 is a flowchart illustrating a model evaluation method provided in an embodiment of this disclosure. This method can be executed by a model evaluation device, which can be implemented using software and / or hardware and is generally integrated into an electronic device. As shown in Figure 1, the method includes:

[0041] Step 101: Obtain multiple models to be tested and historical evaluation results.

[0042] The model evaluation method of this disclosure can be executed by a model evaluation platform, which may include modules such as a user management module, an evaluation module, and a calculation module. Through this model platform, pairwise model comparison evaluation and ranking with high ease of use and confidence can be achieved.

[0043] The model to be tested can be a visual model whose functionality needs to be evaluated. In this embodiment, the model to be tested can include a text-to-image model and / or a text-to-video model. The text-to-image model can be a model that generates images based on text, and the text-to-video model can be a model that generates videos based on text. This embodiment does not limit the language of the model to be tested; for example, the model to be tested can include models in different languages ​​such as Chinese and English. The number of models to be tested is unlimited. In this embodiment, the evaluation of the models to be tested is a pairwise comparison, also known as model battle. Historical evaluation results can be all historical evaluation results for multiple models to be tested. These historical evaluation results can include multiple historical single evaluation results. Each single evaluation result refers to the specific evaluation result of two outputs of two models to be tested for the same prompt word, judged by an evaluator to determine the degree of matching with the prompt word and which generated result is better. This can include the model names of the two models to be tested, the prompt word, the two outputs, the evaluator, and the specific evaluation result.

[0044] The model evaluation device can acquire multiple test models input by the user in the evaluation control system, and retrieve stored historical evaluation results. If no historical evaluation results exist and the system is in a cold start state, then retrieving those historical evaluation results is unnecessary. Furthermore, the model evaluation platform can also determine the number of evaluations for multiple test models. This number controls the amount of evaluation data used in the current evaluation; setting this number is optional and can be adjusted according to the specific circumstances.

[0045] Step 102: Generate multiple evaluation tasks based on multiple models to be tested and historical evaluation results. Each evaluation task defines two models to be tested for comparison and evaluation.

[0046] Among them, the evaluation task can be a task automatically generated by the model evaluation platform for multiple models to be tested, which determines which two models to be compared and evaluated. It can also be called a competition, and there are multiple evaluation tasks.

[0047] In some embodiments, generating multiple evaluation tasks based on multiple models to be tested and historical evaluation results may include: determining multiple confidence interval reduction amounts for multiple model pairs composed of pairs of models to be tested based on historical evaluation results; and generating multiple evaluation tasks based on the multiple confidence interval reduction amounts for multiple model pairs.

[0048] A model pair can include two models to be tested. Multiple model pairs are obtained by combining multiple models in pairs. Each evaluation task corresponds to one model pair. The specific number of model pairs is determined by the number of models to be tested; for example, if there are 3 models to be tested, there are 3 model pairs. A model's confidence interval can be an interval estimate calculated from sample data. It includes the probability of the model's current score being the true value, representing a specific numerical range. The confidence level indicates the probability that the model's current score falls within the confidence interval. The confidence interval reduction can be the specific numerical reduction in the confidence interval of a model after one comparative evaluation. The confidence interval reduction of a model pair can be the average of the confidence interval reductions of the two models in the pair.

[0049] After acquiring historical evaluation results, the model evaluation device can first determine the confidence interval reduction amount for each model pair based on the historical evaluation results. Based on the multiple confidence interval reduction amounts of multiple model pairs, multiple evaluation tasks are generated. The larger the confidence interval reduction amount of a model pair, the earlier the corresponding evaluation task is generated, indicating that the model pair should be compared first and is selected first to generate the corresponding evaluation task. For example, the evaluation task can be generated by sorting multiple model pairs in descending order of confidence interval reduction amount, and generating the evaluation task for each model pair in sequence according to the sorting result. This is just an example.

[0050] In the above scheme, the model evaluation platform can automatically generate corresponding evaluation tasks based on the changes in the confidence interval of the model under test in the current evaluation, according to the historical evaluation results and the current evaluation target. The evaluation task creation process is simple, avoids code editing, improves usability, and the generated evaluation tasks are constantly changing as the historical evaluation results are constantly updated, which improves the adaptability of model evaluation.

[0051] Step 103: Generate multiple evaluation data for multiple evaluation tasks.

[0052] The evaluation data can be questions that specifically evaluate the two models to be tested for the evaluation task, as well as the results of the two models to be tested for the questions. An evaluation task can correspond to at least one prompt word, that is, an evaluation task corresponds to at least one set of evaluation data.

[0053] Specifically, the model evaluation platform can use the built-in evaluation data management module to randomly select at least one question for each evaluation task for multiple evaluation tasks, and input each question into the two test models corresponding to the evaluation task to obtain the corresponding results. One question and two results are determined as one evaluation data, resulting in multiple evaluation data.

[0054] In some embodiments, generating multiple evaluation data corresponding to multiple evaluation tasks may include: extracting corresponding prompt words for each evaluation task; for each evaluation task, inputting the prompt words of the evaluation task into two corresponding models to be tested to obtain two output results, and determining the prompt words and the two output results as the corresponding evaluation data.

[0055] Prompts are contextual information generated by accurately translating user requirements into a form that the model can effectively understand and process. Prompts represent specific questions for the model under test. In this embodiment, the model evaluation platform can include a prompt library containing multiple prompts, and the languages ​​of these prompts are not limited. The output can be the result obtained by the model under test through analysis and processing based on the prompts. Specifically, the output varies depending on the function of the model under test. When the model under test is a text-to-image model, the output can be an image; when the model under test is a text-to-video model, the output can be a video.

[0056] After generating multiple evaluation tasks, the model evaluation platform can randomly extract at least one prompt word from the prompt word library for each task. The prompt word extraction can be based on the languages ​​of the two test models corresponding to the task. For example, if both test models are in Chinese, Chinese prompt words can be extracted; if both are in English, English prompt words can be extracted. Each prompt word is then input into the two test models corresponding to the evaluation task to obtain two output results. One prompt word and its two corresponding output results for one evaluation task constitute one set of evaluation data. This allows for multiple sets of evaluation data to be obtained for multiple tasks. Optionally, inputting one prompt word into one test model yields multiple initial results. One of these initial results is then randomly selected as the output result. The number of initial results can be set according to the actual situation, for example, up to five.

[0057] In the above scheme, the model evaluation platform can automatically extract prompt words for the generated evaluation task and input them into the model to be tested to generate evaluation data, which has a high efficiency in creating evaluation data.

[0058] Step 104: Distribute multiple evaluation data to evaluation users for evaluation, and obtain multiple single evaluation results corresponding to multiple evaluation tasks.

[0059] The evaluation users are specific users who subjectively judge which model's result is better in pairwise comparisons of multiple models under test. The identities of the evaluation users are unrestricted; for example, they can include design users, product users, and R&D users. The number of evaluation users is also unlimited and determined based on the actual situation. A single evaluation result refers to the specific evaluation result of two outputs of two models under test for the same prompt word, judged by an evaluation user, determining the degree of matching with the prompt word and which generated result is better. This can include the model names of the two models under test, the prompt word, the two outputs, the evaluation user, and the specific evaluation result. Each single evaluation result includes whether one model wins and the other loses, or whether both models are tied in the corresponding evaluation task. A tie means the evaluation user chooses one model that they believe matches the prompt word better and has a more satisfactory result; the chosen model wins, and the unchosen model loses. A tie means the evaluation user either chooses neither model or both models.

[0060] Specifically, after generating multiple evaluation data points for multiple evaluation tasks, the model evaluation platform can distribute these data points to evaluation users. The distribution method involves acquiring multiple currently logged-in evaluation users. For each evaluation data point, a user can be randomly selected from among the multiple users or matched according to preset rules. The evaluation data is then sent to that user. After seeing the evaluation data, the user makes a subjective judgment and inputs the specific evaluation result. The model evaluation platform can then obtain the specific evaluation result corresponding to each evaluation data point and combine the model names, prompts, two output results, the evaluation user, and the specific evaluation result of the two models under test to generate a corresponding single evaluation result. Ultimately, multiple single evaluation results are obtained for multiple evaluation data points.

[0061] In some embodiments, multiple evaluation data are distributed to evaluation users for evaluation, resulting in multiple single evaluation results corresponding to multiple evaluation tasks. This includes: displaying the evaluation page of each evaluation data to the evaluation users; and in response to the evaluation trigger operation of the evaluation users on the evaluation page, obtaining the single evaluation result corresponding to each evaluation data.

[0062] The evaluation page can be a page that displays evaluation data and allows users to input specific evaluation results. Evaluation triggering operations include triggering evaluation controls or evaluation data. Evaluation controls can be functional controls used to input specific evaluation results; these controls can be used to select a single model to be tested, select two models to be tested, or input a specific score for a model to be tested. Triggering operations on evaluation data can be operations to select the output results of the model to be tested within the evaluation data, such as selecting an image generated by a Wenshengtu model.

[0063] When distributing multiple evaluation data sets to evaluation users, the model evaluation platform can generate a corresponding evaluation page for each set of data and display it to the evaluation user matching that data. The evaluation page also includes evaluation controls. When the platform receives an evaluation trigger operation from the evaluation user on the evaluation page, targeting the evaluation controls or the output results in the evaluation data, it combines the specific evaluation result corresponding to the trigger operation, along with the model names of the two models under test, prompt words, the two output results, and the evaluation user, to obtain a single evaluation result. Thus, multiple single evaluation results can be obtained for multiple sets of evaluation data.

[0064] For example, Figure 2 is a schematic diagram of an evaluation page provided in an embodiment of this disclosure. As shown in Figure 2, an exemplary evaluation page 200 is displayed. The evaluation page 200 may include a prompt word 201 and two output results of two test models in response to the prompt word 201. Taking the two test models as Chinese text-to-image models as an example, the two output results are the two images in the figure. Furthermore, the evaluation page 200 may include four evaluation controls in area 202: left image is better, right image is better, both are good, and both are poor. When the evaluation user clicks one of the evaluation controls or triggers an operation such as clicking the arrow on the keyboard, they can input their subjective judgment on which of the two images in the figure matches the prompt word 201 better and which result is more satisfactory. In addition, the evaluation page 200 may also include a translation control. This voice control is used to translate the content of the current page from one language to another, so that the evaluation user can choose according to their needs, improving the evaluation adaptability and ease of use.

[0065] Step 105: Based on the initial scores of multiple models to be tested and multiple single evaluation results, determine the current scores of multiple models to be tested using a game-level scoring method.

[0066] The initial score can be the score set by the model under test before participating in the comparative evaluation, that is, the score set by the model at the beginning of the game. The initial scores of different models under test are the same, for example, the initial score is 1000 points. The game level scoring method can be an evaluation method for measuring the level of various game activities. It is a recognized authoritative method. In this embodiment, the game level scoring method can be used to iteratively score the score based on historical scores, winning probabilities, and the results of each game. This method has better authority and confidence. In this embodiment, multiple models under test have initial data. According to the pairwise comparative evaluation results, the model's score will increase or decrease accordingly. The winning probability in each comparison is determined by considering the score differences between different models. If a model under test wins against another model under test that is expected to be stronger in the comparative evaluation, it can obtain more points. The current score can be the latest score of the model under test determined when sorting multiple models under test. The current score is determined once every preset time interval, or when the total number of evaluations reaches a preset number. The preset time can be set according to the actual situation, for example, the preset time can be one day or one week.

[0067] In some embodiments, determining the current scores of multiple models to be tested based on their initial scores and multiple single evaluation results using a game-level scoring method may include: extracting the first evaluation result ranked first in evaluation time from among the multiple single evaluation results as the result to be calculated; calculating the comparative winning probability of the two models to be tested based on their initial scores corresponding to the result to be calculated, and determining two updated scores of the two models to be tested after the comparative evaluation of the result to be calculated based on the first evaluation result and the comparative winning probability of the two models to be tested; determining the updated score of the model to be tested corresponding to the result to be calculated as the new initial score, returning to extract the second evaluation result ranked after the first evaluation result in evaluation time from among the multiple single evaluation results as the new result to be calculated and continuing the calculation until all multiple single evaluation results have been calculated and then stopping the calculation; extracting the updated score of each model to be tested at the time of stopping the calculation as the current score.

[0068] Evaluation time can be the time of the user input result corresponding to a single evaluation result. The result to be calculated can be the single evaluation result currently being processed among multiple single evaluation results. The updated score can be the score of the model under test determined through score iteration for a single evaluation result during the calculation process. The probability of winning in a comparison evaluation can be the probability of winning in a comparison evaluation between two models under test, determined based on existing scores.

[0069] When determining the current scores of multiple models to be tested, the model evaluation device can first sort the multiple single evaluation results in chronological order according to the evaluation time. Based on the sorting result, each single evaluation result is extracted as a result to be calculated. First, the first evaluation result in the order of evaluation time is extracted as the result to be calculated. The initial scores of the two models corresponding to this result are input into the win rate formula to obtain the comparison probability of the two models. Then, based on the results to be calculated, the specific result scores of the two models in this comparison evaluation are determined. For a model, if the result is a win, the specific result score is 1; if the result is a loss, the specific result score is 0; if the result is a tie, the specific result score is 0. For each of the two models, the comparison probability and the specific result score are input into the score adjustment formula to obtain the corresponding updated data. For example, if the initial scores of the two models A and B corresponding to the result are Ra and Rb respectively, the expected value of the comparison win probability of A can be expressed as... The expected value of B's ​​comparative winning probability can be expressed as: The updated score of A after the result to be calculated is R′. A =R A +K(S A -E A ), where S A R' represents the specific score of model A in the result to be calculated, and R' represents the updated score of model B after the result to be calculated. B =R B +K(S B -E B ), S B This represents the specific result score of Model B in the result to be calculated, and K represents a constant coefficient, for example, K = 32.

[0070] After determining the updated scores of the two models to be tested for the results to be calculated, the updated data is set as the new initial score. The process continues to extract the next single evaluation result based on the evaluation time as the new result to be calculated, and continues until all single evaluation results are calculated. The updated data of each model at the time of stopping is then set as the current score.

[0071] Step 106: Determine the model ranking result based on the current scores of multiple models to be tested.

[0072] The model ranking result can be obtained by sorting multiple models under test according to the data calculated above, and can also be called a model ranking list. In this embodiment of the disclosure, the model ranking result is determined once at a preset time interval, or when the total number of evaluations reaches a preset number. The preset time can be set according to the actual situation, for example, the preset time can be one day or one week, and the preset number can also be set according to the actual situation, for example, the preset number can be 150,000.

[0073] Specifically, after obtaining the current scores of multiple models to be tested, the model evaluation platform can sort the models in descending order of their current scores, obtaining the model ranking results, and displaying the ranking results. Furthermore, when displaying the model ranking results, the platform can also show information such as the total number of models to be tested, the total number of evaluations performed by evaluation users, and the latest update time, allowing users to quickly obtain more information. The total number of evaluations refers to the amount of evaluation data actually performed by evaluation users.

[0074] Optionally, the model ranking results include the overall ranking results of multiple models under test and the individual ranking results of multiple models under test for a single evaluation user. The overall ranking result is determined by the single evaluation results of multiple models under test for all evaluation users, while the individual ranking result can be determined by the single evaluation results of multiple models under test for a specific evaluation user. In response to result query operations from evaluation control users, the model evaluation platform can obtain the current scores of multiple models under test for single evaluation results for all evaluation users, and sort them according to the current scores to obtain and display the overall ranking result. Alternatively, in response to result query operations from a single evaluation user, it can calculate and sort the current scores of multiple models under test for the single evaluation results in which that user participated, obtaining and displaying the individual ranking result. This solution can display not only the overall ranking results of all models but also the individual ranking results for a specific evaluation user, making the result display more flexible.

[0075] Optionally, the model ranking results display multi-dimensional information for each model under test. This multi-dimensional information includes at least one of the following: ranking position, model name, current score, probability of winning in comparison, confidence interval of a pre-set reliability level, and number of evaluations. Ranking position refers to the specific position of a model under test among multiple models under test, determined by its current score. In this embodiment, the pre-set reliability level for multiple models under test can be set to the same value, and can be set according to actual circumstances; for example, the pre-set reliability level can be 95%. The number of evaluations can be the number of evaluation data points for a model under test that have been actually evaluated by users. By displaying multi-dimensional information for each model under test in the model ranking results, users can quickly obtain various information about the model under test, which helps improve information acquisition efficiency.

[0076] For example, Figure 3 is a schematic diagram of a model ranking result provided by an embodiment of this disclosure. As shown in Figure 3, a page 300 including model ranking results is displayed. Taking multiple test models as examples of text-based image models, the page 300 displays the overall ranking result. The leaderboard in the figure is the ranking result. The figure displays the total number of models, the total number of evaluations, the latest update time, and displays multi-dimensional information such as ranking position, model name, current score, comparison winning probability, 95% confidence interval, and number of evaluations for each test model. The ranking is the ranking position. The figure only displays the corresponding ranking results for 5 Chinese models among the multiple test models. The multiple test models in the figure are sorted from largest to smallest according to their current scores, and reverse sorting is also supported.

[0077] For example, Figure 4 is a schematic diagram of another model ranking result provided by an embodiment of this disclosure. As shown in Figure 4, page 400 displays a ranking distribution diagram of the confidence intervals corresponding to multiple models to be tested for the model ranking result of Figure 3. In this figure, the horizontal axis represents the specific model to be tested, and the vertical axis represents the score. The confidence intervals of different models to be tested are different. The dots in the figure represent the current score of each model to be tested, and the horizontal line segment perpendicular to the line segment of each dot represents the upper and lower boundary values ​​of the confidence interval determined by the model to be tested based on the current score. For example, if the current score of the model to be tested with a dot is 15, the confidence interval can be 11-19. This is just an example.

[0078] In the above scheme, when displaying the model ranking results, not only can multiple dimensions of information of the model under test be displayed, but the corresponding ranking distribution chart can also be displayed according to the confidence interval, which can improve the richness of the information displayed in the model evaluation comparison results, and can also quickly obtain the confidence interval of the model.

[0079] The model evaluation scheme provided in this embodiment acquires multiple models to be tested and historical evaluation results; generates multiple evaluation tasks based on the multiple models to be tested and historical evaluation results, wherein each evaluation task defines two models to be tested for comparison and evaluation; generates multiple evaluation data corresponding to the multiple evaluation tasks; distributes the multiple evaluation data to evaluation users for evaluation, and obtains multiple single evaluation results corresponding to the multiple evaluation tasks; determines the current score of the multiple models to be tested based on the initial scores of the multiple models to be tested and the multiple single evaluation results using a game-level scoring method; and determines the model ranking result based on the current scores of the multiple models to be tested. By adopting the above technical solution, multiple evaluation tasks can be generated based on multiple models to be tested and historical evaluation results, and corresponding multiple evaluation data can be generated. Multiple single evaluation results from evaluation users can be obtained. The current score is determined based on the initial score of each model to be tested and multiple single evaluation results using a game-level scoring method, and then the model ranking result is determined. Since the evaluation tasks are automatically generated, the efficiency and ease of use of model evaluation are improved. Furthermore, the ranking of different models is determined by a game-level scoring method with high authority and accuracy, which improves the reliability and accuracy of the ranking result.

[0080] The model evaluation scheme of this disclosure embodiment will be further illustrated by a specific example. For example, Figure 5 is a schematic diagram of a model evaluation process provided by an embodiment of this disclosure. As shown in Figure 5, the leaderboard refers to the ranking results of multiple models to be tested, and the competition schedule refers to the evaluation task. It illustrates the functional modules included in the model evaluation platform and the specific process of model evaluation through the platform. The model evaluation platform may include three main modules: user management, model evaluation, and leaderboard calculation, as well as an interaction layer and a data layer. The user management module may include modules such as permission management and user registration / login. The model evaluation module may include modules such as process model deployment, internal model invocation, external model invocation, prompt word management, evaluation data management, evaluation task management, and tag management. The leaderboard calculation module may include modules such as leaderboard... The platform includes modules for single-task management and schedule arrangement. The model evaluation platform also includes an interaction layer, through which evaluation control users and evaluation users can log in. Evaluation users can view evaluation data for a single evaluation task, including visual prompts and the output results of two models under test, and receive a single evaluation result. Evaluation users can conduct multiple rounds of evaluation on multiple evaluation data sets. The platform also includes a data layer, which works with the leaderboard calculation module to calculate leaderboards and schedules. This data layer, in conjunction with the model evaluation module, can generate evaluation data based on the calculated schedule and distribute the data to evaluation users to obtain multiple single evaluation results. The calculated leaderboards can be viewed by evaluation control users and evaluation users in the interaction layer.

[0081] For example, Figure 6 is a schematic diagram of another model evaluation process provided by an embodiment of this disclosure. As shown in Figure 6, the figure illustrates the specific process of calculating the competition schedule, generating evaluation data based on the competition schedule, and obtaining a single evaluation result. In the figure, the competition schedule refers to the evaluation task, the competition schedule generator refers to the evaluation task generator, and the participating group refers to the model pair corresponding to an evaluation task. The process of calculating the competition schedule may include inputting historical evaluation results and multiple models to be tested as input data into the competition schedule generator, and outputting a queue of participating groups. The queue of participating groups includes multiple participating groups, and a model pair of a participating group corresponds to a competition schedule, that is, a model comparison evaluation. For the competition schedule generator, during the cold start phase, two pairs of models to be tested are randomly paired, while during the non-cold start phase, the confidence interval of the model to be tested is... The model pair whose confidence interval shrinks the most after an additional comparative evaluation is selected first. For example, if a new model to be tested is added and has a relatively large confidence interval, it will be selected first to participate in the comparative evaluation. The process of generating evaluation data according to the competition schedule includes: extracting prompt words from the queue of participating groups, obtaining the evaluation data of each participating group through the prompt words, and obtaining the evaluation data queue in the figure. For each evaluation data, a logged-in evaluation user is matched and distributed. After the evaluation user performs the evaluation, the platform can obtain a single evaluation result. A single evaluation result includes the model names of the two models to be tested in a comparative evaluation, the prompt words, the two output results, the evaluation user, and the specific evaluation results. Finally, the leaderboard can be calculated based on multiple single evaluation results.

[0082] The model evaluation scheme of this disclosure solves the usability problems of the model evaluation platform in terms of interaction, task generation logic, and evaluation data preparation by improving the interactive interface and automatically performing evaluation tasks, evaluation data, and evaluation distribution. Furthermore, it solves the reliability and accuracy problems of the ranking results by using an authoritative and accurate game level scoring method for ranking calculation and determination.

[0083] Figure 7 is a schematic diagram of a model evaluation device provided in an embodiment of this disclosure. This device can be implemented by software and / or hardware and is generally integrated into an electronic device. As shown in Figure 7, the device includes:

[0084] The acquisition module 701 is used to acquire multiple models to be tested and historical evaluation results;

[0085] Task module 702 is used to generate multiple evaluation tasks based on the multiple models to be tested and the historical evaluation results, wherein each evaluation task defines two models to be tested for comparative evaluation;

[0086] Data module 703 is used to generate multiple evaluation data corresponding to the multiple evaluation tasks;

[0087] Evaluation module 704 is used to distribute the multiple evaluation data to evaluation users for evaluation, and obtain multiple single evaluation results corresponding to the multiple evaluation tasks;

[0088] The scoring module 705 is used to determine the current score of the multiple test models based on the initial scores of the multiple test models and the multiple single evaluation results using a game-level scoring method;

[0089] The sorting module 706 is used to determine the model sorting result based on the current scores of the multiple models to be tested.

[0090] Optionally, task module 702 is used for:

[0091] Based on the historical evaluation results, determine the reduction amount of multiple confidence intervals for multiple model pairs composed of the two models to be tested;

[0092] Multiple evaluation tasks are generated based on the reduction of multiple confidence intervals of the multiple model pairs.

[0093] Optionally, each of the evaluation tasks corresponds to one of the model pairs;

[0094] The greater the reduction in the confidence interval of a model pair, the earlier the corresponding evaluation task is generated.

[0095] Optionally, data module 703 is used for:

[0096] Extract corresponding prompt words for each of the aforementioned evaluation tasks;

[0097] For each evaluation task, the prompt word for the evaluation task is input into the two corresponding models to be tested to obtain two output results. The prompt word and the two output results are determined as the corresponding evaluation data.

[0098] Optionally, evaluation module 704 is used for:

[0099] The evaluation pages for each of the aforementioned evaluation data will be displayed to the evaluation users;

[0100] In response to the evaluation user's evaluation trigger operation on the evaluation page, the individual evaluation result corresponding to each evaluation data is obtained.

[0101] Optionally, the evaluation triggering operation includes triggering the evaluation control or the evaluation data;

[0102] Each single evaluation result includes one win and one loss or two draws in the two models under test in the corresponding evaluation task, where two draws include both wins or both loses.

[0103] Optionally, the fraction module 705 is used for:

[0104] Extract the first evaluation result, which is ranked first by evaluation time, from the multiple single evaluation results as the result to be calculated;

[0105] The comparison winning probability of the two models is calculated based on the initial scores of the two models to be tested corresponding to the calculation result. Based on the first evaluation result and the comparison winning probability of the two models to be tested, the two updated scores of the two models to be tested after the comparison evaluation of the calculation result are determined.

[0106] The updated score of the model to be tested corresponding to the result to be calculated is determined as the new initial score. The second evaluation result, which is ranked after the first evaluation result in terms of evaluation time, is extracted from the multiple single evaluation results and used as the new result to be calculated. The calculation is stopped when all the multiple single evaluation results have been calculated.

[0107] Extract the updated scores of each of the tested models at the time of stopping calculation as the current scores.

[0108] Optionally, the sorting module 706 is used for:

[0109] The multiple test models are sorted in descending order of their current scores to obtain the model ranking result.

[0110] Optionally, the model ranking result is determined once at a preset time interval, or the model ranking result is determined when the total number of evaluations reaches a preset number.

[0111] Optionally, the model ranking result includes the overall ranking result of the multiple models under test and the individual ranking result of the multiple models under test for a single evaluation user;

[0112] The model ranking results display information in multiple dimensions for each model under test. The information in multiple dimensions includes at least one of the following: ranking position, model name, current score, probability of winning in comparison, confidence interval of pre-set confidence level, and number of evaluations.

[0113] Optionally, the plurality of models to be tested includes text-based image models and / or text-based video models.

[0114] The model evaluation apparatus provided in this disclosure can execute the model evaluation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the method execution.

[0115] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the model evaluation method provided in any embodiment of this disclosure.

[0116] Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Referring specifically to Figure 8, it shows a schematic diagram of the structure suitable for implementing the electronic device 800 in the embodiments of this disclosure. The electronic device 800 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in Figure 8 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this disclosure.

[0117] As shown in Figure 8, the electronic device 800 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0118] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although FIG8 shows an electronic device 800 with various devices, it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0119] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, it performs the functions defined above in the model evaluation method of embodiments of this disclosure.

[0120] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0121] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0122] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0123] The aforementioned computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device causes the following: to acquire multiple models to be tested and historical evaluation results; to generate multiple evaluation tasks based on the multiple models to be tested and the historical evaluation results, wherein each evaluation task defines two models to be tested for comparative evaluation; to generate multiple evaluation data corresponding to the multiple evaluation tasks; to distribute the multiple evaluation data to evaluation users for evaluation, obtaining multiple single evaluation results corresponding to the multiple evaluation tasks; to determine the current score of the multiple models to be tested based on the initial scores of the multiple models to be tested and the multiple single evaluation results using a game-level scoring method; and to determine the model ranking result based on the current scores of the multiple models to be tested.

[0124] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0125] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0126] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0127] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0128] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0129] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0130] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0131] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0132] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A model evaluation method, wherein, The method comprises the following steps: obtaining a plurality of to-be-tested models and historical evaluation results; generating a plurality of evaluation tasks based on the plurality of to-be-tested models and the historical evaluation results, wherein each evaluation task defines a comparison evaluation of two to-be-tested models; generating a plurality of evaluation data corresponding to the plurality of evaluation tasks; distributing the plurality of evaluation data to evaluation users for evaluation to obtain a plurality of single evaluation results corresponding to the plurality of evaluation tasks; determining current scores of the plurality of to-be-tested models by adopting a level of play scoring method based on initial scores of the plurality of to-be-tested models and the plurality of single evaluation results; determining a model ranking result according to the current scores of the plurality of to-be-tested models.

2. The method of claim 1, wherein, The method of generating a plurality of evaluation tasks based on the plurality of to-be-tested models and the historical evaluation results comprises the following steps: determining a plurality of confidence interval reduction amounts of a plurality of model pairs each composed of two to-be-tested models according to the historical evaluation results; generating a plurality of evaluation tasks based on the plurality of confidence interval reduction amounts of the plurality of model pairs.

3. The method of claim 2, wherein, Each evaluation task corresponds to one model pair; The greater the confidence interval reduction amount of one model pair, the earlier the generation order of the corresponding evaluation task.

4. The method of claim 1, wherein, The method of generating a plurality of evaluation data corresponding to the plurality of evaluation tasks comprises the following steps: extracting a prompt word corresponding to each evaluation task; for each evaluation task, inputting the prompt word of the evaluation task into the corresponding two to-be-tested models to obtain two output results, and determining the prompt word and the two output results as the corresponding evaluation data.

5. The method of claim 1, wherein, The method of distributing the plurality of evaluation data to evaluation users for evaluation to obtain a plurality of single evaluation results corresponding to the plurality of evaluation tasks comprises the following steps: showing an evaluation page of each evaluation data to an evaluation user; in response to an evaluation trigger operation of the evaluation user in the evaluation page, obtaining a single evaluation result corresponding to each evaluation data.

6. The method of claim 5, wherein, The evaluation trigger operation comprises a trigger operation on an evaluation control or on the evaluation data; Each single evaluation result comprises one win and one loss or two draws of two to-be-tested models in the corresponding evaluation task, and the two draws comprise two wins or two losses.

7. The method of claim 1, wherein, The method of determining current scores of the plurality of to-be-tested models by adopting a level of play scoring method based on initial scores of the plurality of to-be-tested models and the plurality of single evaluation results comprises the following steps: extracting a first evaluation result with the first evaluation time in the plurality of single evaluation results as a to-be-calculated result; calculating comparison win probabilities of two to-be-tested models corresponding to the to-be-calculated result according to the initial scores of the two to-be-tested models, and determining two updated scores of the two to-be-tested models after comparison evaluation of the to-be-calculated result according to the first evaluation result and the comparison win probabilities of the two to-be-tested models; determining the updated score of the to-be-tested model corresponding to the to-be-calculated result as a new initial score, and continuing to calculate by extracting a second evaluation result after the first evaluation result in the plurality of single evaluation results as a new to-be-calculated result until all the plurality of single evaluation results are calculated to stop calculation; extracting the updated score of each to-be-tested model at the time of stopping calculation as a current score.

8. The method of claim 1, wherein, The model ranking result is determined according to the current scores of the plurality of to-be-tested models, and includes: The plurality of to-be-tested models are ranked in descending order of the current scores of the plurality of to-be-tested models, to obtain the model ranking result.

9. The method of claim 1, wherein, The model ranking result is determined every interval of a preset time, or the model ranking result is determined when the total number of evaluation reaches a preset number.

10. The method of claim 1, wherein, The model ranking result includes a total ranking result of the plurality of to-be-tested models and a personal ranking result of the plurality of to-be-tested models for an evaluation user. The model ranking result displays a plurality of dimensions of information for each to-be-tested model, and the plurality of dimensions of information include at least one of a ranking position, a model name, a current score, a comparison winning probability, a confidence interval of a preset confidence level, and an evaluation number.

11. The method of claim 1, wherein, The plurality of to-be-tested models include text-to-image models and / or text-to-video models.

12. A model evaluation device in which, The method includes: An acquisition module is configured to acquire a plurality of to-be-tested models and historical evaluation results; A task module is configured to generate a plurality of evaluation tasks based on the plurality of to-be-tested models and the historical evaluation results, wherein each evaluation task defines a comparison evaluation between two to-be-tested models; A data module is configured to generate a plurality of evaluation data corresponding to the plurality of evaluation tasks; An evaluation module is configured to distribute the plurality of evaluation data to evaluation users for evaluation, to obtain a plurality of single evaluation results corresponding to the plurality of evaluation tasks; A score module is configured to determine current scores of the plurality of to-be-tested models based on initial scores of the plurality of to-be-tested models and the plurality of single evaluation results using a game level scoring method; A ranking module is configured to determine a model ranking result according to the current scores of the plurality of to-be-tested models.

13. An electronic device, comprising: The electronic device includes: A processor; A memory for storing executable instructions of the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the model evaluation method of any one of claims 1-11.

14. A computer readable storage medium, wherein, The storage medium stores a computer program for executing the model evaluation method of any one of claims 1-11. The storage medium stores a computer program for executing the model evaluation method of any one of claims 1-11.

Images