A model evaluation method, device, apparatus and medium

Abstract

The method comprises: obtaining test point configuration information and evaluation configuration information according to a model evaluation scheme provided by the embodiment of the present disclosure; selecting a current test point based on the test point configuration information, and generating a current test question corresponding to the current test point by using a test question generation model; creating an evaluation task for a large model based on the current test question and the evaluation configuration information, and executing the evaluation task to determine an evaluation result. The embodiment of the present disclosure realizes automatic generation of test questions by using the test question generation model, and realizes automatic creation and execution of the evaluation task based on the automatically generated test questions, thereby realizing automatic evaluation of the visual large model, greatly improving the efficiency of model evaluation, reducing the evaluation cost, avoiding the defects of lack of consistency and repeatability caused by manual evaluation, and improving the richness and diversity of the evaluation range.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model evaluation method, apparatus, device and medium. Background Technology

[0002] The evaluation of visual large models such as Wensheng image large model and Wensheng video large model mainly adopts manual evaluation. However, the evaluation process is easily affected by subjectivity, resulting in a lack of consistency and repeatability of results. In addition, manual evaluation is time-consuming and inefficient, and needs to be improved. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides a model evaluation method, apparatus, equipment, and medium.

[0004] This disclosure provides a model evaluation method, the method comprising:

[0005] Obtain test site configuration information and assessment configuration information;

[0006] Based on the test site configuration information, the current test site is selected, and the current test question corresponding to the current test site is generated using the test question generation model;

[0007] An evaluation task for a large model is created based on the current test questions and the evaluation configuration information, and the evaluation task is executed to determine the evaluation results.

[0008] This disclosure also provides a model evaluation apparatus, the apparatus comprising:

[0009] The acquisition module is used to acquire test site configuration information and assessment configuration information;

[0010] The exam question module is used to select the current exam point based on the exam point configuration information and generate the current exam question corresponding to the current exam point using the exam question generation model;

[0011] The evaluation module is used to create an evaluation task for the large model based on the current test question and the evaluation configuration information, and to execute the evaluation task to determine the evaluation result.

[0012] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the executable instructions to implement the model evaluation method provided in this disclosure.

[0013] This disclosure also provides a computer-readable storage medium storing a computer program for executing the model evaluation method provided in this disclosure.

[0014] Compared with the prior art, the technical solution provided in this disclosure has the following advantages: The model evaluation scheme provided in this disclosure obtains test point configuration information and evaluation configuration information; selects the current test point based on the test point configuration information, and generates the current test question corresponding to the current test point using a test question generation model; creates an evaluation task for a large model based on the current test question and evaluation configuration information, and executes the evaluation task to determine the evaluation result. By adopting the above technical solution, test points are selected based on the obtained test point configuration information and test questions are generated using a test question generation model. Then, an evaluation task for a large model can be created based on the generated test questions and evaluation configuration information, and the evaluation task can be executed based on the evaluation model to obtain the evaluation result. The test question generation model enables automatic generation of test questions, and the automatic creation and execution of evaluation tasks are achieved based on the automatically generated test questions. This realizes automated evaluation of large visual models, greatly improving the efficiency of model evaluation, reducing evaluation costs, avoiding the lack of consistency and repetitiveness caused by manual evaluation, and improving the richness and diversity of the evaluation scope. Attached Figure Description

[0015] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0016] Figure 1 A schematic flowchart of a model evaluation method provided in an embodiment of this disclosure;

[0017] Figure 2 A flowchart illustrating another model evaluation method provided in this embodiment of the disclosure;

[0018] Figure 3 A flowchart illustrating another model evaluation method provided in this embodiment of the disclosure;

[0019] Figure 4 A schematic diagram illustrating a model evaluation process provided in an embodiment of this disclosure;

[0020] Figure 5 A schematic diagram illustrating another model evaluation process provided in an embodiment of this disclosure;

[0021] Figure 6 This is a schematic diagram of the structure of a model evaluation device provided in an embodiment of the present disclosure;

[0022] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0025] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0026] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0027] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0028] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0029] Currently, the evaluation of large visual models is mainly carried out manually, but this has the following problems: subjectivity, which is easily affected by the subjective judgment of the evaluator, leading to inconsistent results; lack of consistency and repeatability, as different evaluators or the same evaluator may give different scores to the same output, resulting in a lack of consistency and repeatability; time and cost, as manual evaluation is usually time-consuming, especially when dealing with large amounts of data, and is costly; fatigue effect, as long evaluation time may lead to evaluator fatigue, affecting the evaluation quality.

[0030] To address the aforementioned issues, this disclosure provides a model evaluation method, which will be described below with reference to specific embodiments.

[0031] Figure 1 This is a flowchart illustrating a model evaluation method provided in an embodiment of this disclosure. The method can be executed by a model evaluation device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method includes:

[0032] Step 101: Obtain test site configuration information and assessment configuration information.

[0033] The model evaluation method of this disclosure can be executed by a model evaluation platform to evaluate large models. The large model in this disclosure can be a visual large model, which is a large-scale neural network model designed and trained based on deep learning technology for visual tasks. Visual tasks may include, for example, image classification, image generation, object detection, and semantic segmentation. The visual large model can learn complex features of visual data through massive amounts of data and powerful computing capabilities, enabling it to handle diverse visual tasks. In this disclosure, the visual large model may include at least one of a text-to-image model, a text-to-video model, and an image-to-video model. A text-to-image model can be a model that generates images from text, a text-to-video model can be a model that generates videos from text, and an image-to-video model can be a model that generates videos from images.

[0034] The test point configuration information can be configuration information for the selection of test points involved in the test questions, used to control the selection of test points. Test points can be key features and attributes in images or videos that the visual large model needs to focus on perceiving, recognizing, or understanding during visual task processing. They can cover multiple levels from basic visual elements to high-level semantic information. For example, test points can include object categories such as spiders and volcanoes, color categories such as yellow and black, and spatial categories such as interiors. In this embodiment of the disclosure, the test point configuration information can include the number of categories and the number of test points. The number of categories refers to the specific number of test point categories that need to be selected for this evaluation from a plurality of pre-set test point categories. The test point category can be the category into which the test points are divided; the specific number of test points can be controlled when selecting test points.

[0035] Evaluation configuration information can include information such as the testing environment, parameter settings, dataset selection, and evaluation metrics for the current evaluation of large visual models. Evaluation configuration information can reflect the current evaluation requirements.

[0036] Specifically, when a user needs to evaluate a large visual model, they can input test point configuration information and evaluation configuration information. The model evaluation device responds to the user's input operation and obtains the test point configuration information and evaluation configuration information.

[0037] Step 102: Select the current test center based on the test center configuration information, and use the test question generation model to generate the current test question corresponding to the current test center.

[0038] The current test point can be any test point selected for the current assessment. The selection of current test points must include object test points, and there can be multiple current test points. The question generation model can be a model that automatically generates test questions based on the selected test points. Test questions can be specific questions designed around the test points, used to test the visual model's ability to perceive, recognize, and understand the test points. Test questions can also be input prompts for the visual model; after inputting the test questions, the visual model can generate visual media such as images or videos. The current test question can be a question generated from the current test point. For example, assuming the current test points include spider, volcano, yellow, black, and interior, the current test question could be "A yellow spider is inside a black volcano."

[0039] Specifically, after obtaining the test point configuration information, the model evaluation device can first randomly select the current category of the current category, and then randomly select test points in each current category, until the number of test points is reached, thus obtaining multiple current test points. Then, multiple test point statements of multiple current test points can be input into the test question generation model to output the current test question.

[0040] For example, Figure 2 A flowchart illustrating another model evaluation method provided in this disclosure embodiment is shown below. Figure 2 As shown, in one feasible implementation, step 102 may include the following steps:

[0041] Step 201: Select multiple current categories, including object categories, from multiple test point categories based on the number of categories.

[0042] The number of categories refers to the specific number of test point categories to be selected for this assessment from a pre-set pool of test point categories. Test point categories can be the classifications of test points. In this embodiment, test point categories may include objects, colors, numbers, textures, shapes, sizes, styles, and spaces, etc. The current category can be the category selected for the current assessment. Multiple current categories must include object categories. Test points for object categories may include, for example, cars, chairs, etc.

[0043] Specifically, when the model evaluation device selects the current test point based on the test point configuration information, it can first select multiple current categories from multiple test point categories according to the number of categories. Assuming the number of categories is k+1, where 1 represents the object category and k represents the random selection of k current categories from multiple test point categories, the selection can be, for example, 1 / 4 probability of being the object category and 3 / 4 probability of being selected from other categories besides the object category. When k=4, the final selected 5 current categories can include two object categories, two color categories and one space category, which is just an example.

[0044] Step 202: Select the current test point from multiple current categories based on the number of test points.

[0045] The number of test centers can be controlled when selecting test centers. The number of test centers can include the total number of test centers selected and / or the number of test centers selected for each test center category in this selection. For example, the number of test centers can be set to select one test center for each test center category.

[0046] After determining multiple current categories, the model evaluation device can randomly select test points in each current category. The specific number is determined according to the number of test points. For example, if the total number of test points selected is 10 and the number of test points selected for the object category is 4, then 6 current test points are randomly selected from other current categories, resulting in 10 current test points.

[0047] In the above scheme, by setting the number of categories and the number of test centers in the test center configuration information, it is possible to effectively control whether the automatic test center selection meets the evaluation requirements. The selected categories must include object categories to ensure that the subsequent test questions are generated with the appropriate objects and to avoid errors.

[0048] The above selection of the current test center based on test center configuration information includes steps 201 and 202.

[0049] Step 203: Bind the object test points in the current test point to the associated test points to obtain the test point binding statement.

[0050] Among them, object test points can be test points under object categories. Associated test points are test points that need to be bound to object test points to demonstrate their attributes or provide an accurate description. Associated test points can be test points under other test point categories besides object test points; for example, associated test points can include test points under categories such as color, number, texture, shape, size, style, and space. One associated test point can be bound to one or two object test points. Test point binding statements are specific statements used to bind object test points in the current test point to associated test points. They can include multiple relational statements between multiple associated test points and object test points. Since the relationships between different associated test points and object test points are different, different associated test points can use different relational statements to bind to object test points. Test point binding statements can be code statements.

[0051] Specifically, when the model evaluation device generates the current test question corresponding to the current test point using the test question generation model, it can first randomly select one or two object test points to bind to each associated test point in the current test point according to the binding requirements of the associated test point. When the associated test point needs to be bound to two object test points and the current test point only includes one object test point, it can continue to randomly select an object test point for binding, thus obtaining the test point binding statement.

[0052] For example, when the associated test point is a test point under the spatial category, the associated test point is bound to two random object test points. The binding statement can be that object A is to the left of object B in space. When the associated test point is a test point under the size category, the associated test point can be bound to one random object test point.

[0053] Step 204: Input the test point category statement and test point binding statement corresponding to the current test point into the test question generation model, output visual prompt words, and use the visual prompt words as the current test question.

[0054] Test point category statements can include multiple category description statements for the current test point and its corresponding categories. Category description statements can describe the characteristics of a test point category; for example, for the size category, the category description statement could be the dimensions of the object test point. Category description statements can also be code statements. Visual prompts can be prompts used in the input visual model to guide the execution of visual tasks. Prompts can be text or sentence fragments used to trigger and guide the model to perform specific tasks and generate specific output content.

[0055] After determining the test point binding statement, the model evaluation device can input the test point category statement corresponding to the current test point and the test point binding statement into the test question generation model. The test question generation model will analyze and process the data to generate visual prompts. These visual prompts can describe images or videos that match the test point category statement and the test point binding statement. The visual prompts will then be used as the current test question.

[0056] Optionally, after outputting the visual cue, it can be validated. Specifically, a validation model can be used to determine if the visual cue is unrealistic or illogical. If so, the validation result is "failed," and the process returns to step 201 to regenerate; if not, the validation result is "passed," and the visual cue is used as the current test question. By adding validation to the visual cue, the accuracy and effectiveness of the test questions can be ensured.

[0057] Optionally, the test question generation model can also generate multiple test point questions corresponding to the current test point, and input these multiple test point questions during subsequent evaluations. This allows the test point questions to be considered during the subsequent evaluation process, which helps improve the accuracy of the visual large model evaluation.

[0058] For example, assuming the current test point includes spider, volcano, yellow, black, and interior, the visual cue word could be "a yellow spider inside a black volcano". The current test question could include only this visual cue word, or it could include this visual cue word and the test point question for each current test point.

[0059] The above-mentioned method of generating the current exam question corresponding to the current exam point using the exam question generation model may include steps 203 and 204.

[0060] In the above scheme, after selecting test points, the relationship between test points is established by binding test points. Then, test questions are automatically generated using a test question generation model, which effectively improves the efficiency of test question generation and solves the limitation of manually generated test questions. This allows for a more comprehensive evaluation of the capabilities of the large visual model and obtains more accurate evaluation results.

[0061] For large-scale model evaluation, test questions can be automatically generated based on the structured combination of test points. The model can generate test questions for test points in a batch in an efficient and agile manner, and the test questions have structured explanations of test points, which effectively improves the efficiency and diversity of test question generation.

[0062] Step 103: Create an evaluation task for the large model based on the current test questions and evaluation configuration information, and execute the evaluation task to determine the evaluation results.

[0063] The evaluation configuration information can be information such as the testing environment, parameter settings, dataset selection, and evaluation metrics for the current evaluation of the large visual model. This information reflects the current evaluation requirements. In this embodiment, the evaluation configuration information can include the current evaluation task type and evaluation model. The task type can be a specific classification of the evaluation task; in this embodiment, the task type can include scoring, comparison, etc. The large model in this embodiment can be a large visual model, which can be a large-scale neural network model designed and trained for visual tasks based on deep learning technology. The evaluation model can be a model used to evaluate the output results of the large visual model using one or more dimensions of metrics. The evaluation model can be a pre-trained large model, specifically determined according to the actual situation. In this embodiment, the evaluation model can include at least one of a multimodal evaluation model targeting multiple evaluation metrics and a single evaluation model targeting only one evaluation metric. The number of evaluation models can be one or more. Evaluation metrics can be standards used to quantify the performance of the large model in a specific task or scenario. In this embodiment, the evaluation metrics quantify the performance of the large visual model in visual tasks; for example, evaluation metrics can include image or video clarity, motion quality, probability of meeting requirements, etc. A multimodal evaluation model can be a comprehensive standard evaluation model that can perform evaluations under multiple evaluation metrics, while a single evaluation model can be a model that focuses on a single evaluation metric and performs evaluations for a specific vertical scenario or a specific dimension.

[0064] The evaluation task can be a series of evaluation processes set up for a large visual model, used to systematically verify and quantify the capabilities of the large visual model in a specific task or scenario. In this embodiment of the disclosure, the evaluation task may include steps such as determining the dataset, generating prompt words, and determining the evaluation model. The evaluation result may be the result of evaluating the data output by the large visual model using the evaluation model under one or more evaluation metrics, or it may be the result of automated evaluation.

[0065] The large visual model disclosed in this embodiment can be a large visual model in the training process or a large visual model that has already been trained. It can not only evaluate the performance of the model at each stage of the training process and guide the model training, but also quickly evaluate the capability diversity and boundary of the trained model. The application scenarios of automated evaluation are wide.

[0066] Specifically, after generating the current test question, the model evaluation device can create an evaluation task using the current test question and the task type and evaluation model in the evaluation configuration information. Since there can be multiple evaluation tasks, the evaluation tasks can be asynchronously scheduled and executed to obtain the evaluation results.

[0067] For example, Figure 3A flowchart illustrating another model evaluation method provided in this disclosure embodiment is shown below. Figure 3 As shown, in one feasible implementation, step 103 may include the following steps:

[0068] Step 301: Generate a large model evaluation dataset based on the current exam questions.

[0069] The evaluation dataset can be a collection of visual media output by the current large visual model being evaluated, and the evaluation dataset is obtained based on the same current test question.

[0070] Specifically, when creating an evaluation task, the model evaluation device can first input the current test question into a large model to generate visual media. There are multiple large models and multiple visual media, with a one-to-one correspondence between the large models and visual media. These multiple visual media serve as the evaluation dataset. Visual media can be media that conveys information or expresses content visually; in this embodiment, visual media may include images, videos, etc. By inputting the automatically generated test questions into the large visual model to generate the subsequent evaluation dataset, the richness and diversity of the evaluation dataset are ensured due to the wider range of test questions.

[0071] Step 302: Generate evaluation prompts based on the prompt template of the task type.

[0072] The task type can be a specific category of the evaluation task. The prompt word template can be a template that configures the content and position of the prompt words, used to generate prompt words. In this embodiment, the prompt word template is used to generate evaluation prompt words, and the prompt word template includes multiple placeholders for filling different information. The evaluation prompt words can be used to guide the evaluation model to perform specific evaluations of a certain task type on the evaluation dataset. In this embodiment, multiple prompt word templates are pre-set, and different task types can correspond to different prompt word templates.

[0073] The model evaluation device obtains the corresponding prompt word template based on the task type in the evaluation configuration information, and fills in the corresponding information according to the prompt word template. The filled information may include the aforementioned evaluation dataset, the aforementioned multiple test point questions, etc., and is specifically set according to the actual situation, thereby generating evaluation prompt words. For example, when the task type is scoring, the evaluation prompt words may include system prompt words and user prompt words. The system prompt words may be "The user will input an image and a raw image requirement. Please judge whether this image faithfully follows the requirements. Please pay attention to the details emphasized in the requirements, please pay attention to whether the subject, the relationship between subjects, and the requirements for the picture style are presented, and please pay attention to whether the subject and details in the picture are depicted naturally and without abnormalities." The user prompt words may be "Please give an overall judgment, directly answer yes or no. Directly output yes / no." The calculation model is used to convert the answer into a specific score.

[0074] Step 303: Define the evaluation steps determined based on the evaluation dataset, evaluation prompts, and evaluation model as the evaluation task.

[0075] Evaluation steps can be specific components of an evaluation task, and an evaluation task can include multiple evaluation steps. In this embodiment of the disclosure, the evaluation task is obtained by combining the evaluation steps of obtaining the above-mentioned evaluation dataset, obtaining the above-mentioned evaluation prompt words, and evaluating using an evaluation model.

[0076] The above-mentioned creation of an evaluation task for a large model based on the current test questions and evaluation configuration information may include steps 301-303, realizing the automatic creation of evaluation tasks, which effectively improves efficiency and reduces costs compared to manual creation.

[0077] Step 304: Input the evaluation dataset and evaluation prompts into the evaluation model, and output the evaluation results of the large model.

[0078] Specifically, when the model evaluation device performs the evaluation task and determines the evaluation result, it can use the evaluation model to evaluate the input evaluation dataset according to the evaluation prompt words and output the evaluation result. The evaluation result includes the result of each evaluation model on each large model in each evaluation index. The evaluation result can be displayed in tabular form.

[0079] The above solution achieves effective structured decomposition of the evaluation process by automating the creation and scheduling of evaluation tasks, solving the unstructured bottleneck of large visual models, and improving the agility and efficiency of evaluation by defining structured results and exploring feasible paths and best practices for automating the evaluation process.

[0080] The model evaluation scheme provided in this disclosure acquires test point configuration information and evaluation configuration information; selects the current test point based on the test point configuration information, and generates the current test question corresponding to the current test point using a test question generation model; creates an evaluation task for a large model based on the current test question and evaluation configuration information, and executes the evaluation task to determine the evaluation result. By adopting the above technical solution, test points are selected based on the acquired test point configuration information, and test questions are generated using a test question generation model. Then, an evaluation task for a large model can be created based on the generated test questions and evaluation configuration information, and the evaluation task can be executed based on the evaluation model to obtain the evaluation result. The test question generation model enables automatic generation of test questions, and the automatic creation and execution of evaluation tasks are based on the automatically generated test questions, thereby achieving automated evaluation of large visual models. This greatly improves the efficiency of model evaluation, reduces evaluation costs, avoids the lack of consistency and repetitiveness caused by manual evaluation, and enhances the richness and diversity of the evaluation scope.

[0081] In some embodiments, the model evaluation method may further include: performing result analysis based on the evaluation results, determining the evaluation analysis results, and displaying an evaluation report including the evaluation results and the evaluation analysis results.

[0082] The evaluation analysis results can be obtained through further statistical or effective analysis of the evaluation results obtained from the evaluation model. Statistical analysis may include, for example, significant difference analysis. The evaluation report can be a document that evaluates, analyzes, and summarizes the performance and effectiveness of the large visual model, and may include both the evaluation results and the evaluation analysis results. Specifically, after obtaining the evaluation results of the large model, the model evaluation device can perform result analysis to obtain the evaluation analysis results. Then, the evaluation results and evaluation analysis results can be organized in tabular form to obtain an evaluation report, which is then presented to the user. In the above solution, after the automated evaluation of the large visual model, it supports the rapid and automated generation of evaluation reports, improving the efficiency of evaluation report generation, and displays further evaluation analysis results, allowing users to obtain in-depth evaluation information.

[0083] The model evaluation scheme of this disclosure embodiment will be further illustrated by a specific example below. Figure 4 This is a schematic diagram of a model evaluation process provided in an embodiment of the present disclosure, such as... Figure 4 As shown, the model evaluation process can include: automatically generating test questions, inputting the test questions into the large visual model to obtain the evaluation dataset, automatically evaluating the evaluation dataset to obtain the evaluation results, automatically generating test questions in batches based on scalable test points, which can improve the diversity and flexibility of test questions, explore the boundaries of the model, and inputting the structured context of the test questions during automatic evaluation. The structured context of the test questions can be the test point questions generated for the test points when generating the test questions, realizing the completion reading of the test questions based on the evaluation results of the test questions and test point questions, effectively improving the accuracy of the evaluation of the large visual model.

[0084] For example, Figure 5 A schematic diagram of another model evaluation process provided in this disclosure embodiment, as shown below. Figure 5As shown, the model evaluation system of this disclosure can be used for both model-accompanying evaluation and model evaluation. Model-accompanying evaluation refers to evaluating the large visual model during the training process, while model evaluation refers to evaluating the large visual model after training. The model evaluation system may include a test question module and an evaluation module. The test question module may include a test point management model and a test question generation module. The evaluation module includes an automated evaluation unified interface, a multimodal evaluation model, and a single evaluation model. The test question module is used to automatically select test points and generate test questions, and perform test question management, test point management, and test question availability judgment, etc. The test point management module is used to define test points, determine the scope of test points, and select test points, etc. The test question generation module is used to declare or define test questions, set up a test question generation model, and manage test question generation tasks, etc. The evaluation module is used to create and schedule evaluation tasks, analyze and manage evaluation results, and generate and display evaluation reports, etc. The automated evaluation unified interface is used to define the standard input and output specifications for evaluation execution.

[0085] This model evaluation solution addresses the issues of inconsistency, repetitiveness, and excessive time consumption caused by the subjectivity of manual evaluation, as well as the impact of fatigue on evaluation quality. The automated evaluation focuses on improving evaluation efficiency and the richness and diversity of the evaluation scope at a lower cost. In terms of evaluation efficiency, it complements manual evaluation, allowing users to conduct preliminary or high-frequency informal evaluations at a low cost. Furthermore, in terms of evaluation scope, automated question design overcomes the limitations of manually constructed questions, enabling a more comprehensive assessment of model capabilities and yielding more accurate evaluation results.

[0086] Figure 6 This is a schematic diagram of a model evaluation device provided in an embodiment of this disclosure. The device can be implemented by software and / or hardware, and is generally integrated into an electronic device. Figure 6 As shown, the device includes:

[0087] Module 601 is used to obtain test site configuration information and assessment configuration information;

[0088] The test question module 602 is used to select the current test point based on the test point configuration information and generate the current test question corresponding to the current test point using the test question generation model;

[0089] Evaluation module 603 is used to create an evaluation task for a large model based on the current test question and the evaluation configuration information, and to execute the evaluation task to determine the evaluation result.

[0090] Optionally, the test site configuration information includes the number of categories and the number of test sites. The test question module 602 includes test site units, used for:

[0091] Based on the number of categories, select multiple current categories, including object categories, from multiple test point categories;

[0092] The current test point is selected from the multiple current categories based on the number of test points.

[0093] Optionally, exam module 602 includes exam question units for:

[0094] Bind the object test points in the current test point to the associated test points to obtain the test point binding statement;

[0095] The test point category statement corresponding to the current test point and the test point binding statement are input into the test question generation model, and visual prompt words are output and the visual prompt words are used as the current test question.

[0096] Optionally, the evaluation configuration information includes the current evaluation task type and evaluation model, and the evaluation module 603 includes task units for:

[0097] The evaluation dataset for the large model is generated based on the current exam questions;

[0098] Generate evaluation prompts based on the prompt template for the task type;

[0099] The evaluation steps determined based on the evaluation dataset, the evaluation prompts, and the evaluation model are defined as the evaluation task.

[0100] Optionally, task units are used for:

[0101] The current test question is input into the large model to generate multiple visual media.

[0102] The aforementioned visual media were used as the evaluation dataset.

[0103] Optionally, the evaluation module 603 includes an execution unit for:

[0104] The evaluation dataset and the evaluation prompts are input into the evaluation model, and the evaluation results of the large model are output.

[0105] Optionally, the evaluation model includes at least one of a multimodal evaluation model for multiple evaluation metrics and a single evaluation model for only one evaluation metric.

[0106] Optionally, the device further includes a result analysis module for:

[0107] Based on the evaluation results, the results are analyzed to determine the evaluation analysis results, and an evaluation report including the evaluation results and the evaluation analysis results is displayed.

[0108] The model evaluation apparatus provided in this disclosure can execute the model evaluation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the method execution.

[0109] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the model evaluation method provided in any embodiment of this disclosure.

[0110] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure.

[0111] The following is a detailed reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing the electronic device 700 in the embodiments of this disclosure. The electronic device 700 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0112] like Figure 7 As shown, the electronic device 700 may include a processing unit 701 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 702 or a program loaded from storage device 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0113] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0114] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the model evaluation method of embodiments of this disclosure.

[0115] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an electrically erasable programmable read-only memory (EPROM), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, radio frequency (RF), etc., or any suitable combination thereof.

[0116] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as Hypertext Transfer Protocol (HTTP), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include Local Area Networks (LANs), Wide Area Networks (WANs), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0117] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0118] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire test site configuration information and evaluation configuration information; select the current test site based on the test site configuration information and generate the current test question corresponding to the current test site using a test question generation model; create an evaluation task for a large model based on the current test question and the evaluation configuration information, and execute the evaluation task to determine the evaluation result.

[0119] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0121] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0122] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field-Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0123] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, flash memory, optical fiber, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0125] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0126] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0127] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A model evaluation method, characterized in that, include: Obtain test site configuration information and assessment configuration information; Based on the test site configuration information, the current test site is selected, and the current test question corresponding to the current test site is generated using the test question generation model; An evaluation task for a large model is created based on the current test questions and the evaluation configuration information, and the evaluation task is executed to determine the evaluation results.

2. The method according to claim 1, characterized in that, The test site configuration information includes the number of categories and the number of test sites. Selecting the current test site based on this configuration information includes: Based on the number of categories, select multiple current categories, including object categories, from multiple test point categories; The current test point is selected from the multiple current categories based on the number of test points.

3. The method according to claim 1, characterized in that, The step of generating the current exam question corresponding to the current exam point using the exam question generation model includes: Bind the object test points in the current test point to the associated test points to obtain the test point binding statement; The test point category statement corresponding to the current test point and the test point binding statement are input into the test question generation model, and visual prompt words are output and the visual prompt words are used as the current test question.

4. The method according to claim 1, characterized in that, The evaluation configuration information includes the current evaluation task type and evaluation model. Based on the current test question and the evaluation configuration information, an evaluation task is created for the large model, including: The evaluation dataset for the large model is generated based on the current exam questions; Generate evaluation prompts based on the prompt template for the task type; The evaluation steps determined based on the evaluation dataset, the evaluation prompts, and the evaluation model are defined as the evaluation task.

5. The method according to claim 4, characterized in that, The evaluation dataset for the large model generated based on the current exam questions includes: The current test question is input into the large model to generate visual media, and the visual media is used as the evaluation dataset.

6. The method according to claim 4, characterized in that, The process of performing the evaluation task and determining the evaluation result includes: The evaluation dataset and the evaluation prompts are input into the evaluation model, and the evaluation results of the large model are output.

7. The method according to claim 1 or 4, characterized in that, The evaluation model includes at least one of a multimodal evaluation model for multiple evaluation metrics and a single evaluation model for only one evaluation metric.

8. The method according to claim 1, characterized in that, The method further includes: Based on the evaluation results, the results are analyzed to determine the evaluation analysis results, and an evaluation report including the evaluation results and the evaluation analysis results is displayed.

9. A model evaluation device, characterized in that, include: The acquisition module is used to acquire test site configuration information and assessment configuration information; The exam question module is used to select the current exam point based on the exam point configuration information and generate the current exam question corresponding to the current exam point using the exam question generation model; The evaluation module is used to create an evaluation task for the large model based on the current test question and the evaluation configuration information, and to execute the evaluation task to determine the evaluation result.

10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the model evaluation method according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the model evaluation method according to any one of claims 1-8.

Images