Model evaluation method and system, computer program product and electronic equipment

Abstract

The invention relates to the technical field of computers, and relates to a model evaluation method and system, a computer program product and electronic equipment. The model evaluation method comprises the steps that according to the attribute type of a to-be-evaluated model, multiple evaluation tasks are distributed, and different evaluation tasks have respective corresponding evaluation data sets; according to the evaluation data set of the current evaluation task, executing the current evaluation task on the to-be-evaluated model to obtain an initial evaluation result, the initial evaluation result comprising a plurality of evaluation dimensions; adjusting the evaluation content of the to-be-evaluated task based on the initial evaluation result, so as to execute evaluation operation on the to-be-evaluated model according to the adjusted to-be-evaluated task; and generating a comprehensive evaluation result of the to-be-evaluated model according to the initial evaluation results of the plurality of evaluation tasks. According to the invention, the efficiency and coverage range of large model evaluation can be improved, and the accuracy of model evaluation is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a model evaluation method, a model evaluation system, a computer program product, and an electronic device. Background Art

[0002] With the development of stochastic computers and artificial intelligence technology, general-purpose large models and industry-specific large models have emerged one after another and are widely used in natural language processing, computer vision and other fields. Large models have shown excellent performance in many tasks, but due to the complex structure and diverse tasks of large models, their evaluation methods are still immature.

[0003] The current large model evaluation method evaluates the performance of large models through traditional indicators such as accuracy and F1 score (F1 Score, an indicator used to measure model accuracy). However, this evaluation method cannot meet the multi-task adaptability of large models in multiple scenarios, and the evaluation efficiency is low and the coverage is not comprehensive, which affects the accuracy of the evaluation results and, to a certain extent, affects the development, selection and application of large models.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a model evaluation method and system, a computer program product and an electronic device, thereby improving the efficiency and coverage of large model evaluation and improving the accuracy of model evaluation at least to a certain extent.

[0006] According to one aspect of the present disclosure, a model evaluation method is provided, including: assigning a plurality of evaluation tasks according to the attribute type of a model to be evaluated, wherein different evaluation tasks have their own corresponding evaluation data sets; executing the current evaluation task on the model to be evaluated according to the evaluation data set of the current evaluation task to obtain an initial evaluation result, wherein the initial evaluation result includes multiple evaluation dimensions; adjusting the evaluation content of the task to be evaluated based on the initial evaluation result to perform an evaluation operation on the model to be evaluated according to the adjusted task to be evaluated; and generating a comprehensive evaluation result of the model to be evaluated according to the initial evaluation results of the plurality of evaluation tasks.

[0007] In an exemplary embodiment of the present disclosure, multiple evaluation tasks are assigned according to the attribute type of the model to be evaluated, including: obtaining multiple evaluation tasks matching the attribute type and assigning them to the model to be evaluated; and obtaining evaluation data sets corresponding to the multiple evaluation tasks and assigning them to the model to be evaluated according to the correspondence between the evaluation tasks and the evaluation data sets.

[0008] In an exemplary embodiment of the present disclosure, according to an evaluation data set of the current evaluation task, the current evaluation task is performed on the model to be evaluated to obtain an initial evaluation result, including: according to the data required for different evaluation dimensions included in the evaluation data set, evaluation operations are performed on the model to be evaluated in different evaluation dimensions to obtain reference evaluation results of each evaluation dimension; based on each reference evaluation result, an initial evaluation result is generated; wherein the different evaluation dimensions include at least performance evaluation, robustness evaluation and fairness evaluation of the model.

[0009] In an exemplary embodiment of the present disclosure, an evaluation operation is performed on the model to be evaluated in a performance evaluation dimension to obtain a first reference evaluation result, including: calculating the performance indicator data of the model to be evaluated in executing the current evaluation task, and obtaining the reasoning time data of the model to be evaluated in executing the current evaluation task; collecting the resource consumption data of the model to be evaluated in executing the current evaluation task; and determining the first reference evaluation result based on the performance indicator data, the reasoning time data and the resource consumption data.

[0010] In an exemplary embodiment of the present disclosure, an evaluation operation is performed on the model to be evaluated in a fairness evaluation dimension to obtain a second reference evaluation result, including: obtaining multi-group evaluation data corresponding to the fairness evaluation dimension, the multi-group evaluation data including evaluation data of different groups; performing an evaluation operation on the model to be evaluated using the multi-group evaluation data, and generating a second reference evaluation result based on the evaluation results of the model to be evaluated under the evaluation data of different groups.

[0011] In an exemplary embodiment of the present disclosure, the evaluation content of the task to be evaluated is adjusted based on the initial evaluation results, including: according to the preset correspondence between the evaluation results and the model performance information, obtaining the target model performance information corresponding to the initial evaluation results, the target model performance information is used to reflect the performance of the model to be evaluated in the current evaluation task; based on the target model performance information, adjusting the difficulty and / or type of the task to be evaluated to obtain an adjusted task to be evaluated.

[0012] In an exemplary embodiment of the present disclosure, there are multiple models to be evaluated; the method also includes: performing evaluation operations on multiple models to be evaluated in parallel; wherein, when assigning evaluation tasks to be assigned to models to be evaluated, if there are target evaluation tasks to be assigned with priority labels, the target evaluation tasks to be assigned are assigned to the corresponding models to be evaluated based on the priority labels.

[0013] In an exemplary embodiment of the present disclosure, multiple evaluation tasks are allocated according to the attribute type of the model to be evaluated, and also include: obtaining a reference reasoning time of the evaluation task; and scheduling corresponding computing resources and / or evaluation time for the multiple evaluation tasks according to the reference reasoning time.

[0014] According to one aspect of the present disclosure, there is provided a model evaluation system, including: a task scheduling module, for allocating a variety of evaluation tasks according to the attribute type of the model to be evaluated, wherein different evaluation tasks have their own corresponding evaluation data sets; an evaluation module, for executing the current evaluation task on the model to be evaluated according to the evaluation data set of the current evaluation task, and obtaining an initial evaluation result, wherein the initial evaluation result includes multiple evaluation dimensions; an adjustment module, for adjusting the evaluation content of the task to be evaluated based on the initial evaluation result, so as to perform an evaluation operation on the model to be evaluated according to the adjusted task to be evaluated; and a result generation module, for generating a comprehensive evaluation result of the model to be evaluated according to the initial evaluation results of the various evaluation tasks.

[0015] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, any one of the above methods is implemented.

[0016] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above methods by executing the executable instructions.

[0017] In the technical solution provided by the exemplary embodiment of the present disclosure, on the one hand, according to the attribute type of the model to be evaluated, a variety of evaluation tasks are assigned, and different evaluation tasks have their own corresponding evaluation data. Then, according to the evaluation data set of the current evaluation task, the current evaluation task is performed on the model to be evaluated to obtain the initial evaluation result, which includes multiple evaluation dimensions. The multi-task evaluation of the large model is realized through an automated process, covering a variety of task types, and each task type is evaluated from multiple evaluation dimensions, which improves the evaluation efficiency while improving the coverage and accuracy of the evaluation, avoiding the waste of time and resources caused by traditional manual intervention. On the other hand, based on the initial evaluation results, the evaluation content of the task to be evaluated is adjusted to perform evaluation operations on the model to be evaluated according to the adjusted task to be evaluated, and finally, based on the initial evaluation results of the multiple evaluation tasks, a comprehensive evaluation result of the model to be evaluated is generated. This process can adaptively adjust the evaluation content of subsequent tasks to be evaluated according to the performance of the large model in the evaluated evaluation tasks during the evaluation process, which can ensure the task adaptability and comprehensiveness of the model evaluation, so as to obtain more comprehensive and accurate evaluation results for the large model in a shorter time, ensure that the evaluation results are closer to the actual capabilities of the large model, and improve the adaptability and flexibility of the evaluation. In addition, since the initial evaluation results containing multiple evaluation dimensions are obtained for each evaluation task, the task performance of the large model is comprehensively evaluated from multiple tasks and dimensions, providing support for the development, selection and application of the large model, and having practical applications for optimizing the large model.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0020] Figure 1 A schematic diagram schematically illustrates an application environment of an exemplary embodiment of the present disclosure;

[0021] Figure 2 A flowchart schematically illustrates a model evaluation method of an exemplary embodiment of the present disclosure;

[0022] Figure 3 A flowchart schematically illustrates a method for implementing the allocation of evaluation tasks in an exemplary embodiment of the present disclosure;

[0023] Figure 4 A flowchart of an exemplary embodiment of the present disclosure for evaluating a model to be evaluated in a performance evaluation dimension is schematically shown;

[0024] Figure 5 A flowchart of an evaluation operation of a model to be evaluated in a fairness evaluation dimension in an exemplary embodiment of the present disclosure is schematically shown;

[0025] Figure 6 A schematic diagram schematically showing an initial evaluation result corresponding to a current evaluation task in an exemplary embodiment of the present disclosure;

[0026] Figure 7 A complete flow chart of a model evaluation method in an exemplary embodiment of the present disclosure is schematically shown;

[0027] Figure 8 A schematic diagram schematically shows the composition of a model evaluation system in an exemplary embodiment of the present disclosure;

[0028] Fig. 9 A schematic diagram of an electronic device to which the embodiments of the present disclosure can be applied is shown. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0030] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0031] The current large model evaluation methods cannot fully and objectively reflect the overall performance of large models in multi-task, multi-domain and cross-scenario model evaluation. For example, large model evaluation requires not only accuracy and efficiency, but also multi-dimensional factors such as model transparency, credibility, robustness and fairness. When the evaluation of large models is focused on a single task or indicator, it is impossible to reflect the versatility and multi-task adaptability of the model in multiple scenarios. If feature analysis is performed on large models and then targeted evaluations are conducted, the evaluation process is cumbersome and the computational cost is high. It is not flexible enough when multiple large models need to be evaluated quickly. In addition, with a static task allocation mechanism, the evaluation content (such as difficulty and type) of the evaluation task is fixed before the evaluation process begins, so that the actual capabilities of the large model cannot be adapted during the evaluation process, affecting the efficiency and accuracy of the evaluation, and there is also the problem of incomplete evaluation.

[0032] Based on this, an exemplary embodiment of the present disclosure provides a model evaluation method, which improves the evaluation efficiency, coverage and accuracy of large models through multi-task, multi-dimensional evaluation operations and adaptive adjustment of evaluation content, and provides evaluation support for multi-task, multi-scenario and other application fields. For example, it can provide accurate and systematic evaluation support for the development, selection and application of large models.

[0033] It should be noted that the model evaluation method of the exemplary embodiment of the present disclosure can be applied to any scenario that requires evaluation of a large model, such as a model development scenario, a model selection scenario, etc. For example, in the fields of finance, law, and medicine, scenarios that require high model accuracy or low error rate can provide support for the application of large models by automatically evaluating the performance of large models. The exemplary embodiment of the present disclosure does not limit the application scenario of the model evaluation method.

[0034] The model evaluation method provided by the exemplary embodiments of the present disclosure can be applied to Figure 1 In the application environment shown, the terminal 101 communicates with the server 102 via a network. The data storage system can store data that the server 102 needs to process. The data storage system can be integrated on the server 102, or placed on the cloud or other network servers.

[0035] In an exemplary embodiment, the model evaluation method provided by the exemplary embodiment of the present disclosure may be executed by the server 102, and the corresponding model evaluation device is disposed in the server 102. Correspondingly, in this mode of execution by the server 102, the server 102 may start to execute the steps in the technical solution of the exemplary embodiment of the present disclosure in response to a trigger command, wherein the trigger command may be sent by a terminal used by a user, or may be triggered locally by the server in response to some automation events.

[0036] The server 102 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server 102 may perform background tasks.

[0037] Furthermore, in another exemplary embodiment, the terminal 101 may also have similar functions to the server 102, so as to execute the model evaluation method provided by the exemplary embodiment of the present disclosure.

[0038] The terminal 101 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, an IoT device, etc. The IoT device may be a smart TV, a smart air conditioner, a smart vehicle-mounted device, etc. The terminal 101 may also be referred to as a mobile terminal, a terminal device, a mobile device, etc. The exemplary embodiments of the present disclosure do not limit the type of the terminal 101.

[0039] In addition, the technical solutions of the exemplary embodiments of the present disclosure may also be performed collaboratively by the terminal 101 and the server 102. In this manner of collaborative execution by the terminal 101 and the server 102, some steps in the technical solutions provided by the exemplary embodiments of the present disclosure are performed by the terminal 101, while other steps are performed by the server 102.

[0040] It should be noted that in this mode of collaborative execution by the terminal 101 and the server 102, the steps respectively executed by the terminal 101 and the server 102 can be dynamically adjusted according to actual conditions, and there is no special restriction on this.

[0041] The terminal 101 and the server 102 may be connected directly or indirectly via wireless communication, and the exemplary embodiments of the present disclosure do not impose any special limitation thereto.

[0042] The exemplary embodiment of the present disclosure first provides a model evaluation method, referring to Figure 2 As shown, the model evaluation method of the exemplary embodiment of the present disclosure may include the following steps S210 and S240:

[0043] In step S210, various evaluation tasks are assigned according to the attribute type of the model to be evaluated.

[0044] In an exemplary embodiment of the present disclosure, the model to be evaluated can be a pre-trained model uploaded or a model loaded through an API (Application Programming Interface). The model to be evaluated can be a fine-tuned model for a specific task or a basic pre-trained model. The exemplary embodiment of the present disclosure supports common model frameworks (such as TensorFlow learning framework, PyTorch learning framework) and multiple pre-trained model formats, without limitation.

[0045] The attribute type of the model to be evaluated can refer to the task processing type of the model to be evaluated, such as text generation, classification, language understanding, question answering, etc. A variety of evaluation tasks can be assigned to the model to be evaluated according to its attribute type. Of course, the task assignment can also be combined with the evaluation requirements of the model to be evaluated.

[0046] Among them, different evaluation tasks have their own corresponding evaluation data sets, which means that it is necessary to ensure that the correspondence between the data sets and the evaluation tasks is accurate.

[0047] Specifically, it is possible to obtain data sets from local or cloud sources, manage data set versions and storage structures, and provide accurate data support for model evaluation. Evaluation data sets include general evaluation data sets (such as GLUE data sets, SQuAD data sets, etc.) and business-related specific data sets. It also provides an interface for users to upload custom data sets, without any restrictions.

[0048] Optionally, the data can be standardized according to the requirements of the evaluation task, including but not limited to text cleaning, annotation format conversion, data segmentation, etc., and it can also ensure that the data is in a usable state before entering the model.

[0049] In step S220, the current evaluation task is executed on the model to be evaluated according to the evaluation data set of the current evaluation task to obtain an initial evaluation result, which includes multiple evaluation dimensions.

[0050] In an exemplary embodiment of the present disclosure, the current evaluation task refers to the evaluation task currently being evaluated among multiple evaluation tasks assigned to the model to be evaluated. Each evaluation task includes multiple evaluation dimensions, and accordingly, the obtained initial evaluation result also includes multiple evaluation dimensions. The initial evaluation result is a comprehensive result of the model to be evaluated in the current evaluation task, which is used to reflect the performance of the model to be evaluated in the current evaluation task.

[0051] Among them, multiple evaluation dimensions include at least performance evaluation, robustness evaluation and fairness evaluation. Performance evaluation includes the performance of the model in terms of accuracy, precision, recall rate, F1 score, BLEU (Bilingual Evaluation Understudy) score and other indicators, and also includes the evaluation of model reasoning efficiency, such as the reasoning time and resource consumption data for each evaluation task. Of course, other performance evaluation dimensions can also be included according to the type of evaluation task. For example, for generation tasks, the quality of the generated text can also be evaluated through an automatic scoring mechanism (such as BLEU).

[0052] Robustness refers to the ability of a large model to maintain stability and effectiveness in the face of various abnormal situations, and the ability of a large model to correctly output results when the input data changes. When facing abnormal data or noisy data, the large model has strong anti-interference ability, which means that it has high robustness, and its performance and generalization ability are relatively better. In addition, a large model with high robustness can still maintain stable output when there is a small deviation in the input data without "catastrophic" effects. The performance of the model to be evaluated under harsh conditions can be evaluated by introducing adversarial samples, noisy data or boundary conditions. For example, adversarial samples can be generated using adversarial sample testing tools (such as FGSM (Fast Gradient Sign Method)) to test the robustness of the model to be evaluated, and there is no restriction on this.

[0053] Fairness evaluation refers to whether the evaluation model has biases against different groups (such as different genders, races, occupations, etc.), and is used to measure whether the model performs fairly to different groups. Fairness evaluation can be performed using data corresponding to different groups.

[0054] In step S230, the evaluation content of the task to be evaluated is adjusted based on the initial evaluation result, so as to perform an evaluation operation on the model to be evaluated according to the adjusted task to be evaluated.

[0055] In an exemplary embodiment of the present disclosure, when the model to be evaluated completes the current evaluation task, the performance of the model to be evaluated can be analyzed based on the initial evaluation results, so as to dynamically adjust the evaluation content of subsequent evaluation tasks, thereby ensuring that the evaluation process is efficient and comprehensive. For example, if the model to be evaluated performs well in the current evaluation task, the difficulty of the task can be increased (such as more complex input or longer context) to test the model's limit capabilities. If the model to be evaluated performs stably in the current evaluation task, cross-domain or cross-language evaluation tasks can be added to detect the generalization ability of the model.

[0056] In step S240, a comprehensive evaluation result of the model to be evaluated is generated based on the initial evaluation results of the various evaluation tasks.

[0057] In an exemplary embodiment of the present disclosure, a comprehensive evaluation result of the model to be evaluated is generated based on the initial evaluation results of various evaluation tasks. The comprehensive evaluation results can list in detail the performance of the model to be evaluated in each task and each dimension. At the same time, the comprehensive evaluation results can also provide suggestion information for the evaluation tasks or dimensions. For example, in the evaluation process of the model to be evaluated, if the accuracy of the question and answer results of the model to be evaluated is low, it can be suggested to fine-tune the model with the question and answer as the direction of fine-tuning. For another example, during the evaluation process, it is found that the accuracy of understanding foreign languages ​​is much higher than that of understanding Chinese. It can be suggested to fine-tune the model or adjust the model architecture with Chinese understanding as the direction of fine-tuning. The exemplary embodiment of the present disclosure does not limit the specific content of the suggestion information, and it is generated according to the specific evaluation results.

[0058] Among them, when generating the comprehensive evaluation results of the model to be evaluated based on the initial evaluation results of multiple evaluation tasks, the evaluation results of multiple evaluation tasks and multiple evaluation dimensions for each evaluation task may be included. Exemplarily, the performance indicator data (such as accuracy, recall rate, BLEU score, etc.) of each evaluation task, reasoning time data, resource consumption data, robustness evaluation results, first reference evaluation results and second evaluation results, as well as suggestion information for each evaluation result may be included.

[0059] Based on this, the evaluation results of the model to be evaluated under multiple evaluation tasks can be fully characterized, providing accurate and systematic evaluation support for the development, selection and application of large models. By comprehensively providing comprehensive evaluation results of the model to be evaluated, it is conducive to improving the transparency and credibility of the selection and development of large models, ensuring that the model complies with technical standards and ethical norms, and promoting the safe and rational use of large models.

[0060] In the technical solution provided by the exemplary embodiment of the present disclosure, on the one hand, according to the attribute type of the model to be evaluated, a variety of evaluation tasks are assigned, and different evaluation tasks have their own corresponding evaluation data. Then, according to the evaluation data set of the current evaluation task, the current evaluation task is performed on the model to be evaluated to obtain the initial evaluation result, which includes multiple evaluation dimensions. The multi-task evaluation of the large model is realized through an automated process, covering a variety of task types, and each task type is evaluated from multiple evaluation dimensions, which improves the evaluation efficiency while improving the coverage and accuracy of the evaluation, avoiding the waste of time and resources caused by traditional manual intervention. On the other hand, based on the initial evaluation results, the evaluation content of the task to be evaluated is adjusted to perform evaluation operations on the model to be evaluated according to the adjusted task to be evaluated, and finally, based on the initial evaluation results of the multiple evaluation tasks, a comprehensive evaluation result of the model to be evaluated is generated. This process can adaptively adjust the evaluation content of subsequent tasks to be evaluated according to the performance of the large model in the evaluated evaluation tasks during the evaluation process, which can ensure the task adaptability and comprehensiveness of the model evaluation, so as to obtain more comprehensive and accurate evaluation results for the large model in a shorter time, ensure that the evaluation results are closer to the actual capabilities of the large model, and improve the adaptability and flexibility of the evaluation. In addition, since the initial evaluation results containing multiple evaluation dimensions are obtained for each evaluation task, the task performance of the large model is comprehensively evaluated from multiple tasks and dimensions, providing support for the development, selection and application of the large model, and having practical applications for optimizing the large model.

[0061] In an exemplary embodiment, a method for allocating evaluation tasks is provided. Figure 3 As shown in the figure, various evaluation tasks are assigned according to the attribute type of the model to be evaluated, including:

[0062] Step S310: Obtain multiple evaluation tasks with matching attribute types and assign them to the model to be evaluated.

[0063] Step S320: According to the correspondence between the evaluation tasks and the evaluation data sets, the evaluation data sets corresponding to the various evaluation tasks are obtained and allocated to the models to be evaluated.

[0064] Among them, a task queue can be maintained, and the task queue can be based on the FIFO (First Input First Output) principle, but the task priority can be dynamically adjusted according to the evaluation requirements. For example, it can respond to the user's labeling operation on a certain / certain evaluation tasks, and determine the priority label of the evaluation task according to the labeling result, so as to achieve priority evaluation of urgent tasks according to the priority label of the evaluation task. That is, when the evaluation task to be assigned is assigned to the model to be evaluated, if there is a target evaluation task to be assigned with a priority label, the target evaluation task to be assigned is assigned to the corresponding model to be evaluated based on the priority label.

[0065] Of course, if the evaluation task is not marked, the evaluation task can be scheduled and assigned to the model to be evaluated according to the execution order of the task queue.

[0066] It should be understood that when allocating evaluation tasks to models to be evaluated based on task queues, in addition to considering priorities, the matching relationship between the evaluation tasks (and their evaluation data sets) and the attribute types of the models to be evaluated should also be considered.

[0067] Through the matching relationship, it can be ensured that the evaluation task that meets the characteristics of the model to be evaluated is assigned to it, thereby ensuring the accuracy of the model evaluation.

[0068] In an exemplary embodiment, performing the current evaluation task on the to-be-evaluated model according to the evaluation data set of the current evaluation task to obtain the initial evaluation result may include:

[0069] First, according to the data required for different evaluation dimensions included in the evaluation data set, evaluation operations are performed on the evaluation model in different evaluation dimensions to obtain reference evaluation results of each evaluation dimension, and then initial evaluation results are generated based on each reference evaluation result.

[0070] Specifically, as mentioned above, different evaluation dimensions include at least the performance evaluation, robustness evaluation and fairness evaluation of the model. The data and data types required for each evaluation dimension are different. Therefore, it is necessary to use the data required by each evaluation dimension to perform the corresponding evaluation operation on the model to be evaluated, and obtain the reference evaluation results corresponding to each evaluation dimension. Finally, the initial evaluation results of the model to be evaluated for executing the current evaluation task can be obtained based on all the reference evaluation results.

[0071] in, Figure 4 A flowchart of evaluating the performance evaluation dimension of the model to be evaluated is shown. Figure 4 As shown, the process includes:

[0072] Step S410: Calculate the performance indicator data of the model to be evaluated for executing the current evaluation task, and obtain the reasoning time consumption data of the model to be evaluated for executing the current evaluation task.

[0073] Among them, the performance indicator data include but are not limited to accuracy, precision, recall rate, F1 score, BLEU score, etc. The calculation methods of these performance indicator data and related technologies are the same and no special restrictions are made on this. In addition, the first reference evaluation result corresponding to the performance evaluation dimension may also include task logs for performance indicator data for viewing, analysis and other processing.

[0074] The inference time consumption data of the model to be evaluated for executing the current evaluation task can be obtained by monitoring from the start to the end of the evaluation of the model to be evaluated.

[0075] Step S420: Collect resource consumption data of the model to be evaluated when executing the current evaluation task.

[0076] Resource consumption data and inference time data can both measure the computational efficiency of the model to be evaluated. Resource consumption data includes GPU (Graphics Processing Unit) time, memory usage, CPU (Central Processing Unit) usage, etc. Resource consumption data can also be obtained through monitoring from the start to the end of the evaluation of the model to be evaluated.

[0077] Step S430: Determine a first reference evaluation result based on the performance indicator data, the reasoning time consumption data, and the resource consumption data.

[0078] The performance indicator data, reasoning time data and resource consumption data in the evaluation process can be integrated to determine the first reference evaluation result.

[0079] Among them, the first reference evaluation result can be an evaluation score value, which can be comprehensively calculated based on performance indicator data, reasoning time data and resource consumption data.

[0080] Specifically, the performance indicator data, reasoning time data and resource consumption data can be standardized first, and then weighted summed based on the standardized results and the weight information corresponding to each dimension to obtain the first reference evaluation result. Among them, the range of the weight information corresponding to each dimension is [0,1], and the specific value can be set according to the actual task requirements. For example, if you need to focus on computing efficiency, then increase the weight information corresponding to the reasoning time data and resource consumption data, and reduce the weight information corresponding to the performance indicator data. Alternatively, if you pay more attention to the performance indicator data, increase the weight information corresponding to the performance indicator data. Of course, the exemplary embodiments of the present disclosure can also calculate the first reference evaluation result in other ways, such as simply summing the standardized results, and so on.

[0081] By obtaining the first reference evaluation result through comprehensive performance indicator data, the reasoning time data and the resource consumption data, the model can be evaluated from the perspective of model performance and computing efficiency, which increases the dimension of model evaluation and improves the coverage of model evaluation.

[0082] Figure 5 A flowchart of evaluating the evaluation model in the fairness evaluation dimension is shown. Figure 5 As shown, the process includes:

[0083] Step S510: Obtain multi-group evaluation data corresponding to the fairness evaluation dimension.

[0084] As mentioned above, fairness refers to whether the model performs consistently across different groups. A fair model should perform consistently across groups of different genders, races, ages, and so on, and should not have significant deviations. Among them, fairness indicators may include equal opportunities and equal errors. Multi-group evaluation data includes evaluation data from different groups. Based on the evaluation data from different groups, the performance differences of the model to be evaluated in different groups (such as equal opportunities, equal errors, etc.) can be calculated. The specific calculation method is the same as that of the relevant technology and will not be repeated here.

[0085] Step S520: using the multi-group evaluation data to perform an evaluation operation on the model to be evaluated, and generating a second reference evaluation result according to the evaluation results of the model to be evaluated under the evaluation data of different groups.

[0086] Based on the evaluation data of different groups, the performance differences of the model to be evaluated in different groups (such as fairness indicators such as equal opportunity and equal error) can be calculated, so as to calculate the second reference evaluation result based on at least one fairness indicator obtained. For example, at least one fairness indicator can be standardized, and the standardized fairness indicators can be fused and calculated to obtain the second reference evaluation result. The fusion method can be simple summation, weighted summation, etc., which is not limited.

[0087] By performing fairness testing on the model to be tested, the model evaluation dimension is increased and the coverage of model evaluation is improved.

[0088] After obtaining the reference evaluation results corresponding to each evaluation dimension, the exemplary embodiments of the present disclosure may generate an initial evaluation result according to the reference evaluation results corresponding to each evaluation dimension and the comprehensive results of each reference evaluation result.

[0089] For example, Figure 6 A schematic diagram showing an initial evaluation result corresponding to a current evaluation task is shown, such as Figure 6 As shown in , the initial evaluation results include the evaluation process information of performance evaluation, robustness evaluation and fairness evaluation, the evaluation results corresponding to each evaluation dimension and the comprehensive results. Figure 6 A detailed description of the current evaluation task of the evaluation model can be obtained, providing comprehensive information support for evaluation analysis and management. Figure 6 Just an example.

[0090] In an exemplary embodiment, adjusting the evaluation content of the task to be evaluated based on the initial evaluation result may include:

[0091] First, according to the correspondence between the preset evaluation results and the model performance information, the target model performance information corresponding to the initial evaluation results is obtained, and then based on the target model performance information, the difficulty and / or type of the task to be evaluated is adjusted to obtain the adjusted task to be evaluated.

[0092] Among them, the model performance information is used to reflect the performance of the model to be evaluated in the current evaluation task, such as excellent, stable, poor, etc. The corresponding relationship between the evaluation results and different levels of model performance information can be pre-set, for example, "80-100" corresponds to "excellent", "75-80" corresponds to "stable", "65-75" corresponds to "good", and "below 65" corresponds to "poor". Of course, this is only an example, and the corresponding relationship between the evaluation results and the model performance information can be set according to actual needs.

[0093] Furthermore, after obtaining the initial evaluation results, the target model performance information corresponding to the initial evaluation results can be obtained according to the preset correspondence between the evaluation results and the model performance information.

[0094] Based on the target model performance information, the difficulty and / or type of the task to be evaluated can be adjusted according to the set rules and the target model performance information. For example, if the model to be tested performs well under the current evaluation task, the task difficulty can be increased (such as more complex input or longer context), or if the model performance is stable, the task to be evaluated is adjusted to a task that is not cross-domain or cross-language, so as to test the model's limit ability and generalization ability. Alternatively, if the performance of the model to be evaluated on the current evaluation task is far lower than expected (preset), the subsequent evaluation operations of similar tasks can be reduced or cancelled, or, if the model to be evaluated performs poorly in the current evaluation task, the task difficulty of the task to be evaluated is reduced or a simpler task is assigned, so that the evaluation process always meets the ability of the model to be evaluated, improves the adaptability and flexibility of the evaluation, and obtains a more comprehensive evaluation result.

[0095] In an exemplary embodiment, the number of models to be evaluated can be multiple. In this scenario, the evaluation operations of multiple models to be evaluated can be performed in parallel, that is, multiple evaluation tasks are allowed to run simultaneously, and evaluation tasks are performed for different models to be evaluated respectively, so as to meet the needs of multi-task and multi-model scenarios. For example, the efficiency of parallel execution of evaluation operations of multiple models to be evaluated can be improved with the help of a distributed computing framework (such as multiple GPUs or multiple TPUs (tensor processing units)).

[0096] In an exemplary embodiment, allocating multiple evaluation tasks according to the attribute type of the model to be evaluated further includes:

[0097] First, the reference reasoning time of the evaluation task is obtained, and then the corresponding computing resources and / or evaluation time are scheduled for various evaluation tasks according to the reference reasoning time.

[0098] The reference reasoning time can reflect the reasoning time required for the model to be evaluated on the evaluation task, and can be set based on experience or task requirements. For example, it can be determined based on the time required for the evaluation task in historical evaluation operations, such as taking the average of the time required in historical evaluation operations. Based on the reference reasoning time, the computing resources and / or evaluation time allocated to the evaluation task can be automatically adjusted.

[0099] For example, if the reference reasoning time of a certain evaluation task indicates that the evaluation task requires a longer reasoning time, more computing resources can be allocated to the evaluation task or the evaluation time can be extended to ensure that the process of executing the evaluation task on the evaluation model proceeds normally and improve the stability of the evaluation process.

[0100] In addition, the exemplary embodiments of the present disclosure may also expand more or finer-grained evaluation dimensions for each evaluation task, which are not limited to the above-mentioned ones, for example, may also include security compliance detection dimensions, etc.

[0101] Figure 7 A complete flow chart of a model evaluation method is shown below. Figure 7 A model evaluation method according to an exemplary embodiment of the present disclosure is described.

[0102] First, load the model to be evaluated.

[0103] Secondly, various evaluation tasks are assigned according to the attribute type of the model to be evaluated.

[0104] Next, based on the evaluation data set of the current evaluation task, the current evaluation task is performed on the model to be evaluated.

[0105] Among them, the performance indicator data of the model to be evaluated executing the current evaluation task is calculated, the reasoning time consumption data of the model to be evaluated executing the current evaluation task is obtained, and the resource consumption data of the model to be evaluated executing the current evaluation task is collected.

[0106] And perform robustness testing on the model to be evaluated to obtain robustness evaluation results, and obtain multi-group evaluation data corresponding to the fairness evaluation dimension, and use the multi-group evaluation data to perform evaluation operations on the model to be evaluated, so as to obtain fairness evaluation results based on the evaluation results of the model to be evaluated under the evaluation data of different groups.

[0107] Finally, based on the initial evaluation results of various evaluation tasks, the comprehensive evaluation results of the model to be evaluated are generated.

[0108] It should be noted that the details of each step have been recorded in the above exemplary embodiments and will not be repeated here.

[0109] In the technical solution provided by the exemplary embodiment of the present disclosure, on the one hand, according to the attribute type of the model to be evaluated, a variety of evaluation tasks are assigned, and different evaluation tasks have their own corresponding evaluation data. Then, according to the evaluation data set of the current evaluation task, the current evaluation task is performed on the model to be evaluated to obtain the initial evaluation result, which includes multiple evaluation dimensions. The multi-task evaluation of the large model is realized through an automated process, covering a variety of task types, and each task type is evaluated from multiple evaluation dimensions, which improves the evaluation efficiency while improving the coverage and accuracy of the evaluation, avoiding the waste of time and resources caused by traditional manual intervention. On the other hand, based on the initial evaluation results, the evaluation content of the task to be evaluated is adjusted to perform evaluation operations on the model to be evaluated according to the adjusted task to be evaluated, and finally, based on the initial evaluation results of the multiple evaluation tasks, a comprehensive evaluation result of the model to be evaluated is generated. This process can adaptively adjust the evaluation content of subsequent tasks to be evaluated according to the performance of the large model in the evaluated evaluation tasks during the evaluation process, which can ensure the task adaptability and comprehensiveness of the model evaluation, so as to obtain more comprehensive and accurate evaluation results for the large model in a shorter time, ensure that the evaluation results are closer to the actual capabilities of the large model, and improve the adaptability and flexibility of the evaluation. In addition, since the initial evaluation results containing multiple evaluation dimensions are obtained for each evaluation task, the task performance of the large model is comprehensively evaluated from multiple tasks and dimensions, providing support for the development, selection and application of the large model, and having practical applications for optimizing the large model.

[0110] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0111] For further reference, Figure 8 As shown, in an exemplary embodiment of the present disclosure, a model evaluation system 800 is provided, including a task scheduling module 810, an evaluation module 820, an adjustment module 830 and a result generation module 840. Among them:

[0112] The task scheduling module 810 is used to allocate multiple evaluation tasks according to the attribute type of the model to be evaluated, wherein different evaluation tasks have their own corresponding evaluation data sets; the evaluation module 820 is used to execute the current evaluation task on the model to be evaluated according to the evaluation data set of the current evaluation task, and obtain the initial evaluation result, which includes multiple evaluation dimensions; the adjustment module 830 is used to adjust the evaluation content of the task to be evaluated based on the initial evaluation result, so as to perform evaluation operations on the model to be evaluated according to the adjusted task to be evaluated; the result generation module 840 is used to generate a comprehensive evaluation result of the model to be evaluated based on the initial evaluation results of multiple evaluation tasks.

[0113] In an exemplary embodiment of the present disclosure, the task scheduling module 810 is configured to execute: obtaining multiple evaluation tasks with matching attribute types and assigning them to the model to be evaluated; obtaining evaluation data sets corresponding to the multiple evaluation tasks and assigning them to the model to be evaluated based on the correspondence between the evaluation tasks and the evaluation data sets.

[0114] In an exemplary embodiment of the present disclosure, the evaluation module 820 is configured to perform: according to the data required for different evaluation dimensions included in the evaluation data set, respectively perform evaluation operations on the evaluation model in different evaluation dimensions to obtain reference evaluation results of each evaluation dimension; based on each reference evaluation result, generate an initial evaluation result; wherein the different evaluation dimensions include at least performance evaluation, robustness evaluation and fairness evaluation of the model.

[0115] In an exemplary embodiment of the present disclosure, the evaluation module 820 is configured to perform: calculating the performance indicator data of the model to be evaluated when executing the current evaluation task, and obtaining the reasoning time data of the model to be evaluated when executing the current evaluation task; collecting the resource consumption data of the model to be evaluated when executing the current evaluation task; and determining the first reference evaluation result based on the performance indicator data, the reasoning time data and the resource consumption data.

[0116] In an exemplary embodiment of the present disclosure, the evaluation module 820 is configured to execute: obtaining multi-group evaluation data corresponding to the fairness evaluation dimension, the multi-group evaluation data including evaluation data of different groups; performing an evaluation operation on the model to be evaluated using the multi-group evaluation data, and generating a second reference evaluation result based on the evaluation results of the model to be evaluated under the evaluation data of different groups.

[0117] In an exemplary embodiment of the present disclosure, the adjustment module 830 is configured to perform: according to a preset correspondence between the evaluation results and the model performance information, obtain the target model performance information corresponding to the initial evaluation results, the target model performance information is used to reflect the performance of the model to be evaluated in the current evaluation task; based on the target model performance information, adjust the difficulty and / or type of the task to be evaluated to obtain an adjusted task to be evaluated.

[0118] In an exemplary embodiment of the present disclosure, there are multiple models to be evaluated; the model testing system 800 performs evaluation operations on multiple models to be evaluated in parallel; wherein, when assigning evaluation tasks to be assigned to models to be evaluated, if there are target evaluation tasks to be assigned with priority labels, the target evaluation tasks to be assigned are assigned to the corresponding models to be evaluated based on the priority labels.

[0119] In an exemplary embodiment of the present disclosure, the task scheduling module 810 is further configured to execute: obtaining a reference reasoning time of the evaluation task; and scheduling corresponding computing resources and / or evaluation time for various evaluation tasks respectively according to the reference reasoning time.

[0120] Since the details of each functional module of the model evaluation system of the exemplary embodiment of the present disclosure have been recorded in the exemplary embodiment of the above-mentioned model evaluation method, they will not be repeated here.

[0121] It should be noted that although several modules or units of the model evaluation system are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0122] The exemplary embodiments of the present disclosure also provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the above-mentioned model evaluation method is implemented.

[0123] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0124] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0125] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0126] The computer program may be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device may convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program is run on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device may execute) the method steps of various exemplary embodiments of the present disclosure, such as the model evaluation system described above.

[0127] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided. It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as a system, method, or program product. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which may be collectively referred to herein as a "circuit", "module", or "system".

[0128] Refer to the following Fig. 9 hereinafter describes an electronic device 900 according to such an embodiment of the present disclosure. Fig. 9 The electronic device 900 shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure. Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include, but are not limited to: the at least one processing unit 910, the at least one storage unit 920, a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910), and a display unit 940.

[0129] The storage unit stores program codes, which can be executed by the processing unit 910, so that the processing unit 910 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0130] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 921 and / or a cache storage unit 922, and may further include a read-only storage unit (ROM) 923. The storage unit 920 may also include a program / utility 924 having a set (at least one) of program modules 925, such program modules 925 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include the implementation of a network environment.

[0131] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0132] The electronic device 900 may also communicate with one or more external devices 1000 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 950. Furthermore, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0133] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0134] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0135] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A model evaluation method, characterized in that: The method comprises: According to the attribute type of the model to be evaluated, a variety of evaluation tasks are assigned, where different evaluation tasks have their own corresponding evaluation data sets; According to the evaluation data set of the current evaluation task, the current evaluation task is executed on the model to be evaluated to obtain an initial evaluation result, wherein the initial evaluation result includes multiple evaluation dimensions; Adjusting the evaluation content of the task to be evaluated based on the initial evaluation result, so as to perform an evaluation operation on the model to be evaluated according to the adjusted task to be evaluated; Based on the initial evaluation results of the multiple evaluation tasks, a comprehensive evaluation result of the model to be evaluated is generated.

2. The method according to claim 1, characterized in that According to the attribute type of the model to be evaluated, various evaluation tasks are assigned, including: Acquire multiple evaluation tasks matching the attribute type and assign them to the model to be evaluated; According to the correspondence between the evaluation tasks and the evaluation data sets, the evaluation data sets corresponding to the multiple evaluation tasks are obtained and allocated to the model to be evaluated.

3. The method according to claim 1, characterized in that The step of performing the current evaluation task on the model to be evaluated based on the evaluation data set of the current evaluation task to obtain an initial evaluation result includes: According to the data required for different evaluation dimensions included in the evaluation data set, respectively, performing evaluation operations on the model to be evaluated in different evaluation dimensions to obtain reference evaluation results for each evaluation dimension; Based on the reference evaluation results, generating initial evaluation results; Among them, different evaluation dimensions include at least model performance evaluation, robustness evaluation and fairness evaluation.

4. The method according to claim 3, characterized in that The model to be evaluated is evaluated in a performance evaluation dimension to obtain a first reference evaluation result, including: Calculate the performance index data of the model to be evaluated for executing the current evaluation task, and obtain the reasoning time consumption data of the model to be evaluated for executing the current evaluation task; Collecting resource consumption data of the model to be evaluated when executing the current evaluation task; Based on the performance indicator data, the reasoning time data and the resource consumption data, the first reference evaluation result is determined.

5. The method according to claim 3, characterized in that: The model to be evaluated is evaluated in the fairness evaluation dimension to obtain a second reference evaluation result, including: Acquire multi-group evaluation data corresponding to the fairness evaluation dimension, wherein the multi-group evaluation data includes evaluation data of different groups; The model to be evaluated is evaluated using the multi-group evaluation data, and the second reference evaluation result is generated according to the evaluation results of the model to be evaluated under the evaluation data of different groups.

6. The method according to claim 1, characterized in that The adjusting the evaluation content of the task to be evaluated based on the initial evaluation result includes: According to the preset correspondence between the evaluation result and the model performance information, the target model performance information corresponding to the initial evaluation result is obtained, and the target model performance information is used to reflect the performance of the model to be evaluated in the current evaluation task; Based on the target model performance information, the difficulty and / or type of the task to be evaluated is adjusted to obtain an adjusted task to be evaluated.

7. The method according to claim 1, characterized in that The number of the models to be evaluated is multiple; the method further includes: Execute the evaluation operation of multiple models to be evaluated in parallel; When allocating the evaluation task to be assigned to the model to be evaluated, if there is a target evaluation task to be assigned with a priority label, the target evaluation task to be assigned is allocated to the corresponding model to be evaluated based on the priority label.

8. The method according to any one of claims 1 to 7, characterized in that The method of allocating multiple evaluation tasks according to the attribute type of the model to be evaluated also includes: Obtain reference reasoning time for the evaluation task; According to the reference reasoning time consumption, corresponding computing resources and / or evaluation time are respectively scheduled for the various evaluation tasks.

9. A model evaluation system, characterized in that: The system comprises: The task scheduling module is used to allocate multiple evaluation tasks according to the attribute type of the model to be evaluated, wherein different evaluation tasks have their own corresponding evaluation data sets; An evaluation module, configured to execute the current evaluation task on the model to be evaluated according to the evaluation data set of the current evaluation task, and obtain an initial evaluation result, wherein the initial evaluation result includes multiple evaluation dimensions; An adjustment module, configured to adjust the evaluation content of the task to be evaluated based on the initial evaluation result, so as to perform an evaluation operation on the model to be evaluated according to the adjusted task to be evaluated; The result generation module is used to generate a comprehensive evaluation result of the model to be evaluated based on the initial evaluation results of the multiple evaluation tasks.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

11. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 8 by executing the executable instructions.

Images