Model data processing method and device, equipment and medium

Abstract

The embodiment of the invention discloses a model data processing method and device, equipment and a medium. The method comprises the following steps: if an evaluation request for a specified model is received, obtaining task category information of a task needing to be executed by the specified model and scene information of a scene applied by the specified model, and then obtaining test data and evaluation parameters corresponding to the specified model based on the task category information and the scene information, and testing the specified model based on the test data to generate a model performance evaluation result of the specified model under the evaluation parameters. According to the technical scheme, the reliability of model performance evaluation is improved, and the reliability of model data processing is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular, to a model data processing method, a model data processing device, an electronic device, and a computer-readable medium. Background Art

[0002] At present, the application of machine learning models (models for short) is ubiquitous, covering all aspects from personal life to business operations. By continuously analyzing and learning large amounts of data, models can help users improve work efficiency, optimize decision-making processes, save time and resources, and even improve health and quality of life. However, there is a lack of effective solutions for model performance evaluation in related technologies, and the reliability of model performance evaluation is low. Summary of the invention

[0003] The embodiments of the present application provide a model data processing method, apparatus, device and medium, which improve the reliability of model performance evaluation and ensure the reliability of model data processing.

[0004] In the first aspect, an embodiment of the present application provides a model data processing method, the method comprising: if an evaluation request for a specified model is received, obtaining task category information of the task to be performed by the specified model, and scenario information of the scenario in which the specified model is applied; based on the task category information and the scenario information, obtaining test data and evaluation parameters corresponding to the specified model; based on the test data, testing the specified model, generating a model performance evaluation result of the specified model under the evaluation parameters.

[0005] In the second aspect, an embodiment of the present application provides a model data processing device, which includes: a first acquisition module, configured to obtain task category information of the task to be performed by the specified model and scenario information of the scenario in which the specified model is applied if an evaluation request for the specified model is received; a second acquisition module, configured to obtain test data and evaluation parameters corresponding to the specified model based on the task category information and the scenario information; and a generation module, configured to generate a model performance evaluation result of the specified model under the evaluation parameters based on the test data for the specified model.

[0006] In one embodiment of the present application, based on the aforementioned scheme, the second acquisition module is specifically configured as follows: selecting test data that matches both the task category information and the scenario information from the test data set, and using the selected test data as the test data for the specified model; and selecting an evaluation dimension that matches both the task category information and the scenario information from the evaluation dimension set, and using the evaluation indicator corresponding to the selected evaluation dimension as the evaluation parameter of the specified model.

[0007] In one embodiment of the present application, based on the aforementioned scheme, the second acquisition module is further configured to: obtain task category information of tasks to be performed by each evaluation model, and scenario information of the scenarios in which each evaluation model is applied; based on the task category information and scenario information of each evaluation model, obtain sample data corresponding to each evaluation model; perform data preparation processing on the sample data corresponding to each evaluation model to obtain test data corresponding to each evaluation model; wherein the data preparation processing includes at least one of data cleaning, data preprocessing, and label generation; associate the task category information, scenario information, and test data of each evaluation model to generate the test data set.

[0008] In one embodiment of the present application, based on the aforementioned scheme, the second acquisition module is further configured to: obtain task category information of tasks to be performed by each evaluation model, and scenario information of the scenarios in which each evaluation model is applied; determine the evaluation dimensions of each evaluation model based on the task category information and scenario information of each evaluation model, and determine the evaluation indicators based on the determined evaluation dimensions; wherein the determined evaluation dimensions include at least one of task execution quality dimension, generalization ability dimension, sample bias dimension, and resource consumption dimension; associate the task category information, scenario information, evaluation dimension, and evaluation indicator of each evaluation model to generate the evaluation dimension set.

[0009] In one embodiment of the present application, based on the aforementioned scheme, the evaluation parameters include evaluation indicators corresponding to the evaluation dimensions; the generation module is specifically configured as follows: inputting the test data into the specified model to execute the task and obtain the task execution result; calculating the indicator value corresponding to the evaluation indicator based on the task execution result; and generating the model performance evaluation result of the specified model under the evaluation parameters based on the indicator value.

[0010] In one embodiment of the present application, based on the aforementioned scheme, the generation module is further specifically configured as follows: if the evaluation dimension includes at least one of a task execution quality dimension and a generalization ability dimension, then the loss value between the task execution result and the expected result of the task is obtained, and the indicator value corresponding to the evaluation indicator is calculated based on the loss value; if the evaluation dimension includes at least one of a sample bias dimension and a resource consumption dimension, then the indicator value corresponding to the evaluation indicator is calculated based on the attribute information of the test data and at least one of the loss values.

[0011] In one embodiment of the present application, based on the aforementioned scheme, the device also includes an optimization module, configured to: determine the potential problems of the specified model based on the model performance evaluation results; optimize the specified model from the optimization direction corresponding to the potential problems to obtain the optimized specified model; and perform the task through the optimized specified model.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors; and a memory for storing one or more computer programs, wherein when the one or more computer programs are executed by the one or more processors, the electronic device implements the model data processing method as described above.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processor, the model data processing method as described above is implemented.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, including computer instructions, which, when executed by a processor, implement the model data processing method as described above.

[0015] In the technical solution provided in the embodiments of the present application: in response to receiving an evaluation request for a specified model, task category information of the task to be performed by the specified model and scenario information of the scenario in which the specified model is applied are obtained, and the task category information and scenario information are used to obtain test data and evaluation parameters corresponding to the specified model. Thereafter, the specified model is tested using the test data to generate a model performance evaluation result of the specified model under the evaluation parameters, thereby achieving a simple and accurate performance evaluation of the specified model.

[0016] Therefore, with the help of the task category information and scenario information corresponding to the specified model, the test data and evaluation parameters corresponding to the specified model are obtained to perform performance evaluation on the specified model. The actual task category and application scenario of the specified model are taken into consideration, the accuracy of the model performance evaluation is improved, and it is applicable to various models, which improves the efficiency and flexibility of the model performance evaluation, and avoids the problem of low evaluation accuracy caused by the large limitations of the model performance evaluation tool, thereby further improving the accuracy of the model performance evaluation; therefore, the reliability of the model performance evaluation in the present application scheme is high, thereby making the reliability of the model data processing high.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of an exemplary implementation environment to which the technical solution of the embodiments of the present application can be applied.

[0019] Figure 2 is a flow chart of a model data processing method shown in an exemplary embodiment of the present application.

[0020] Figure 3 It is a schematic diagram of a model data processing system shown in an exemplary embodiment of the present application.

[0021] Figure 4 is a flowchart of a model data processing method shown in another exemplary embodiment of the present application.

[0022] Figure 5 is a schematic diagram of a model data processing method shown in another exemplary embodiment of the present application.

[0023] Figure 6 It is a block diagram of a model data processing device shown in an exemplary embodiment of the present application.

[0024] Figure 7 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0025] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments that are the same as the present application. Instead, they are only examples of devices and methods that are the same as some aspects of the present application as detailed in the attached claims.

[0026] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0027] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0028] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0029] It should be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0030] At present, the application of machine learning models (models for short) is ubiquitous, covering all aspects from personal life to business operations. By continuously analyzing and learning large amounts of data, models can help users improve work efficiency, optimize decision-making processes, save time and resources, and even improve health and quality of life. However, there is a lack of effective solutions for model performance evaluation in related technologies, and the reliability of model performance evaluation is low.

[0031] Therefore, in order to better implement the performance evaluation of the model and improve the reliability of the model performance evaluation, this application provides a model data processing solution. Figure 1 , Figure 1 1 is a schematic diagram of an implementation environment involved in the present application. The implementation environment mainly includes a terminal device 101 and a server 102; wherein:

[0032] Terminal devices 101 include but are not limited to extended reality devices (virtual reality devices, augmented reality devices, mixed reality devices, etc.), mobile phones, computers (tablet computers, laptops, desktop computers, etc.), smart home devices (televisions, refrigerators, air conditioners, washing machines, sweeping and mopping robots, etc.), smart wearable devices (bracelets, watches, etc.), etc.

[0033] Server 102 can be an independent physical server or a server cluster or distributed system composed of multiple physical servers, wherein the server cluster or distributed system includes cloud servers for providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0034] It is understood that the terminal device 101 and the server 102 establish a communication connection through a wired or wireless network. Exemplarily, the wireless network or wired network uses standard communication technology and / or protocols. The network is usually the Internet, but it can also be any other network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network, etc.

[0035] In one embodiment of the present application, the model data processing method can be executed by the server 102. Specifically, if the server 102 receives an evaluation request for a specified model, it obtains the task category information of the task to be performed by the specified model, and the scenario information of the scenario in which the specified model is applied. Then, based on the task category information and the scenario information, it obtains the test data and evaluation parameters corresponding to the specified model. Then, based on the test data, it tests the specified model and generates a model performance evaluation result of the specified model under the evaluation parameters.

[0036] In other embodiments of the present application, the model data processing method can also be executed by the terminal device 101 alone, or jointly executed by the terminal device 101 and the server 102. In actual applications, the execution subject of the model data processing method can be flexibly adjusted according to the specific application scenario.

[0037] It should be made clear that Figure 1 The number of terminal devices 101 and servers 102 in the figure is only for illustration, and any number of terminal devices 101 and servers 102 may be provided according to actual needs.

[0038] In the embodiment of the present application, by using the task category information and scenario information corresponding to the specified model, the test data and evaluation parameters corresponding to the specified model are obtained to perform performance evaluation on the specified model. The actual task category and application scenario of the specified model are taken into consideration, the accuracy of the model performance evaluation is improved, and it is applicable to various models, which improves the efficiency and flexibility of the model performance evaluation, and avoids the problem of low evaluation accuracy caused by the large limitations of the model performance evaluation tool, thereby further improving the accuracy of the model performance evaluation; that is, the reliability of the model performance evaluation in the embodiment of the present application is high, thereby making the reliability of the model data processing high.

[0039] The following is a detailed description of various implementation details of the technical solution of the embodiment of the present application:

[0040] See also Figure 2 , Figure 2FIG. 1 is a flow chart of a model data processing method according to an embodiment of the present application. Figure 2 As shown, the model data processing method at least includes S201 to S203, which are described in detail as follows:

[0041] S201, if an evaluation request for a specified model is received, then task category information of a task to be performed by the specified model and scenario information of a scenario in which the specified model is applied are obtained.

[0042] In the embodiment of the present application, an evaluation request refers to a request initiated by a user for performing performance evaluation on a model when there is an evaluation requirement. Accordingly, a specified model refers to a model that requires performance evaluation.

[0043] In the embodiment of the present application, the task category information refers to the category of the task that the specified model needs to perform, such as a generation category (that is, the task to be performed is a generation task), a classification category (that is, the task to be performed is a classification task), a regression category (that is, the task to be performed is a regression task), etc.

[0044] In the embodiment of the present application, the scenario information refers to the identification of the scenario to which the specified model is applied, such as the scenario name, scenario identification number, etc.; wherein the scenarios include but are not limited to finance, medical care, education, transportation, entertainment, etc.

[0045] In an embodiment of the present application, if an evaluation request for a specified model is received, it indicates that a performance evaluation of the specified model is required. At this time, task category information and scenario information corresponding to the specified model can be obtained to perform performance evaluation on the model.

[0046] In one embodiment of the present application, a model data processing system corresponding to the model data processing method can be deployed on a machine, and multiple models can also be deployed on the machine. Optionally, the multiple models are multiple large language models (LLM); wherein: selection controls corresponding to the multiple models are displayed on the model performance evaluation interface; in response to the triggering operation of the selection control, the model corresponding to the triggered selection control is determined as the specified model, and an evaluation request for the specified model is generated.

[0047] That is, in an optional embodiment, selection controls corresponding to multiple models are displayed on the model performance evaluation interface, so that the user can trigger the corresponding selection control according to actual needs, thereby generating an evaluation request for the specified model.

[0048] Among them, in the optional embodiment, multiple models can also be deployed on other machines. In actual application, the specific deployment situation of the model shall prevail. At the same time, the display method of the model performance evaluation interface can be flexibly adjusted, as long as the display method of the specified model can be determined through interaction with the user.

[0049] In this way, by implementing the optional embodiment, an evaluation request for a specified model can be generated simply and accurately, thereby triggering a performance evaluation of the specified model.

[0050] S202, based on the task category information and the scenario information, obtain the test data and evaluation parameters corresponding to the specified model.

[0051] In the embodiment of the present application, the task category information and scenario information corresponding to the specified model are obtained, and then the task category information and scenario information corresponding to the specified model can be used to obtain the test data and evaluation parameters corresponding to the specified model.

[0052] In the embodiment of the present application, test data refers to data that is input into a specified model to test the specified model, which is also called a test sample. Test samples are usually used in the testing phase of the model, and are distinguished from training samples. Training samples are usually used in the training phase of the model. Optionally, there are some similarities between the test samples and the training samples.

[0053] The evaluation parameters in the embodiment of the present application refer to the standards, setting conditions, etc. for evaluating the performance of a specified model, which include evaluation dimensions, wherein the evaluation dimensions correspond to evaluation indicators.

[0054] In one embodiment of the present application, the process of obtaining test data and evaluation parameters corresponding to the specified model based on task category information and scenario information in S202 may include: selecting test data that matches both the task category information and the scenario information from the test data set, and using the selected test data as the test data for the specified model; and selecting evaluation dimensions that match both the task category information and the scenario information from the evaluation dimension set, and using the evaluation indicators corresponding to the selected evaluation dimensions as the evaluation parameters of the specified model.

[0055] That is, in an optional embodiment, a test data set is pre-set, wherein test data that matches both the task category information and the scenario information is selected from the test data set, and the test data selected at this time is the test data of the specified model; at the same time, an evaluation dimension set is pre-set, wherein evaluation dimensions that match both the task category information and the scenario information are selected from the evaluation dimension set, and the evaluation indicators corresponding to the selected evaluation dimensions are the evaluation parameters of the specified model.

[0056] In this way, by implementing the optional embodiment, the test data and evaluation parameters corresponding to the specified model can be obtained simply and accurately, providing strong support for the performance evaluation of the specified model.

[0057] In one embodiment of the present application, before the process of selecting test data that matches both task category information and scenario information from a test data set, it may also include: obtaining task category information of tasks that each evaluation model needs to perform, and scenario information of scenarios to which each evaluation model is applied; based on the task category information and scenario information of each evaluation model, obtaining sample data corresponding to each evaluation model; performing data preparation processing on the sample data corresponding to each evaluation model to obtain test data corresponding to each evaluation model; wherein the data preparation processing includes at least one of data cleaning, data preprocessing, and label generation; associating the task category information, scenario information, and test data of each evaluation model to generate a test data set.

[0058] That is, the process of generating the test data set in the optional embodiment is to first obtain the task category information of the tasks that each evaluation model needs to perform, and the scenario information of the scenarios in which each evaluation model is applied, and then use the task category information and scenario information of each evaluation model to obtain the sample data corresponding to each evaluation model, and then perform data preparation processing on the sample data corresponding to each evaluation model to obtain the test data corresponding to each evaluation model, and finally associate the task category information, scenario information, and test data of each evaluation model to generate a test data set.

[0059] Among them, in the optional embodiment, data preparation processing refers to processing used to make sample data meet the test requirements, including but not limited to at least one of data cleaning, data preprocessing, and label generation. Specifically, data cleaning refers to screening-related processing, such as removing duplicate data, removing irrelevant data, removing low-quality data, etc.; data preprocessing refers to data enhancement (such as synonym replacement, text expansion, etc.)-related processing; label generation refers to annotation-related processing, wherein format unification / standardization processing can also be performed before label generation. For the sake of distinction, in the optional embodiment, the sample data after data preparation processing is referred to as test data.

[0060] Please see Table 1 for an example test dataset.

[0061]

[0062] Table 1

[0063] As shown in Table 1, for evaluation model 1, its corresponding task category A, scenario 1, and test data A_1 are associated, its corresponding task category A, scenario 2, and test data A_2 are associated, and its corresponding task category A, scenario 3, and test data A_3 are associated. The same is true for evaluation models 2 and 3, and no further details are given here. It can be understood that for different evaluation models, the task categories and scenarios can be the same or different, and in actual applications, they can be flexibly adjusted according to specific application scenarios.

[0064] In this way, by implementing the optional embodiment, a test data set can be generated simply and accurately, so that test data corresponding to a specified model can be obtained simply and accurately.

[0065] In one embodiment of the present application, before the process of selecting test data that matches both task category information and scenario information from a test data set, it may also include: obtaining task category information of tasks that each evaluation model needs to perform, and scenario information of scenarios to which each evaluation model is applied; based on the task category information and scenario information of each evaluation model, determining the evaluation dimensions of each evaluation model, and determining evaluation indicators based on the determined evaluation dimensions; wherein the determined evaluation dimensions include at least one of a task execution quality dimension, a generalization ability dimension, a sample bias dimension, and a resource consumption dimension; associating the task category information, scenario information, evaluation dimensions, and evaluation indicators of each evaluation model to generate an evaluation dimension set.

[0066] That is, the process of deriving the evaluation dimension set in the optional embodiment is to first obtain the task category information of the tasks that each evaluation model needs to perform, as well as the scenario information of the scenarios in which each evaluation model is applied, and then use the task category information and scenario information of each evaluation model to determine the evaluation dimensions of each evaluation model, and then determine the evaluation indicators based on the determined evaluation dimensions, and finally associate the task category information, scenario information, evaluation dimensions, and evaluation indicators of each evaluation model to generate an evaluation dimension set.

[0067] Among them, the evaluation dimension in the optional embodiment includes but is not limited to at least one of the task execution quality dimension, the generalization ability dimension, the sample bias dimension, and the resource consumption dimension. Specifically, the task execution quality dimension is used to evaluate the task execution quality of the model executing the task, such as task execution accuracy, task execution efficiency, etc.; the generalization ability dimension refers to the performance of the model when processing new data; the sample bias dimension refers to the bias of the model to the sample data, so as to distribute the sample data more evenly; the resource consumption dimension refers to the resource consumption of the model when executing the task, such as memory consumption, computing resource consumption, etc.

[0068] It can be understood that the evaluation dimension includes evaluation indicators, where the evaluation dimension corresponds to the performance evaluation of the model from a larger direction, and the evaluation indicator corresponds to the performance evaluation of the model from a smaller direction. An evaluation dimension includes at least one evaluation indicator, and the evaluation indicators corresponding to different evaluation dimensions can be the same or different. In practical applications, they can be flexibly adjusted according to specific application scenarios.

[0069] See Table 2 for an example set of evaluation dimensions.

[0070]

[0071]

[0072] Table 2

[0073] As shown in Table 1, for evaluation model 1, its corresponding task category A, scenario 1, evaluation dimension 1 (corresponding to evaluation index 1_1 and evaluation index 1_2), and evaluation dimension 2 (corresponding to evaluation index 2_1) are associated; its corresponding task category A, scenario 2, evaluation dimension 2 (corresponding to evaluation index 2_1), evaluation dimension 3 (evaluation index 3_1, evaluation index 3_2), and evaluation dimension 4 (corresponding to evaluation index 4_1) are associated; its corresponding task category A, scenario 3, evaluation dimension 5 (corresponding to evaluation index 5_1), and evaluation dimension 6 (corresponding to evaluation index 6_1) are associated; and the same applies to evaluation models 2 and 3, which will not be repeated here.

[0074] In this way, by implementing the optional embodiment, an evaluation dimension set can be easily and accurately generated, so that evaluation parameters corresponding to a specified model can be easily and accurately obtained.

[0075] It can be understood that, for the same evaluation model, the more evaluation dimensions it corresponds to, the more comprehensive its performance evaluation is, and accordingly, the higher the accuracy of the model performance evaluation is; for example, for evaluation model 2, when it is applied to scenario 2, it corresponds to 4 evaluation dimensions. Assuming that the 4 evaluation dimensions are task execution quality dimension, generalization ability dimension, sample bias dimension, and resource consumption dimension, then its performance evaluation is a comprehensive performance evaluation from the dimensions of task execution quality, generalization ability, sample bias, and resource consumption at the same time.

[0076] It should be noted that the specific storage form of the test data set and the evaluation dimension set can be flexibly adjusted according to the actual application scenario. For example, the test data set shown in Table 1 above can be combined with the evaluation dimension set shown in Table 2 above into the same table.

[0077] S203, testing the specified model based on the test data, and generating a model performance evaluation result of the specified model under the evaluation parameters.

[0078] In the embodiment of the present application, test data and evaluation parameters corresponding to the specified model are obtained, and then the test data corresponding to the specified model can be used to test the specified model to generate a model performance evaluation result of the specified model under the evaluation parameters corresponding to the specified model.

[0079] In one embodiment of the present application, the evaluation parameters include evaluation indicators corresponding to the evaluation dimensions; accordingly, the process of generating a model performance evaluation result of the specified model under the evaluation parameters based on the test data for the specified model in S203 may include: inputting the test data into the specified model to perform the task and obtaining the task execution result; calculating the indicator value corresponding to the evaluation indicator based on the task execution result; and generating the model performance evaluation result of the specified model under the evaluation parameters based on the indicator value.

[0080] That is, in an optional embodiment, the test data is first input into the specified model to execute the task and obtain the task execution result, and then the task execution result is used to calculate the indicator data corresponding to the evaluation indicator, and finally the indicator value is used to generate the model performance evaluation result of the specified model under the evaluation parameters.

[0081] In this way, by implementing the optional embodiment, the model performance evaluation result can be obtained simply and accurately, thereby achieving performance evaluation of the specified model.

[0082] In one embodiment of the present application, the process of calculating an indicator value corresponding to an evaluation indicator based on a task execution result may include: if the evaluation dimension includes at least one of a task execution quality dimension and a generalization ability dimension, then obtaining a loss value between the task execution result and the expected result of the task, and calculating the indicator value corresponding to the evaluation indicator based on the loss value; if the evaluation dimension includes at least one of a sample bias dimension and a resource consumption dimension, then calculating the indicator value corresponding to the evaluation indicator based on at least one of the attribute information of the test data and the loss value.

[0083] Among them, in an optional embodiment, if the evaluation dimension includes a task execution quality dimension, the loss value between the task execution result and the expected result of the task is obtained, and the loss value is used to calculate the indicator value corresponding to the evaluation indicator of the task execution quality dimension; accordingly, if the evaluation dimension includes a generalization ability dimension, the loss value between the task execution result and the expected result of the task is obtained, and the loss value is used to calculate the indicator value corresponding to the evaluation indicator of the generalization ability dimension.

[0084] Optionally, the evaluation index of the generalization ability dimension includes at least one of a cross-validation index and a generalization error upper bound index.

[0085] As an example, it can be calculated by the following formula:

[0086]

[0087] In this formula, Validation Error model refers to the cross-validation value, K refers to the number of split test data, and Loss i It refers to the corresponding loss value in the i-th cross validation.

[0088] As an example, it can be calculated by the following formula:

[0089] Generalization Error model =Bias 2 +Variance+Irreducible Error

[0090] In this formula, Generalization Error model It refers to the upper bound of the generalization error, Bias refers to the average value of multiple loss values ​​based on the test phase, Variance refers to the variance obtained based on multiple loss values ​​in the test phase, and Irreducible Error refers to unavoidable errors (such as noise).

[0091] Among them, in an optional embodiment, if the evaluation dimension includes a sample bias dimension, the loss value between the task execution result and the expected result of the task is obtained, and the loss value and the attribute information of the test data are used to calculate the indicator value corresponding to the evaluation indicator of the sample bias dimension; accordingly, if the evaluation dimension includes a resource consumption dimension, the loss value between the task execution result and the expected result of the task is obtained, and the loss value and the attribute information of the test data are used to calculate the indicator value corresponding to the evaluation indicator of the resource consumption dimension.

[0092] Optionally, the evaluation index of the sample bias dimension includes at least one of a sample distribution balance index and a sample fairness index.

[0093] As an example, it can be calculated by the following formula:

[0094]

[0095] In this formula, D Bias Refers to the sample distribution balance value, n i represents the number of test samples of the i-th category, N represents the number of test samples included in the test data, and n is the number of test sample categories.

[0096] As an example, it can be calculated by the following formula:

[0097]

[0098] In this formula, E Fairness refers to the sample fairness value, n is the number of test sample categories, E i It refers to the loss value corresponding to the i-th test sample, E max Refers to the maximum loss value.

[0099] Optionally, the evaluation index of the resource consumption dimension includes at least one of an inference time index, a memory consumption index, and a computing resource consumption index.

[0100] As an example, it can be calculated by the following formula:

[0101]

[0102] In this formula, T infer is the inference time value, N is the number of inferences (the same as the number of test samples included in the test data), and t i It refers to the time of the i-th reasoning.

[0103] As an example, it can be calculated by the following formula:

[0104]

[0105] In this formula, R memory refers to the memory consumption value, N refers to the number of inferences (the same as the number of test samples included in the test data), r i is the memory consumption during the i-th inference.

[0106] As an example, it can be calculated by the following formula:

[0107]

[0108] In this formula, C FLOPs refers to the value of computing resource consumption, N refers to the number of inferences (the same as the number of test samples included in the test data), FLOP i is the number of floating-point operations consumed by the i-th inference.

[0109] In one embodiment of the present application, after the process of testing the specified model based on the test data in S203 to generate the model performance evaluation results of the specified model under the evaluation parameters, it may also include: determining the potential problems of the specified model based on the model performance evaluation results; optimizing the specified model from the optimization direction corresponding to the potential problems to obtain the optimized specified model; and executing the task through the optimized specified model.

[0110] That is, in an optional embodiment, the model performance evaluation results are first used to determine the potential problems of the specified model, and then the specified model is optimized from the optimization direction corresponding to the potential problems, so as to obtain the optimized specified model, and then the task can be performed through the optimized specified model.

[0111] Among them, in an optional embodiment, the potential problems and optimization directions of the specified model can be recorded in the model performance evaluation report, so that the user can determine the actual optimization method to be adopted by viewing the model performance evaluation report, and then use the optimization method to optimize to obtain the optimized specified model.

[0112] In this way, by implementing the optional embodiment, the optimization of the specified model can be achieved simply and accurately, so that the optimized specified model can be obtained simply and accurately, so that the corresponding task can be better executed.

[0113] In the embodiment of the present application, by using the task category information and scenario information corresponding to the specified model, the test data and evaluation parameters corresponding to the specified model are obtained to perform performance evaluation on the specified model. The actual task category and application scenario of the specified model are taken into consideration, the accuracy of the model performance evaluation is improved, and it is applicable to various models, which improves the efficiency and flexibility of the model performance evaluation, and avoids the problem of low evaluation accuracy caused by the large limitations of the model performance evaluation tool, thereby further improving the accuracy of the model performance evaluation; that is, the reliability of the model performance evaluation in the embodiment of the present application is high, thereby making the reliability of the model data processing high.

[0114] The specific scenarios of the embodiments of the present application are described in detail below:

[0115] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram showing a model data processing system according to an embodiment of the present application. Figure 3 As shown in the figure, the model data processing system mainly includes a model performance evaluation framework module, a data acquisition and processing module, an evaluation algorithm module, and a performance analysis and diagnosis module, among which:

[0116] (1) Model performance evaluation framework module, which is mainly used to design a multi-dimensional evaluation framework and to conduct a comprehensive evaluation on the large language model during model performance evaluation, so as to obtain the model performance evaluation results; it includes but is not limited to the model generalization ability evaluation submodule, the test sample bias detection submodule, and the algorithm efficiency evaluation submodule, among which:

[0117] The model generalization ability evaluation submodule is mainly used to evaluate the performance of the model when processing new data (i.e., generalization ability); it mainly includes cross-validation and generalization error upper bound.

[0118] Among them, cross validation can use the formula:

[0119]

[0120] In this formula, Validation Error model refers to the cross-validation value, K refers to the number of split test data, and Loss i It refers to the corresponding loss value in the i-th cross validation.

[0121] Among them, the upper bound of the generalization error can be expressed as:

[0122] Generalization Error model =Bias 2 +Variance+Irreducible Error

[0123] In this formula, Generalization Error model It refers to the upper bound of the generalization error, Bias refers to the average value of multiple loss values ​​based on the test phase, Variance refers to the variance obtained based on multiple loss values ​​in the test phase, and Irreducible Error refers to unavoidable errors (such as noise).

[0124] The test sample bias detection submodule is mainly used to evaluate the bias of the model on the test sample; it mainly includes sample distribution balance detection and sample fairness detection.

[0125] Among them, the sample distribution balance test can use the formula:

[0126]

[0127] In this formula, D Bias Refers to the sample distribution balance value, n i represents the number of test samples of the i-th category, N represents the number of test samples included in the test data, and n is the number of test sample categories.

[0128] Among them, the sample fairness test can use the formula:

[0129]

[0130] In this formula, E Fairness refers to the sample fairness value, n is the number of test sample categories, E i It refers to the loss value corresponding to the i-th test sample, E max Refers to the maximum loss value.

[0131] The algorithm efficiency evaluation submodule is mainly used to evaluate the resource consumption of the model when executing tasks; it mainly includes inference time detection, memory consumption detection, and computing resource consumption detection.

[0132] Among them, the reasoning time detection can be carried out using the formula:

[0133]

[0134] In this formula, T infer is the inference time value, N is the number of inferences (the same as the number of test samples included in the test data), and t i It refers to the time of the i-th reasoning.

[0135] Among them, memory consumption detection can adopt the formula:

[0136]

[0137] In this formula, R memory refers to the memory consumption value, N refers to the number of inferences (the same as the number of test samples included in the test data), r i is the memory consumption during the i-th inference.

[0138] Among them, the computing resource consumption detection can adopt the formula:

[0139]

[0140] In this formula, C FLOPs refers to the value of computing resource consumption, N refers to the number of inferences (the same as the number of test samples included in the test data), FLOP i is the number of floating-point operations consumed by the i-th inference.

[0141] (2) A data acquisition and processing module, which is mainly used to obtain test data corresponding to each evaluation model by using the task category information and scenario information of each evaluation model, so as to generate the test data set in the aforementioned embodiment, and to obtain test data for a large language model during model performance evaluation; it includes but is not limited to a data filtering and screening submodule, a data labeling and standardization submodule, and a data enhancement submodule, etc., wherein:

[0142] The data filtering and screening submodule is mainly used to utilize the task category information and scenario information of each evaluation model to obtain the sample data corresponding to each evaluation model from multiple data sources (such as existing data sets, open source data sets, user interaction data, etc.), and then filter and screen the sample data, for example, by removing duplicate, irrelevant or low-quality sample data to ensure the high quality of the sample data.

[0143] The data labeling and standardization submodule is mainly used to label and standardize the sample data of each evaluation model, such as using automated labeling technology to label, and standardize the sample data to ensure that sample data from different sources can be in a unified format.

[0144] The data enhancement submodule is mainly used to perform data enhancement on the sample data of each evaluation model, such as synonym replacement, text reorganization, etc., to improve the diversity of sample data.

[0145] (3) An evaluation algorithm module, which is mainly used to determine the evaluation parameters corresponding to each evaluation model by using the task category information and scenario information of each evaluation model, so as to generate the evaluation dimension set in the aforementioned embodiment, and to obtain the evaluation parameters for the large language model during the model performance evaluation; it includes but is not limited to an adaptive algorithm selection submodule and a multi-level evaluation structure submodule, etc., wherein:

[0146] The adaptive algorithm selection submodule is mainly used to first use the task category information of each evaluation model to determine the candidate evaluation indicators corresponding to each evaluation model; illustratively, for generation tasks (such as machine translation), commonly used candidate evaluation indicators include but are not limited to bilingual evaluation understudy (BLEU), recall-oriented summary evaluation indicators (ROUGE), and translation evaluation indicators with explicit ordering (METEOR) with explicit ordering. These evaluations mainly measure the quality, fluency and matching degree of the text generated by the model with the reference answer; for classification tasks (such as text classification, sentiment analysis), commonly used candidate evaluation indicators include but are not limited to accuracy, F1 score, confusion matrix, area under the receiver operating characteristic curve (Area Under the Curve, AUC), etc. These evaluation indicators can quantify the performance of the model in terms of classification accuracy; for regression tasks (such as sentiment scoring, price prediction, etc.), commonly used candidate evaluation indicators include but are not limited to mean squared error (MSE), mean absolute error (MAE), etc. Then, the scenario information of each evaluation model is used to determine the target evaluation indicator corresponding to each evaluation model from the candidate evaluation indicators corresponding to each evaluation model. For example, in financial scenarios, more emphasis may be placed on the robustness and interpretability of the model, while in medical scenarios, more emphasis may be placed on accuracy and recall, etc. At this time, the target evaluation indicator corresponding to each evaluation model is the final evaluation indicator. It can be understood that the target evaluation indicator is the evaluation parameter during model performance evaluation, and the target evaluation indicator can belong to the task execution quality dimension in the aforementioned embodiment.

[0147] The multi-level evaluation structure submodule is mainly used to stratify the evaluation indicators (i.e., target evaluation indicators) according to different granularities; illustratively, it includes but is not limited to the basic layer, which mainly evaluates the basic performance of the model, such as accuracy and loss value, which usually correspond to the evaluation indicators in the task execution quality dimension; the middle layer mainly evaluates the performance of the model in a specific scenario, such as task-related classification accuracy, recall rate, BLEU score, etc., which usually correspond to the evaluation indicators in the task execution quality dimension; the advanced layer mainly conducts in-depth analysis of the model and identifies potential problems of the model, such as sample bias, interpretability, diversity, generalization ability, security, ethics, etc., which usually correspond to the evaluation indicators in the generalization ability dimension, sample bias dimension, and resource consumption dimension in the aforementioned embodiments.

[0148] (4) Performance analysis and diagnosis module, which is mainly used to optimize the large language model by using the model performance evaluation results during model performance evaluation; it includes but is not limited to the intelligent diagnosis engine submodule and the performance bottleneck analysis engine submodule, among which:

[0149] The intelligent diagnosis engine submodule mainly performs automated diagnosis through machine learning technology, thereby helping to identify potential problems that may exist in the model during training or reasoning, such as overfitting, underfitting, sample bias, performance bottlenecks, etc. Optionally, the intelligent diagnosis engine submodule can perform diagnosis based on the latest data (such as the latest model performance evaluation results, test data, etc.) and historical data (such as historical model performance evaluation results, test data, training data, etc.), so as to obtain more accurate diagnosis results. Optionally, the intelligent diagnosis engine submodule can model the evaluation data using machine learning algorithms (such as regression analysis, cluster analysis, decision trees, etc.), extract key features that affect model performance, and then use key features for automated diagnosis; for example, using regression models to analyze the relationship between input features and model outputs, or using clustering algorithms to discover the law of model error distribution. Optionally, the intelligent diagnosis engine submodule can continuously obtain new model performance evaluation results to analyze and feedback problems that occur in the model during reasoning in real time, thereby improving the timeliness and accuracy of diagnosis.

[0150] The performance bottleneck analysis submodule is mainly used to deeply analyze the various performances of the model to identify the bottlenecks that the model may encounter in various stages such as reasoning and training. Optionally, the performance bottleneck analysis submodule helps developers identify performance bottlenecks and provides specific directions for optimization through comparative analysis, distributed reasoning efficiency, data dependency, model structure and other multi-dimensional evaluation results; illustratively, the multi-dimensional evaluation comparison includes: comparing the performance data of the model with historical data, benchmark models, and advanced models in the same industry to find the relative bottlenecks of the model, such as comparing the differences in the model in multiple dimensions such as reasoning time, accuracy, and resource consumption, revealing its potential bottlenecks; illustratively, the computing resource utilization analysis includes: evaluating the resource utilization of the model on different hardware platforms (such as central processing unit (CPU), graphics processing unit (GPU), tensor processing unit (TPU), etc.), and diagnosing whether there are problems such as resource waste or hardware adaptation by analyzing the reasoning speed and memory consumption of the model under different hardware configurations.

[0151] based on Figure 3 For the system shown, see Figure 4 , Figure 4FIG. 1 is a flow chart of a model data processing method according to an embodiment of the present application. Figure 4 As shown, the model data processing method at least includes S401 to S404, which are described in detail as follows:

[0152] S401, if an evaluation request for a specified model is received, then task category information of a task to be performed by the specified model and scenario information of a scenario in which the specified model is applied are obtained.

[0153] S402, selecting test data that matches both the task category information and the scenario information from the test data set, and using the selected test data as the test data of the specified model, and selecting evaluation dimensions that match both the task category information and the scenario information from the evaluation dimension set, and using the evaluation indicators corresponding to the selected evaluation dimensions as the evaluation parameters of the specified model.

[0154] S403, input the test data into the specified model to execute the task, obtain the task execution result, and calculate the indicator value corresponding to the evaluation indicator based on the task execution result, and generate the model performance evaluation result of the specified model under the evaluation parameters based on the indicator value.

[0155] S404, determining potential problems of the specified model based on the model performance evaluation results, optimizing the specified model from the optimization direction corresponding to the potential problems, obtaining the optimized specified model, and executing the task through the optimized specified model.

[0156] It should be noted that Figure 4 For the detailed description of S401 to S404 shown, please refer to the above-mentioned embodiment, which will not be repeated here.

[0157] For ease of understanding, Figure 5 As shown, in response to a received evaluation task for a specified model (i.e., an evaluation request), the model task is automatically identified, and the identification result may be any one of a generation task, a classification task, and a regression task. At the same time, the scenario to which the specified model is applied is analyzed, and a scenario-driven evaluation is performed based on the scenario obtained from the analysis, and then test data and evaluation dimensions (including evaluation indicators) that match both the task category and the scenario are obtained, and then the test data and evaluation dimensions are used to perform the evaluation task (i.e., perform performance evaluation on the specified model) to obtain an evaluation result (i.e., a model performance evaluation result).

[0158] To facilitate understanding, several example tasks are introduced.

[0159] For text generation tasks, for example, if the input is a context, the model needs to generate coherent natural language text based on the context. It mainly includes: Step 1, collect task data sets for various text generation tasks (such as news generation, novel writing), which need to contain different types of contexts and task requirements, and annotate the text; Step 2, combine the task data set and the scenario to which the model is applied, and evaluate the model performance from multi-dimensional evaluation indicators (such as text generation accuracy, coherence score, fluency, etc.) to obtain the model performance evaluation results; Step 3, generate a detailed evaluation report based on the model performance evaluation results (for example, it can include task completion, language quality of generated text, model performance, etc.).

[0160] For question-answering tasks, for example, inputting a corresponding question, the model needs to generate a response content based on the corresponding question. It mainly includes: Step 1, collecting question-answering datasets (such as SQuAD, TriviaQA, etc.), which need to include different types of question-answering pairs (such as factual, reasoning question-answering pairs, etc.); Step 2, combining the question-answering dataset and the scenario to which the model is applied, and evaluating the model performance from multi-dimensional evaluation indicators (such as the accuracy of each question, the fluency of the answer, the grammatical structure, etc.) to obtain the model performance evaluation results; Step 3, generating a detailed evaluation report based on the model performance evaluation results (for example, it can include the accuracy of the answer, the recall rate, etc.).

[0161] For sentiment analysis tasks, for example, after inputting a piece of text, the model needs to determine the sentiment tendency of the text (such as positive, negative, neutral, etc.). It mainly includes: Step 1, collect public sentiment analysis data sets (such as IMDb, Amazon product reviews, etc.), and annotate the sentiment labels of each review; Step 2, combine the sentiment analysis data set and the scenario to which the model is applied, and evaluate the performance of the model from multi-dimensional evaluation indicators (such as accuracy, recall rate, F1 value, etc.) to obtain the model performance evaluation results; Step 3, generate a detailed evaluation report based on the model performance evaluation results (for example, it can include improvement suggestions, etc.).

[0162] The technical solution provided by the embodiments of the present application has at least the following beneficial effects:

[0163] The performance of the specified model is evaluated with the help of the task category information and scenario information corresponding to the specified model, which takes into account the actual task category and application scenario of the specified model, improves the accuracy of the model performance evaluation, and is applicable to various models, which improves the efficiency and flexibility of the model performance evaluation, and avoids the problem of low evaluation accuracy caused by the large limitations of the model performance evaluation tool, thereby further improving the accuracy of the model performance evaluation.

[0164] The performance of the specified model is evaluated with the help of evaluation indicators in multiple dimensions, making the model performance evaluation more comprehensive, thereby further improving the accuracy of the model performance evaluation.

[0165] Generating a detailed evaluation report based on the model performance evaluation results can help developers find the shortcomings of the model, thereby promoting model optimization and iteration so that the corresponding tasks can be better executed.

[0166] Figure 6 FIG. 1 is a block diagram of a model data processing device according to an embodiment of the present application. Figure 6 As shown, the device comprises:

[0167] The first acquisition module 601 is configured to, if an evaluation request for a specified model is received, acquire task category information of a task to be performed by the specified model and scenario information of a scenario in which the specified model is applied;

[0168] A second acquisition module 602 is configured to acquire test data and evaluation parameters corresponding to the specified model based on the task category information and the scenario information;

[0169] The generation module 603 is configured to generate a model performance evaluation result of the specified model under the evaluation parameters based on the test data on the specified model.

[0170] In one embodiment of the present application, based on the aforementioned scheme, the second acquisition module 602 is specifically configured as follows: selecting test data that matches both the task category information and the scenario information from the test data set, and using the selected test data as the test data for the specified model; and selecting an evaluation dimension that matches both the task category information and the scenario information from the evaluation dimension set, and using the evaluation indicator corresponding to the selected evaluation dimension as the evaluation parameter of the specified model.

[0171] In one embodiment of the present application, based on the aforementioned scheme, the second acquisition module 602 is further configured to: obtain task category information of tasks to be performed by each evaluation model, and scenario information of the scenarios in which each evaluation model is applied; based on the task category information and scenario information of each evaluation model, obtain sample data corresponding to each evaluation model; perform data preparation processing on the sample data corresponding to each evaluation model to obtain test data corresponding to each evaluation model; wherein the data preparation processing includes at least one of data cleaning, data preprocessing, and label generation; associate the task category information, scenario information, and test data of each evaluation model to generate the test data set.

[0172] In one embodiment of the present application, based on the aforementioned scheme, the second acquisition module 602 is further configured to: obtain task category information of tasks to be performed by each evaluation model, and scenario information of the scenarios in which each evaluation model is applied; determine the evaluation dimensions of each evaluation model based on the task category information and scenario information of each evaluation model, and determine the evaluation indicators based on the determined evaluation dimensions; wherein the determined evaluation dimensions include at least one of the task execution quality dimension, the generalization ability dimension, the sample bias dimension, and the resource consumption dimension; associate the task category information, scenario information, evaluation dimensions, and evaluation indicators of each evaluation model to generate the evaluation dimension set.

[0173] In one embodiment of the present application, based on the aforementioned scheme, the evaluation parameters include evaluation indicators corresponding to the evaluation dimensions; the generation module 603 is specifically configured as follows: inputting the test data into the specified model to execute the task and obtain the task execution result; calculating the indicator value corresponding to the evaluation indicator based on the task execution result; and generating the model performance evaluation result of the specified model under the evaluation parameters based on the indicator value.

[0174] In one embodiment of the present application, based on the aforementioned scheme, the generation module 603 is further specifically configured as follows: if the evaluation dimension includes at least one of the task execution quality dimension and the generalization ability dimension, then the loss value between the task execution result and the expected result of the task is obtained, and the indicator value corresponding to the evaluation indicator is calculated based on the loss value; if the evaluation dimension includes at least one of the sample bias dimension and the resource consumption dimension, then the indicator value corresponding to the evaluation indicator is calculated based on the attribute information of the test data and at least one of the loss values.

[0175] In one embodiment of the present application, based on the aforementioned scheme, the device also includes an optimization module, configured to: determine the potential problems of the specified model based on the model performance evaluation results; optimize the specified model from the optimization direction corresponding to the potential problems to obtain the optimized specified model; and perform the task through the optimized specified model.

[0176] It should be noted that the apparatus provided in the aforementioned embodiment and the method provided in the aforementioned embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment.

[0177] An embodiment of the present application also provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein when the one or more computer programs are executed by the one or more processors, the electronic device implements the model data processing method as described above.

[0178] Figure 7 It is an electronic device suitable for implementing the embodiments of the present application (for example Figure 1 A schematic diagram of the structure of a computer system of a terminal device or server as shown.

[0179] It should be noted that Figure 7 The computer system 700 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0180] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage part 708 to a random access memory (RAM) 703, such as executing the method in the above embodiment. In RAM 703, various computer programs and data required for system operation are also stored. CPU 701, ROM 702 and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0181] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 708 as needed.

[0182] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes computer instructions for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 709, and / or installed from a removable medium 711. When the computer program is executed by a central processing unit (CPU) 701, various functions defined in the system of the present application are executed.

[0183] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable media can be, for example, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium can be any tangible medium containing or storing a computer program, which can be used by an instruction execution system, a device or a device or used in combination with it. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit computer programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0184] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0185] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0186] Another aspect of the present application also provides a computer readable medium having a computer program stored thereon, which implements the above model data processing method when executed by a processor. The computer readable medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0187] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable medium. A processor of an electronic device reads the computer instruction from the computer-readable medium, and the processor executes the computer instruction, so that the electronic device executes the model data processing method provided in each of the above embodiments.

[0188] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A model data processing method, characterized in that: include: If an evaluation request for a specified model is received, task category information of the task to be performed by the specified model and scenario information of the scenario in which the specified model is applied are obtained; Based on the task category information and the scenario information, obtaining test data and evaluation parameters corresponding to the specified model; Based on the test of the specified model with the test data, a model performance evaluation result of the specified model under the evaluation parameters is generated.

2. The method according to claim 1, characterized in that The acquiring, based on the task category information and the scenario information, test data and evaluation parameters corresponding to the specified model includes: Selecting test data that matches both the task category information and the scene information from the test data set, and using the selected test data as test data for the specified model; and An evaluation dimension that matches both the task category information and the scenario information is selected from the evaluation dimension set, and an evaluation indicator corresponding to the selected evaluation dimension is used as an evaluation parameter of the specified model.

3. The method according to claim 2, characterized in that Before selecting the test data matching both the task category information and the scene information from the test data set, the method further includes: Obtaining task category information of tasks to be performed by each evaluation model and scenario information of scenarios to which each evaluation model is applied; Based on the task category information and scenario information of each evaluation model, obtaining sample data corresponding to each evaluation model; Performing data preparation processing on the sample data corresponding to each evaluation model to obtain test data corresponding to each evaluation model; wherein the data preparation processing includes at least one of data cleaning, data preprocessing, and label generation; The task category information, scenario information, and test data of each evaluation model are associated to generate the test data set.

4. The method according to claim 2, characterized in that: Before selecting the evaluation dimension that matches both the task category information and the scenario information from the evaluation dimension set, the method further includes: Obtaining task category information of tasks to be performed by each evaluation model and scenario information of scenarios to which each evaluation model is applied; Based on the task category information and scenario information of each evaluation model, determining the evaluation dimension of each evaluation model, and determining the evaluation index based on the determined evaluation dimension; wherein the determined evaluation dimension includes at least one of a task execution quality dimension, a generalization ability dimension, a sample bias dimension, and a resource consumption dimension; The task category information, scenario information, evaluation dimensions, and evaluation indicators of each evaluation model are associated to generate the evaluation dimension set.

5. The method according to claim 1, characterized in that The evaluation parameters include evaluation indicators corresponding to the evaluation dimensions; the test of the specified model based on the test data to generate a model performance evaluation result of the specified model under the evaluation parameters includes: Inputting the test data into the specified model to execute the task and obtain a task execution result; Calculate the indicator value corresponding to the evaluation indicator based on the task execution result; A model performance evaluation result of the specified model under the evaluation parameters is generated based on the indicator value.

6. The method according to claim 5, characterized in that The step of calculating the indicator value corresponding to the evaluation indicator based on the task execution result includes: If the evaluation dimension includes at least one of a task execution quality dimension and a generalization ability dimension, then obtaining a loss value between the task execution result and the task expected result, and calculating an indicator value corresponding to the evaluation indicator based on the loss value; If the evaluation dimension includes at least one of a sample bias dimension and a resource consumption dimension, then based on the attribute information of the test data and at least one of the loss values, an indicator value corresponding to the evaluation indicator is calculated.

7. The method according to any one of claims 1 to 6, characterized in that After the test of the specified model based on the test data to generate a model performance evaluation result of the specified model under the evaluation parameters, the method further includes: determining potential problems of the specified model based on the model performance evaluation results; Optimizing the specified model from the optimization direction corresponding to the potential problem to obtain an optimized specified model; The task is performed by using the optimized specified model.

8. A model data processing device, characterized in that: include: A first acquisition module is configured to, if an evaluation request for a specified model is received, acquire task category information of a task to be performed by the specified model and scenario information of a scenario in which the specified model is applied; A second acquisition module is configured to acquire test data and evaluation parameters corresponding to the specified model based on the task category information and the scenario information; A generation module is configured to generate a model performance evaluation result of the specified model under the evaluation parameters based on the test data for the specified model.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more computer programs, which, when executed by the electronic device, enables the electronic device to implement the model data processing method as described in any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model data processing method according to any one of claims 1 to 7 is implemented.

Images