A model generalization evaluation method and device, a terminal and a storage medium

Abstract

The application discloses a kind of to be evaluated model generalization evaluation method, device, terminal and storage medium, the method is by obtaining the target application scene corresponding to the pre-trained to be evaluated model;Actual data set is obtained, and test set corresponding to to be evaluated model is constructed according to target application scene and actual data set, wherein the training set of to be evaluated model comes from actual data set;According to test set, to be evaluated model is tested, and the model generalization performance of to be evaluated model under target application scene is obtained.The application constructs test set according to the application scene of to be evaluated model, solves the problem that the test set obtained by simply splitting actual data set in the prior art is difficult to match various application scenes of model, so that the model generalization obtained by evaluating the model based on this test set cannot well reflect the actual running condition after online deployment.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more particularly to a method, apparatus, terminal, and storage medium for evaluating the generalization ability of a model to be evaluated. Background Technology

[0002] During the model design and development phase, engineers design the model structure and select appropriate optimization methods based on the context of the actual problem and the collected dataset to obtain a high-performance intelligent model on the test set. Current methods for generating test sets simply divide the collected dataset into two parts: a training set and a test set. This approach leads to the following problems with the test set:

[0003] 1. The sampling of the test set is usually biased;

[0004] 2. Data changes dynamically over time and location, and the currently collected data may not reflect the trend of data change. For example, the data structure collected in the process of building a face recognition model contains a large number of young people's faces (e.g., 80% young people and 20% elderly people), but the model may have different performance on different (young people: elderly people) datasets.

[0005] In short, test sets obtained by simply splitting the actual dataset are difficult to match the various application scenarios of the model. As a result, the generalization of the model evaluated based on such test sets cannot well reflect its actual operation after deployment.

[0006] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a method, device, terminal and storage medium for evaluating the generalization of a model, in order to address the above-mentioned defects of the prior art. The invention aims to solve the problem that the test set obtained by simply splitting the actual dataset in the prior art is difficult to match the various application scenarios of the model, so the generalization of the model evaluated based on such a test set cannot well reflect its actual operation after online deployment.

[0008] The technical solution adopted by this invention to solve the problem is as follows:

[0009] In a first aspect, embodiments of the present invention provide a method for evaluating the generalization ability of a model to be evaluated, wherein the method includes:

[0010] Obtain the target application scenario corresponding to the pre-trained model to be evaluated;

[0011] Obtain the actual dataset, and construct the test set corresponding to the model to be evaluated based on the target application scenario and the actual dataset, wherein the training set of the model to be evaluated comes from the actual dataset;

[0012] The model to be evaluated is tested according to the test set to obtain the model generalization performance of the model to be evaluated in the target application scenario.

[0013] In one implementation method, constructing a test set corresponding to the model to be evaluated based on the target application scenario and the actual dataset includes:

[0014] Obtain a preset attribute library, wherein the attribute library includes several scene features;

[0015] Based on the target application scenario, obtain several target scenario features from the attribute library;

[0016] The test set is obtained by extracting data from the actual dataset based on the target application scenario and the characteristics of each target scenario.

[0017] In one implementation method, each of the scenario features is pre-classified based on application scenario category, attribute category, and feature category. The attribute library includes several application scenario sets, each application scenario set includes several attribute sets, and each attribute set includes several scenario features.

[0018] In one implementation method, obtaining several target scenario features from the attribute library based on the target application scenario includes:

[0019] The target application scenario set is determined from the attribute library based on the target application scenario;

[0020] Determine the target attribute set from the target application scenario set based on the target application scenario;

[0021] Based on the target application scenario, several target scenario features are determined from the target attribute set.

[0022] In one implementation method, the step of extracting data from the actual dataset based on the target application scenario and the characteristics of each target scenario to obtain the test set includes:

[0023] Based on the target application scenario, the distribution expectation corresponding to each of the target scenario features is determined, wherein the distribution expectation is a numerical distribution expectation or a category distribution expectation;

[0024] Based on the expected distributions, a joint distribution function is determined, wherein the joint distribution function is used to extract data that conforms to the expected distributions.

[0025] The test set is obtained by extracting data from the actual dataset based on the joint distribution function.

[0026] In one implementation, the method further includes:

[0027] Obtain the generalization performance of the model to be evaluated in several application scenarios;

[0028] Based on the generalization performance of each model, the overall model generalization performance corresponding to the model to be evaluated is determined.

[0029] In one implementation method, determining the comprehensive model generalization performance corresponding to the model to be evaluated based on the generalization performance of each of the models includes:

[0030] Obtain the scenario weights corresponding to each of the aforementioned application scenarios;

[0031] The generalization performance of the comprehensive model is determined based on the weighted average of the scenario weights and the generalization performance of each model.

[0032] Secondly, embodiments of the present invention also provide a generalization evaluation device for a model to be evaluated based on an attribute library, wherein the device includes:

[0033] The acquisition module is used to acquire the target application scenario corresponding to the pre-trained model to be evaluated;

[0034] A construction module is used to obtain the actual dataset and construct the test set corresponding to the model to be evaluated based on the target application scenario and the actual dataset, wherein the training set of the model to be evaluated comes from the actual dataset;

[0035] The evaluation module is used to test the model to be evaluated based on the test set to obtain the model generalization performance of the model to be evaluated in the target application scenario.

[0036] Thirdly, embodiments of the present invention also provide a terminal, the terminal including a memory and one or more processors; the memory stores one or more programs; the programs include instructions for executing the generalization evaluation method of the model to be evaluated as described above; the processor is used to execute the programs.

[0037] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a plurality of instructions, wherein the instructions are adapted to be loaded and executed by a processor to implement any of the above-described methods for evaluating the generalization of the model to be evaluated.

[0038] The beneficial effects of this invention are as follows: This invention obtains the target application scenario corresponding to the pre-trained model to be evaluated; acquires the actual dataset; constructs a test set corresponding to the model to be evaluated based on the target application scenario and the actual dataset, wherein the training set of the model to be evaluated comes from the actual dataset; and tests the model to be evaluated based on the test set to obtain the model generalization performance of the model under the target application scenario. This invention constructs a test set specifically according to the application scenario of the model to be evaluated, solving the problem in existing technologies where test sets obtained by simply splitting the actual dataset are difficult to match various application scenarios of the model, resulting in the model generalization performance evaluated based on such test sets not accurately reflecting its actual operation after deployment. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the generalization evaluation method for the model to be evaluated provided in this embodiment of the invention.

[0041] Figure 2 This is a schematic diagram of the model generalization performance evaluation process for a single target application scenario provided in an embodiment of the present invention.

[0042] Figure 3 This is a flowchart of the attribute library construction process provided in an embodiment of the present invention.

[0043] Figure 4 This is a schematic diagram of the module connections of the attribute library-based generalization evaluation device for the model to be evaluated, provided in an embodiment of the present invention.

[0044] Figure 5 This is a schematic diagram of the terminal provided in the embodiment of the present invention. Detailed Implementation

[0045] This invention discloses a method, apparatus, terminal, and storage medium for evaluating the generalization ability of a model to be evaluated. To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0046] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0047] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0048] During the model design and development phase, engineers design the model structure and select appropriate optimization methods based on the context of the actual problem and the collected dataset to obtain a high-performance intelligent model on the test set. Current methods for generating test sets simply divide the collected dataset into two parts: a training set and a test set. This approach leads to the following problems with the test set:

[0049] 1. The sampling of the test set is usually biased;

[0050] 2. Data changes dynamically over time and location, and the currently collected data may not reflect the trend of data change. For example, the data structure collected in the process of building a face recognition model contains a large number of young people's faces (e.g., 80% young people and 20% elderly people), but the model may have different performance on different (young people: elderly people) datasets.

[0051] In short, test sets obtained by simply splitting the actual dataset are difficult to match the various application scenarios of the model. As a result, the generalization of the model evaluated based on such test sets cannot well reflect its actual operation after deployment.

[0052] To address the aforementioned shortcomings of existing technologies, this invention provides a method for evaluating the generalization performance of a model to be evaluated. The method involves: acquiring a target application scenario corresponding to a pre-trained model to be evaluated; acquiring a real-world dataset; constructing a test set for the model to be evaluated based on the target application scenario and the real-world dataset, wherein the training set of the model to be evaluated is derived from the real-world dataset; and testing the model to be evaluated using the test set to obtain the model's generalization performance in the target application scenario. This invention constructs a test set specifically for the application scenario of the model to be evaluated, solving the problem in existing technologies where test sets obtained by simply splitting the real-world dataset are difficult to match with various application scenarios of the model. This results in the model's generalization performance, evaluated using such a test set, not accurately reflecting its actual operational status after deployment.

[0053] For example, when designing and deploying a face recognition model, the existing method involves dividing the already obtained real-world dataset D∈P(x,y|s=1) into a training set D. tn and test set D tt The model is trained on the training set to improve its performance, and its generalization ability is evaluated on the test set. As time / location changes, the actual dataset D∈P(x,y|s=1) will become D'∈P(x,y|s=n) (for example, D contains more young people while D' contains more middle-aged people). Therefore, the model's performance on D... tt The accuracy on D' cannot effectively represent the accuracy of the model.

[0054] In this embodiment, it is based on the training set D. tn The model is trained to obtain the face recognition model to be evaluated. Considering that the data structure will change with time and location, this embodiment determines the corresponding target application scenario based on time, location, and model task. Data is obtained from the actual dataset to generate a corresponding test set based on the target application scenario (in this embodiment, the actual dataset size is larger than that of existing methods). The model to be evaluated is then tested using the test set to obtain its generalization performance in the target application scenario.

[0055] Exemplary methods

[0056] like Figure 1 As shown, the method includes:

[0057] Step S100: Obtain the target application scenario corresponding to the pre-trained model to be evaluated.

[0058] Specifically, the model to be evaluated in this embodiment is a trained but untested model. Since the structure of data collected in different application scenarios varies, to ensure the reliability of the test results, this embodiment first needs to determine the application scenario corresponding to the model to be evaluated, i.e., obtain the target application scenario.

[0059] like Figure 1 As shown, the method further includes the following steps:

[0060] Step S200: Obtain the actual dataset and construct the test set corresponding to the model to be evaluated based on the target application scenario and the actual dataset.

[0061] Specifically, such as Figure 2 As shown, this embodiment obtains the actual dataset corresponding to the model to be evaluated, and selects corresponding data from the actual dataset according to the target application scenario to construct the test set corresponding to the target application scenario. In one implementation, the actual dataset includes existing datasets in the prior art and data collected by engineers in multiple models to be evaluated and multiple target application scenarios. This embodiment constructs the test set through application scenarios, which can avoid the problem of the test set not matching the real situation of the application scenario, resulting in the inability to accurately evaluate the generalization performance of the model.

[0062] In one implementation, constructing the test set corresponding to the model to be evaluated based on the target application scenario and the actual dataset includes:

[0063] Step S201: Obtain a preset attribute library, wherein the attribute library includes several scene features;

[0064] Step S202: Obtain several target scene features from the attribute library according to the target application scenario;

[0065] Step S203: Extract data from the actual dataset according to the target application scenario and the characteristics of each target scenario to obtain the test set.

[0066] Specifically, this embodiment pre-constructs an attribute library that stores several scene features. These scene features describe factors in the target application scenario that affect the generalization ability of the model being evaluated. For example, in the target application scenario for a face recognition model, scene features include: age, gender, hairstyle, and ambient lighting. Engineers can, based on experience, select several factors that significantly influence the model's generalization ability from the target application scenario and obtain several target scene features from the attribute library based on these factors. Since the mathematical laws governing the same scene feature differ across application scenarios, this embodiment determines the data type and volume to be extracted for each target scene feature based on the target application scenario. Then, it extracts the data corresponding to each scene feature from the actual dataset to obtain the test set.

[0067] In one implementation, the data in the actual dataset and the scene features in the attribute library can be stored using the same classification principles. This establishes a mapping relationship between the actual dataset and the corresponding data / scene features in the attribute library that belong to the same category. In other words, the attribute library and the actual dataset can be mapped through the various target scene features.

[0068] For example, the actual dataset includes data such as: Zhang San, 23, male, 1.67m, long hair; Li Si, 64, female, 1.58m, short hair, etc. When the scene features in the attribute library are classified based on age groups, the data in the actual database can also be stored according to multiple age groups such as 0-10, 11-20, 21-30, ...

[0069] In one implementation, each of the scenario features is pre-classified based on application scenario category, attribute category, and feature category. The attribute library includes several application scenario sets, each application scenario set includes several attribute sets, and each attribute set includes several scenario features.

[0070] Specifically, to facilitate searching for scene features corresponding to different application scenarios in the attribute library, this embodiment pre-classifies the scene features in the attribute library based on application scenario category, attribute category, and feature category. The data hierarchy in the attribute library is as follows: the attribute library includes multiple application scenario sets, each application scenario set includes multiple attribute sets, and each attribute set includes multiple scene features. In one implementation, each scene feature can correspond to multiple scene feature values.

[0071] For example, the target scene set corresponding to the face recognition model includes a set of human attributes and a set of environmental attributes. The human attribute set includes several scene features such as name, age, height, and hair. This embodiment classifies and stores each scene feature in an attribute library, which facilitates the subsequent extraction of the corresponding scene features from the attribute library and allows for timely identification and correction of omissions, preventing the omission of any scene features.

[0072] In one implementation, step S202 specifically includes:

[0073] Step S2021: Determine the target application scenario set from the attribute library according to the target application scenario;

[0074] Step S2022: Determine the target attribute set from the target application scenario set according to the target application scenario;

[0075] Step S2023: Determine several target scenario features from the target attribute set according to the target application scenario.

[0076] Specifically, a target application scenario can correspond to a large number of scenario features, but not every scenario feature has a significant impact on the model's generalization. Therefore, based on the model builder's understanding of the target application scenario, several scenario features that have a significant impact on the model's generalization can be identified as target scenario features. When searching for these target scenario features in the attribute library, they can be searched sequentially according to their respective categories.

[0077] like Figure 3 As shown, in this embodiment, the corresponding target application scenario set is first determined from the attribute library based on the target application scenario. If the target application scenario set does not exist in the attribute library, a target application scenario set is created in the attribute library, and attribute sets, scenario features, and scenario feature values ​​are created in the target application scenario set based on the information in the target application scenario. After the target application scenario set is determined, several target attribute sets are determined from the target application scenario set. If the target application scenario set does not contain the target attribute set, a target attribute set is created in the target application scenario set. After the target attribute set is determined, several target scenario features are determined from the target attribute set based on the target application scenario. If the target scenario feature does not exist, the target scenario feature is added to the target attribute set accordingly.

[0078] In one implementation, step S203 specifically includes:

[0079] Step S2031: Determine the distribution expectation corresponding to each of the target scenario features according to the target application scenario, wherein the distribution expectation is a numerical distribution expectation or a category distribution expectation;

[0080] Step S2032: Determine the joint distribution function according to the expected distributions, wherein the joint distribution function is used to extract data that conforms to the expected distributions.

[0081] Step S2033: Extract data from the actual dataset according to the joint distribution function to obtain the test set.

[0082] Different target application scenarios exhibit different mathematical patterns for their various characteristics. For example, in a first-tier city A, the ratio of young to elderly population is 9:1, while in a third-tier city B, the ratio is 7:3. The mathematical patterns corresponding to the age characteristic in the population attributes differ between these two application scenarios. Therefore, when extracting data from actual datasets based on the target application scenario and its characteristics, it is also necessary to determine the mathematical patterns corresponding to each target scenario characteristic.

[0083] Specifically, the expected distribution of each target scenario feature is analyzed based on the target application scenario. The scenario feature value category of the target scenario feature is obtained. When the feature value category of the target scenario feature is numerical (e.g., age: 1-100), the mathematical law corresponding to the target scenario feature is the expected distribution of the numerical value; when the feature value category of the target scenario feature is categorical, the mathematical law corresponding to the target scenario is the expected distribution of the categorical value.

[0084] Based on the expected distributions corresponding to the acquired features of each target scenario, a joint distribution function is constructed. Data is then extracted from the actual dataset using the joint distribution function to construct a test set, ensuring that the data structure of the test set is adapted to the target application scenario.

[0085] like Figure 1 As shown, the method further includes the following steps:

[0086] Step S300: Test the model to be evaluated according to the test set to obtain the model generalization performance of the model to be evaluated in the target application scenario.

[0087] In simple terms, since the test set is constructed based on the target application scenario and its data structure conforms to the target application scenario, testing the model using the test set can accurately obtain the model's generalization performance in that target application scenario, thus enabling an accurate evaluation of the model's generalization performance.

[0088] In one implementation, the method further includes:

[0089] Step H10: Obtain the generalization performance of the model to be evaluated in several application scenarios;

[0090] Step H20: Determine the comprehensive model generalization performance corresponding to the model to be evaluated based on the generalization performance of each model.

[0091] Specifically, in practical applications, the model to be evaluated can be used in different application scenarios. Since the generalization performance of the model varies across different application scenarios, this embodiment requires first calculating the generalization performance of the model to be evaluated separately for each application scenario, and then comprehensively calculating the overall generalization performance of the model to be evaluated based on the generalization performance of each application scenario. Through multi-scenario generalization performance analysis, the possibility of model degradation over time after deployment can be reduced, while also increasing the basis for negotiation between model vendors and customers.

[0092] In one implementation, step H20 specifically includes:

[0093] Step H21: Obtain the scene weights corresponding to each of the application scenarios;

[0094] Step H22: Determine the generalization performance of the comprehensive model based on the weighted average of the scenario weights and the generalization performance of each model.

[0095] Specifically, in this embodiment, the scenario weights of each application scenario can be determined based on the usage time of the model to be evaluated in different application scenarios or other factors. For example, if the ratio of the usage time of the model to be evaluated in scenario A to the usage time in scenario B is 99:1, then the scenario weight of scenario A should be greater than that of scenario B. Based on the scenario weights and the model generalization performance corresponding to each application scenario, a weighted average of the generalization performance of each model is calculated, and the overall model generalization performance is determined based on this weighted average.

[0096] Exemplary device

[0097] Based on the above embodiments, the present invention also provides a generalization evaluation device for a model to be evaluated based on an attribute library, such as... Figure 4 As shown, the device includes:

[0098] The acquisition module 01 is used to acquire the target application scenario corresponding to the pre-trained model to be evaluated;

[0099] Module 02 is used to obtain the actual dataset and construct the test set corresponding to the model to be evaluated based on the target application scenario and the actual dataset, wherein the test set of the model to be evaluated comes from the actual dataset;

[0100] Evaluation module 03 is used to test the model to be evaluated according to the test set to obtain the model generalization performance of the model to be evaluated in the target application scenario.

[0101] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 5 As shown, the terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a generalization evaluation method for the model to be evaluated. The display screen can be a liquid crystal display (LCD) or an e-ink display.

[0102] Those skilled in the art will understand that Figure 5The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0103] In one implementation, the terminal's memory stores one or more programs, and these programs are configured to be executed by one or more processors, and the programs contain instructions for performing a generalization evaluation method for the model to be evaluated.

[0104] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0105] In summary, this invention discloses a method, apparatus, terminal, and storage medium for evaluating the generalization performance of a model to be evaluated. The method involves: acquiring a target application scenario corresponding to a pre-trained model to be evaluated; acquiring an actual dataset; constructing a test set corresponding to the model to be evaluated based on the target application scenario and the actual dataset, wherein the training set of the model to be evaluated is derived from the actual dataset; and testing the model to be evaluated using the test set to obtain the model's generalization performance in the target application scenario. This invention constructs a test set specifically for the application scenario of the model to be evaluated, solving the problem in existing technologies where test sets obtained by simply splitting the actual dataset are difficult to match with various application scenarios of the model. This results in the model's generalization performance, evaluated based on such test sets, not accurately reflecting its actual operational status after deployment.

[0106] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for evaluating the generalization ability of a model to be evaluated, characterized in that, The method includes: Obtain the target application scenario corresponding to the pre-trained model to be evaluated; the target application scenario is the application scenario corresponding to the face recognition model. The process involves: acquiring a real-world dataset; constructing a test set corresponding to the model to be evaluated based on the target application scenario and the real-world dataset; wherein the training set of the model to be evaluated is derived from the real-world dataset; and including: acquiring a pre-defined attribute library, wherein the attribute library includes several scene features, each of which is pre-classified based on application scenario category, attribute category, and feature category; the attribute library includes an application scenario set corresponding to a face recognition model, which includes a human attribute set, and the scene features corresponding to the human attribute set include name, age, height, and hair; determining a target application scenario set from the attribute library based on the target application scenario; determining a target attribute set from the target application scenario set based on the target application scenario; determining several target scene features from the target attribute set based on the target application scenario; determining the expected distribution of each target scene feature based on the target application scenario, wherein the expected distribution is a numerical expected distribution or a categorical expected distribution; determining a joint distribution function based on each expected distribution, wherein the joint distribution function is used to extract data that conforms to each expected distribution; and extracting data from the real-world dataset based on the joint distribution function to obtain the test set. The model to be evaluated is tested according to the test set to obtain the model generalization performance of the model to be evaluated in the target application scenario; The method further includes: obtaining the model generalization performance of the model to be evaluated in several application scenarios; obtaining the scenario weights corresponding to each application scenario; and determining the comprehensive model generalization performance based on the weighted average of each scenario weight and each model generalization performance.

2. A generalization evaluation device for a model to be evaluated based on an attribute library, characterized in that, The device includes: The acquisition module is used to acquire the target application scenario corresponding to the pre-trained model to be evaluated; the target application scenario is the application scenario corresponding to the face recognition model. A construction module is used to acquire a real dataset and construct a test set corresponding to the model to be evaluated based on the target application scenario and the real dataset. The training set of the model to be evaluated is derived from the real dataset. The module includes: acquiring a pre-defined attribute library, which includes several scene features, each of which is pre-classified based on an application scenario category, attribute category, and feature category; the attribute library includes an application scenario set corresponding to a face recognition model, which includes a set of human attributes, and the scene features corresponding to the human attributes include name, age, height, and hair; determining a target application scenario set from the attribute library based on the target application scenario; determining a target attribute set from the target application scenario set based on the target application scenario; determining several target scene features from the target attribute set based on the target application scenario; determining the expected distribution of each target scene feature based on the target application scenario, where the expected distribution is a numerical expected distribution or a categorical expected distribution; determining a joint distribution function based on each expected distribution, where the joint distribution function is used to extract data that conforms to each expected distribution; and extracting data from the real dataset based on the joint distribution function to obtain the test set. An evaluation module is used to test the model to be evaluated based on the test set to obtain the model generalization performance of the model to be evaluated in the target application scenario. The device is further configured to: obtain the model generalization performance of the model to be evaluated in several application scenarios; obtain the scenario weights corresponding to each application scenario; and determine the comprehensive model generalization performance based on the weighted average of each scenario weight and each model generalization performance.

3. A terminal, characterized in that, The terminal includes a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the generalization evaluation method for the model to be evaluated as described in claim 1; the processors are used to execute the programs.

4. A computer-readable storage medium storing a plurality of instructions, characterized in that, The instructions are applicable to be loaded and executed by a processor to implement the steps of the generalization evaluation method for the model to be evaluated as described in claim 1.

Images