Model performance analysis method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a model performance analysis method and device, electronic equipment and storage medium, relating to the technical field of artificial intelligence. The method comprises obtaining a first data set and a second data set; performing model shift analysis and covariate shift analysis on the first data set and the second data set to obtain model shift analysis results and covariate shift analysis results; and determining performance analysis results of a target model according to the model shift analysis results and the covariate shift analysis results. Model shift analysis can determine the change in the conditional distribution between model input data and reference data, and covariate shift analysis can determine the shift change in the distribution of model input data; based on the model shift analysis results and the covariate shift analysis results, the performance analysis results can be determined, which can locate the cause of the decline in model performance from the dual dimensions of the change in the distribution of model input data and the mapping relationship between the model input data and the reference data, thereby improving the efficiency and accuracy of model optimization.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for analyzing model performance. Background Technology

[0002] With the continuous development of artificial intelligence technology, the application of AI-based models is becoming more and more widespread. During the application of these models, it is necessary to analyze their performance in order to optimize them based on the results. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for analyzing model performance to solve problems in related technologies.

[0004] The first aspect of this disclosure provides a method for analyzing model performance, the method comprising: Obtain the first dataset and the second dataset; the first dataset is the training dataset of the target model during the training phase, and the second dataset is the inference dataset of the target model during the inference phase. Model shift analysis and covariate shift analysis were performed on the first dataset and the second dataset to obtain the model shift analysis results and covariate shift analysis results. The model shift analysis results are used to characterize the difference between the conditional distribution of the first dataset and the conditional distribution of the second dataset. The conditional distribution is the mapping relationship between the model input data and the corresponding reference data. The covariate shift analysis results are used to characterize the difference between the model input data of the first dataset and the model input data of the second dataset. Based on the model offset analysis results and the covariate offset analysis results, the performance analysis results of the target model are determined.

[0005] In some embodiments, model offset analysis is performed on the first dataset and the second dataset, including any of the following: Using the model input data as a benchmark, the first similarity between the first dataset and the second dataset on the reference data is analyzed, and the model offset analysis results are determined based on the first similarity. The initial model was trained using the first dataset and the second dataset respectively, the training results were obtained, and the model offset analysis results were determined based on the training results. Using the reference data as a benchmark, we analyze the first degree of difference between the first dataset and the second dataset in terms of the model input data, and determine the model offset analysis results based on the first degree of difference.

[0006] In some embodiments, analyzing the first similarity between the first dataset and the second dataset on reference data, based on the model input data, includes: Based on the second similarity between the model input data in the first dataset and the first target model input data, a predetermined number of first model input data in the first dataset are determined; the first target model input data is at least one model input data in the second dataset. A similarity analysis is performed on the reference data corresponding to a preset number of first model input data and the reference data corresponding to the first target model input data to obtain the first similarity.

[0007] In some embodiments, determining a preset number of first model input data in the first dataset based on a second similarity between the model input data in the first dataset and the first target model input data includes: Based on the descending order of the second similarity, a preset number of first model input data are selected from the model input data in the first dataset.

[0008] In some embodiments, a similarity analysis is performed on the reference data corresponding to a preset number of first model input data and the reference data corresponding to the first target model input data to obtain a first similarity, including: In response to a preset quantity of one, a similarity analysis is performed on the reference data corresponding to the input data of the first model and the reference data corresponding to the input data of the first target model to obtain a first similarity score; or, In response to a preset number of reference data, statistical values ​​are calculated for the preset number of reference data corresponding to the first model input data to obtain statistical reference data; A similarity analysis is performed on the reference data corresponding to the statistical reference data and the input data of the first target model to obtain the first similarity.

[0009] In some embodiments, determining the model offset analysis result based on the first similarity includes: In response to a first similarity score greater than a preset similarity threshold, a model bias is determined to exist between the first dataset and the second dataset; or, In response to a first similarity less than or equal to a preset similarity threshold, it is determined that there is no model offset between the first dataset and the second dataset.

[0010] In some embodiments, determining the model offset analysis result based on the first similarity includes: Determine the input data of the second target model in the first dataset; the input data of the second target model is the model input data corresponding to the maximum value in the second similarity. Based on the third similarity between the remaining model input data in the first dataset and the input data of the second target model, a predetermined number of second model input data in the first dataset are determined. A fourth similarity score is obtained by performing a similarity analysis on the reference data corresponding to a preset number of second model input data and the reference data corresponding to the first target model input data. In response to the first similarity being less than the fourth similarity, a model shift is determined to exist between the first dataset and the second dataset.

[0011] In some embodiments, an initial model is trained using a first dataset and a second dataset respectively to obtain training results, and the model offset analysis results are determined based on the training results, including: The initial model is trained using the first subset of data in the first dataset to obtain the first target model; and the initial model is trained using the second subset of data in the second dataset to obtain the second target model. The first target model is tested using the third subset of data in the first dataset to obtain the first test result; and the second target model is tested using the fourth subset of data in the second dataset to obtain the second test result; the training result includes the first test result and the second test result. The model offset analysis results are determined based on the consistency between the first and second test results.

[0012] In some embodiments, determining the model offset analysis result based on the consistency between the first test result and the second test result includes: In response to the consistency between the first and second test results, it is determined that there is no model offset between the first and second datasets; or, In response to the lack of consistency between the first and second test results, it is determined that there is a model bias between the first and second datasets.

[0013] In some embodiments, analyzing the first difference between the first dataset and the second dataset in terms of model input data, based on reference data, includes: Determine the first reference data in the first dataset and the second reference data in the second dataset; the similarity between the first reference data and the second reference data is greater than a preset similarity threshold; The first degree of difference is determined based on the distribution of the model input data of the first reference data and the distribution of the model input data of the second reference data.

[0014] In some embodiments, determining the model offset analysis result based on the first degree of difference includes: In response to a first difference value exceeding a preset first difference threshold, a model offset is determined to exist between the first dataset and the second dataset; or, In response to a first difference degree being less than or equal to a preset first difference threshold, it is determined that there is no model offset between the first dataset and the second dataset.

[0015] In some embodiments, covariate offset analysis is performed on the first dataset and the second dataset, including any of the following: Using the first and second datasets, a classification model is trained, and the results of the covariate offset analysis are determined based on the training results of the classification model. Based on the first performance evaluation index of the target model processing the first dataset, the second performance evaluation index of the target model processing the second dataset, and the distribution density ratio of the model input data between the first dataset and the second dataset, determine the covariate shift analysis results; Based on the distribution of model input data in the first dataset and the distribution of model input data in the second dataset, the covariate offset analysis results are determined.

[0016] In some embodiments, a classification model is trained using the first dataset and the second dataset, and the covariate shift analysis results are determined based on the training results of the classification model, including: The initial classification model is trained using the fifth data subset from the first dataset and the sixth data subset from the second dataset to obtain the classification model. The classification model is tested using the seventh data subset from the first dataset and the eighth data subset from the second dataset to obtain the third test result; the classification model is used to determine whether the data input into the classification model belongs to the first dataset, or whether the classification model is used to determine whether the data input into the classification model belongs to the second dataset. Based on the third performance evaluation index of the third test results, the covariate offset analysis results are determined.

[0017] In some embodiments, determining the covariate shift analysis results based on a third performance evaluation index of the third test results includes: In response to the third performance evaluation metric meeting the preset conditions, it is determined that there is a covariate bias between the first dataset and the second dataset; or, In response to the fact that the third performance evaluation metric does not meet the preset conditions, it is determined that there is no covariate offset between the first dataset and the second dataset.

[0018] In some embodiments, determining the covariate shift analysis results based on a first performance evaluation metric for the target model processing the first dataset, a second performance evaluation metric for the target model processing the second dataset, and the distribution density ratio of the model input data between the first dataset and the second dataset includes: The product of the first performance evaluation index and the distribution density ratio is calculated to obtain the product-calculated first performance evaluation index. The covariate offset analysis results are determined based on the consistency between the first and second performance evaluation indices calculated by the product.

[0019] In some embodiments, determining the covariate shift analysis results based on the consistency between the first performance evaluation index and the second performance evaluation index calculated by the product includes: If the first and second performance evaluation metrics for product calculation are consistent, it is determined that there is a covariate bias between the first and second datasets; or, Since there is no consistency between the first and second performance evaluation metrics in response to the product calculation, it is determined that there is no covariate offset between the first and second datasets.

[0020] In some embodiments, determining the covariate shift analysis results based on the distribution of model input data in the first dataset and the distribution of model input data in the second dataset includes: The model input data in the first dataset is mapped to a preset feature space to obtain the first mapping vector; and, The model input data in the second dataset is mapped to a preset feature space to obtain the second mapping vector; Based on the distribution of the first mapping vector and the distribution of the second mapping vector, the covariate offset analysis results are determined.

[0021] In some embodiments, determining the covariate offset analysis results based on the distribution of the first mapping vector and the distribution of the second mapping vector includes: Based on the distribution of the first mapping vector and the distribution of the second mapping vector, determine the second degree of difference between the first mapping vector and the second mapping vector; The results of the covariate shift analysis are determined based on the second degree of difference.

[0022] In some embodiments, determining the covariate shift analysis results based on the second degree of difference includes: In response to a second dissimilarity level exceeding a preset second dissimilarity threshold, a covariate offset is determined to exist between the first and second datasets; or, In response to a second difference degree being less than or equal to a preset second difference degree threshold, it is determined that there is no covariate offset between the first dataset and the second dataset.

[0023] In some embodiments, the performance analysis results include the reasons for the performance degradation of the target model. Based on the model shift analysis results and the covariate shift analysis results, the performance analysis results of the target model are determined as follows: In response to the model shift analysis showing a model shift between the first and second datasets, and the covariate shift analysis showing no covariate shift between the first and second datasets, the performance degradation is determined to be due to model shift; or, If the model offset analysis results show no model offset between the first and second datasets, and the covariate offset analysis results show a covariate offset between the first and second datasets, then the performance degradation is determined to be due to covariate offset; or, In response to the model offset analysis results showing a model offset between the first and second datasets, and the covariate offset analysis results showing a covariate offset between the first and second datasets, the performance degradation was determined to be caused by model offset and covariate offset.

[0024] A second aspect of this disclosure provides an apparatus for analyzing model performance, comprising: The acquisition unit is used to acquire the first dataset and the second dataset; the first dataset is the training dataset of the target model during the training phase, and the second dataset is the inference dataset of the target model during the inference phase. The analysis unit is used to perform model shift analysis and covariate shift analysis on the first dataset and the second dataset, and obtain the model shift analysis results and covariate shift analysis results. The model shift analysis results are used to characterize the difference between the conditional distribution of the first dataset and the conditional distribution of the second dataset. The conditional distribution is the mapping relationship between the model input data and the corresponding reference data. The covariate shift analysis results are used to characterize the difference between the model input data of the first dataset and the model input data of the second dataset. The determination unit is used to determine the performance analysis results of the target model based on the model offset analysis results and the covariate offset analysis results.

[0025] In some embodiments, the analysis unit is further configured to: Using the model input data as a benchmark, the first similarity between the first dataset and the second dataset on the reference data is analyzed, and the model offset analysis results are determined based on the first similarity. The initial model was trained using the first dataset and the second dataset respectively, the training results were obtained, and the model offset analysis results were determined based on the training results. Using the reference data as a benchmark, we analyze the first degree of difference between the first dataset and the second dataset in terms of the model input data, and determine the model offset analysis results based on the first degree of difference.

[0026] In some embodiments, the analysis unit is further configured to: Based on the second similarity between the model input data in the first dataset and the first target model input data, a predetermined number of first model input data in the first dataset are determined; the first target model input data is at least one model input data in the second dataset. A similarity analysis is performed on the reference data corresponding to a preset number of first model input data and the reference data corresponding to the first target model input data to obtain the first similarity.

[0027] In some embodiments, the analysis unit is further configured to: Based on the descending order of the second similarity, a preset number of first model input data are selected from the model input data in the first dataset.

[0028] In some embodiments, the analysis unit is further configured to: In response to a preset quantity of one, a similarity analysis is performed on the reference data corresponding to the input data of the first model and the reference data corresponding to the input data of the first target model to obtain a first similarity score; or, In response to a preset number of reference data, statistical values ​​are calculated for the preset number of reference data corresponding to the first model input data to obtain statistical reference data; A similarity analysis is performed on the reference data corresponding to the statistical reference data and the input data of the first target model to obtain the first similarity.

[0029] In some embodiments, the analysis unit is further configured to: In response to a first similarity score greater than a preset similarity threshold, a model bias is determined to exist between the first dataset and the second dataset; or, In response to a first similarity less than or equal to a preset similarity threshold, it is determined that there is no model offset between the first dataset and the second dataset.

[0030] In some embodiments, the analysis unit is further configured to: Determine the input data of the second target model in the first dataset; the input data of the second target model is the model input data corresponding to the maximum value in the second similarity. Based on the third similarity between the remaining model input data in the first dataset and the input data of the second target model, a predetermined number of second model input data in the first dataset are determined. A fourth similarity score is obtained by performing a similarity analysis on the reference data corresponding to a preset number of second model input data and the reference data corresponding to the first target model input data. In response to the first similarity being less than the fourth similarity, a model shift is determined to exist between the first dataset and the second dataset.

[0031] In some embodiments, the analysis unit is further configured to: The initial model is trained using the first subset of data in the first dataset to obtain the first target model; and the initial model is trained using the second subset of data in the second dataset to obtain the second target model. The first target model is tested using the third subset of data in the first dataset to obtain the first test result; and the second target model is tested using the fourth subset of data in the second dataset to obtain the second test result; the training result includes the first test result and the second test result. The model offset analysis results are determined based on the consistency between the first and second test results.

[0032] In some embodiments, the analysis unit is further configured to: In response to the consistency between the first and second test results, it is determined that there is no model offset between the first and second datasets; or, In response to the lack of consistency between the first and second test results, it is determined that there is a model bias between the first and second datasets.

[0033] In some embodiments, the analysis unit is further configured to: Determine the first reference data in the first dataset and the second reference data in the second dataset; the similarity between the first reference data and the second reference data is greater than a preset similarity threshold; The first degree of difference is determined based on the distribution of the model input data of the first reference data and the distribution of the model input data of the second reference data.

[0034] In some embodiments, the analysis unit is further configured to: In response to a first difference value exceeding a preset first difference threshold, a model offset is determined to exist between the first dataset and the second dataset; or, In response to a first difference degree being less than or equal to a preset first difference threshold, it is determined that there is no model offset between the first dataset and the second dataset.

[0035] In some embodiments, the analysis unit is further configured to: Using the first and second datasets, a classification model is trained, and the results of the covariate offset analysis are determined based on the training results of the classification model. Based on the first performance evaluation index of the target model processing the first dataset, the second performance evaluation index of the target model processing the second dataset, and the distribution density ratio of the model input data between the first dataset and the second dataset, determine the covariate shift analysis results; Based on the distribution of model input data in the first dataset and the distribution of model input data in the second dataset, the covariate offset analysis results are determined.

[0036] In some embodiments, the analysis unit is further configured to: The initial classification model is trained using the fifth data subset from the first dataset and the sixth data subset from the second dataset to obtain the classification model. The classification model is tested using the seventh data subset from the first dataset and the eighth data subset from the second dataset to obtain the third test result; the classification model is used to determine whether the data input into the classification model belongs to the first dataset, or whether the classification model is used to determine whether the data input into the classification model belongs to the second dataset. Based on the third performance evaluation index of the third test results, the covariate offset analysis results are determined.

[0037] In some embodiments, the analysis unit is further configured to: In response to the third performance evaluation metric meeting the preset conditions, it is determined that there is a covariate bias between the first dataset and the second dataset; or, In response to the fact that the third performance evaluation metric does not meet the preset conditions, it is determined that there is no covariate offset between the first dataset and the second dataset.

[0038] In some embodiments, the analysis unit is further configured to: The product of the first performance evaluation index and the distribution density ratio is calculated to obtain the product-calculated first performance evaluation index. The covariate offset analysis results are determined based on the consistency between the first and second performance evaluation indices calculated by the product.

[0039] In some embodiments, the analysis unit is further configured to: If the first and second performance evaluation metrics for product calculation are consistent, it is determined that there is a covariate bias between the first and second datasets; or, Since there is no consistency between the first and second performance evaluation metrics in response to the product calculation, it is determined that there is no covariate offset between the first and second datasets.

[0040] In some embodiments, the analysis unit is further configured to: The model input data in the first dataset is mapped to a preset feature space to obtain the first mapping vector; and, The model input data in the second dataset is mapped to a preset feature space to obtain the second mapping vector; Based on the distribution of the first mapping vector and the distribution of the second mapping vector, the covariate offset analysis results are determined.

[0041] In some embodiments, the analysis unit is further configured to: Based on the distribution of the first mapping vector and the distribution of the second mapping vector, determine the second degree of difference between the first mapping vector and the second mapping vector; The results of the covariate shift analysis are determined based on the second degree of difference.

[0042] In some embodiments, the analysis unit is further configured to: In response to a second dissimilarity level exceeding a preset second dissimilarity threshold, a covariate offset is determined to exist between the first and second datasets; or, In response to a second difference degree being less than or equal to a preset second difference degree threshold, it is determined that there is no covariate offset between the first dataset and the second dataset.

[0043] In some embodiments, the performance analysis results include the reasons for the performance degradation of the target model, and the determining unit is further configured to: In response to the model shift analysis showing a model shift between the first and second datasets, and the covariate shift analysis showing no covariate shift between the first and second datasets, the performance degradation is determined to be due to model shift; or, If the model offset analysis results show no model offset between the first and second datasets, and the covariate offset analysis results show a covariate offset between the first and second datasets, then the performance degradation is determined to be due to covariate offset; or, In response to the model offset analysis results showing a model offset between the first and second datasets, and the covariate offset analysis results showing a covariate offset between the first and second datasets, the performance degradation was determined to be caused by model offset and covariate offset.

[0044] A third aspect of this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods described in the first aspect of this disclosure.

[0045] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium that, when instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform the methods described in the first aspect of this disclosure.

[0046] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described in the embodiments of the first aspect of this disclosure.

[0047] A sixth aspect of this disclosure provides a chip including one or more interfaces and one or more processors; the interfaces are configured to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory, which, when executed by the processors, cause the electronic device to perform the methods described in the first aspect of this disclosure.

[0048] In summary, the model performance analysis method proposed in this disclosure includes obtaining a first dataset and a second dataset; the first dataset is the training dataset of the target model during the training phase, and the second dataset is the inference dataset of the target model during the inference phase; model shift analysis and covariate shift analysis are performed on the first and second datasets to obtain model shift analysis results and covariate shift analysis results; the model shift analysis results are used to characterize the difference between the conditional distribution of the first dataset and the conditional distribution of the second dataset, where the conditional distribution is the mapping relationship between the model input data and the corresponding reference data; the covariate shift analysis results are used to characterize the difference between the model input data of the first dataset and the model input data of the second dataset; and the performance analysis results of the target model are determined based on the model shift analysis results and the covariate shift analysis results. Model shift analysis can determine the change in the conditional distribution between the model input data and the reference data, and covariate shift analysis can determine the shift change in the distribution of the model input data; determining the performance analysis results based on the model shift analysis results and the covariate shift analysis results allows for the identification of the causes of model performance degradation from two dimensions: the change in the distribution of the model input data and the mapping relationship between the model input data and the reference data, thereby improving the efficiency and accuracy of model optimization.

[0049] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0051] Figure 1 A flowchart illustrating a method for analyzing model performance provided in this embodiment of the disclosure; Figure 2 A flowchart illustrating another method for analyzing model performance provided in this embodiment of the disclosure; Figure 3 A schematic diagram of the structure of a model performance analysis device provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the structure of a chip provided in an embodiment of the present disclosure. Detailed Implementation

[0052] Some embodiments of this disclosure will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0053] The embodiments described in the following examples of this disclosure are not representative of all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0054] With the continuous development of artificial intelligence technology, the application of AI-based models is becoming more and more widespread. During the application of these models, it is necessary to analyze their performance in order to optimize them based on the results.

[0055] Therefore, to address the problems existing in related technologies, this disclosure proposes a method for analyzing model performance. This method includes acquiring a first dataset and a second dataset; the first dataset is the training dataset of the target model during the training phase, and the second dataset is the inference dataset of the target model during the inference phase; performing model shift analysis and covariate shift analysis on the first and second datasets to obtain model shift analysis results and covariate shift analysis results; the model shift analysis results are used to characterize the difference between the conditional distribution of the first dataset and the conditional distribution of the second dataset, where the conditional distribution is the mapping relationship between the model input data and the corresponding reference data; the covariate shift analysis results are used to characterize the difference between the model input data of the first dataset and the model input data of the second dataset; and determining the performance analysis results of the target model based on the model shift analysis results and the covariate shift analysis results. Model shift analysis can determine the changes in the conditional distribution between the model input data and the reference data, while covariate shift analysis can determine the shift changes in the distribution of the model input data; determining the performance analysis results based on the model shift analysis results and the covariate shift analysis results allows for the identification of the causes of model performance degradation from two dimensions: the changes in the distribution of the model input data and the mapping relationship between the model input data and the reference data, thereby improving the efficiency and accuracy of model optimization.

[0056] This disclosure is not exhaustive, but merely illustrative of some embodiments, and is not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined, for example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0057] In each of the disclosed embodiments, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of the embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0058] The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure.

[0059] In this embodiment of the disclosure, unless otherwise stated, elements expressed in the singular form, such as "a," "an," "the," "the aforementioned," "the," "this," etc., can mean "one and only one," or "one or more," "at least one," etc. For example, when using articles such as "a," "an," "the," etc. in translation, the noun following the article can be understood as either a singular or a plural expression.

[0060] In some embodiments, the terms “in response to…”, “in response to determining…”, “in the case of…”, “when…”, “if…”, “if…”, etc., can be used interchangeably.

[0061] In some embodiments, the terms “greater than,” “greater than or equal to,” “not less than,” “more than,” “more than or equal to,” “not less than,” “higher than,” “higher than or equal to,” “not lower than,” and “above” can be used interchangeably, as can the terms “less than,” “less than or equal to,” “not greater than,” “less than,” “less than or equal to,” “not more than,” “lower than,” “lower than or equal to,” “not higher than,” and “below”.

[0062] The prefixes such as "first" and "second" in the embodiments of this disclosure are only for distinguishing different descriptive objects and do not constitute restrictions on the position, order, priority, number or content of the descriptive objects. For the description of the descriptive objects, please refer to the description in the claims or the context of the embodiments. The use of prefixes should not constitute unnecessary restrictions.

[0063] In the embodiments of this disclosure, "multiple" refers to two or more.

[0064] In the embodiments disclosed herein, terms such as “import”, “input”, and “read in” can be used interchangeably.

[0065] In some embodiments, devices, etc., can be interpreted as physical or virtual, and their names are not limited to the names recorded in the embodiments. Terms such as “device”, “equipment”, “circuit”, “network element”, “node”, “function”, “unit”, “section”, “system”, “network”, “chip”, “chip system”, “entity”, and “subject” can be used interchangeably.

[0066] In some embodiments, the collection, storage, use, processing, transmission, provision, and application of user personal information as described in this disclosure comply with relevant laws and regulations and do not violate public order and good morals. It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should be conducted only after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notification and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0067] Figure 1 This is a flowchart illustrating a model performance analysis method provided in an embodiment of this disclosure. This method can be applied to application scenarios such as smart terminals, for example, executed by a terminal with integrated model performance analysis functions or a processor within a terminal, or executed by other devices suitable for performing model performance analysis; this disclosure does not limit its scope. Figure 1 As shown, the method for analyzing the performance of this model includes steps S101-S103.

[0068] Step S101: Obtain the first dataset and the second dataset; the first dataset is the training dataset of the target model during the training phase, and the second dataset is the inference dataset of the target model during the inference phase.

[0069] In the embodiments of this disclosure, the target model can be an artificial intelligence model, which may include, but is not limited to, wireless communication artificial intelligence models, speech processing models, image processing models, and robotic artificial intelligence models. Taking a wireless communication artificial intelligence model as an example, model performance analysis can be used for the lifecycle management of wireless communication artificial intelligence models. However, it should be clarified that this statement is not intended to limit the target model to only the above-mentioned models; other types of models are also possible. The embodiments of this disclosure do not limit the methods for obtaining the first dataset and the second dataset.

[0070] By obtaining the first dataset for the training phase and the second dataset for the inference phase respectively, the subsequent analysis of model performance can compare the differences in the distribution and mapping relationships of the data processed by the target model at different stages. This provides a data basis for distinguishing between model shift and covariate shift, and improves the accuracy of model performance analysis results.

[0071] Step S102: Perform model shift analysis and covariate shift analysis on the first dataset and the second dataset to obtain the model shift analysis results and covariate shift analysis results. The model shift analysis results are used to characterize the difference between the conditional distribution of the first dataset and the conditional distribution of the second dataset. The conditional distribution is the mapping relationship between the model input data and the reference data corresponding to the model input data. The covariate shift analysis results are used to characterize the difference between the model input data of the first dataset and the model input data of the second dataset.

[0072] In the embodiments of this disclosure, the reference data may be the labels corresponding to the model input data or the actual result data. The first dataset may include the model input data and the labels of the model input data, and the second dataset may also include the model input data and the labels corresponding to the model input data.

[0073] By analyzing the differences in conditional distribution and input data respectively, the reasons for the decline in model performance are decomposed into two dimensions: model bias and covariate bias. This allows the performance analysis results to pinpoint the source of performance degradation, reduce misjudgments caused by a single analysis method, and provide direction for targeted model optimization.

[0074] Step S103: Determine the performance analysis results of the target model based on the model offset analysis results and the covariate offset analysis results.

[0075] In the embodiments of this disclosure, the performance analysis results may indicate that the target model has a performance degradation and the reasons for the performance degradation, or the performance analysis results may indicate that the target model does not have a performance degradation.

[0076] By combining the results of offset analysis from two different dimensions to comprehensively evaluate model performance, the bias caused by single-dimensional analysis is reduced, and the accuracy of performance analysis results in reflecting the impact of dataset differences on model performance is improved.

[0077] According to the model performance analysis method proposed in this disclosure, the method includes acquiring a first dataset and a second dataset; the first dataset is the training dataset of the target model during the training phase, and the second dataset is the inference dataset of the target model during the inference phase; performing model shift analysis and covariate shift analysis on the first and second datasets to obtain model shift analysis results and covariate shift analysis results; the model shift analysis results are used to characterize the difference between the conditional distribution of the first dataset and the conditional distribution of the second dataset, where the conditional distribution is the mapping relationship between the model input data and the corresponding reference data; the covariate shift analysis results are used to characterize the difference between the model input data of the first dataset and the model input data of the second dataset; and the performance analysis results of the target model are determined based on the model shift analysis results and the covariate shift analysis results. Model shift analysis can determine the change in the conditional distribution between the model input data and the reference data, and covariate shift analysis can determine the shift change in the distribution of the model input data; determining the performance analysis results based on the model shift analysis results and the covariate shift analysis results allows for the identification of the causes of model performance degradation from two dimensions: the change in the distribution of the model input data and the mapping relationship between the model input data and the reference data, thereby improving the efficiency and accuracy of model optimization.

[0078] For example, when performing model shift analysis on the first dataset and the second dataset, it can be implemented in, but is not limited to, the following ways, including any of the following: using the model input data as a benchmark, analyzing the first similarity between the first dataset and the second dataset on the reference data, and determining the model shift analysis result based on the first similarity; training an initial model using the first dataset and the second dataset respectively, obtaining the training result, and determining the model shift analysis result based on the training result; using the reference data as a benchmark, analyzing the first difference between the first dataset and the second dataset on the model input data, and determining the model shift analysis result based on the first difference.

[0079] In embodiments of this disclosure, the first similarity can be the degree of similarity between the first dataset and the second dataset on reference data, based on the model input data. Embodiments of this disclosure do not limit the specific calculation method of the first similarity.

[0080] In embodiments of this disclosure, the initial model can be an untrained model to be trained. The training result can be the performance of the initial model after training on the dataset. The first dissimilarity can be the degree of difference between the first dataset and the second dataset in terms of model input data when using reference data as a benchmark. Embodiments of this disclosure do not limit the model structure of the initial model.

[0081] By setting three model offset detection methods, different application scenarios can be adapted to improve the applicability of model offset analysis.

[0082] Figure 2 This is a flowchart illustrating a model performance analysis method proposed in this disclosure. Based on Figure 1 The embodiment shown, Figure 2 This may include the following steps: Step S201: Based on the second similarity between the model input data in the first dataset and the first target model input data, determine a preset number of first model input data in the first dataset; the first target model input data is at least one model input data in the second dataset.

[0083] In embodiments of this disclosure, the second similarity is the degree of similarity between the model input data in the first dataset and the first target model input data.

[0084] In embodiments of this disclosure, a model input data point can be selected from the second dataset as the first target model input data, and a second similarity score can be calculated between each model input data point in the first dataset and the first target model input data. This disclosure does not limit the specific calculation method for the second similarity score. Each model input data point in the second dataset can have its second similarity score calculated with the model input data points in the first dataset.

[0085] By using at least one model input data from the second dataset as a reference to filter the model input data from the first dataset, the comparison range can be narrowed, the computational cost of the second similarity can be reduced, and the efficiency of model offset analysis can be improved.

[0086] Step S202: Perform similarity analysis on the reference data corresponding to a preset number of first model input data and the reference data corresponding to the first target model input data to obtain the first similarity.

[0087] In embodiments of this disclosure, the preset quantity can be the number of first model input data selected from the first dataset. This disclosure does not limit the specific value of the preset quantity.

[0088] The preset quantity can be flexibly adjusted according to actual needs and application scenarios, which improves the flexibility of selecting input data for the first model.

[0089] For example, when determining a preset number of first model input data in the first dataset based on the second similarity between the model input data in the first dataset and the first target model input data, the following methods can be used, but are not limited to: selecting a preset number of first model input data from the model input data in the first dataset in descending order of the second similarity.

[0090] In embodiments of this disclosure, the descending order can be an order of sorting by the numerical value of the second similarity from largest to smallest. This disclosure provides an example of selecting first model input data. Assume the first dataset includes five model input data points: model input data 1, model input data 2, model input data 3, model input data 4, and model input data 5. The second similarity between model input data 1 and the first target model input data is 'a', the second similarity between model input data 2 and the first target model input data is 'b', the second similarity between model input data 3 and the first target model input data is 'c', the second similarity between model input data 4 and the first target model input data is 'd', and the second similarity between model input data 5 and the first target model input data is 'e', ​​where a > c > d > b > e. The preset number is 3. Therefore, the first model input data includes model input data 1, model input data 3, and model input data 4.

[0091] By selecting the first model input data in descending order of the second similarity, the selected first model input data has a high similarity to the first target model input data, reducing the interference of irrelevant data on model shift analysis and thus improving the accuracy of model shift analysis.

[0092] For example, when performing a similarity analysis on a preset number of reference data corresponding to the first model input data and the reference data corresponding to the first target model input data to obtain a first similarity, the following methods can be used, but are not limited to: in response to a preset number of one reference data, performing a similarity analysis on the reference data corresponding to the first model input data and the reference data corresponding to the first target model input data to obtain a first similarity; or, in response to a preset number of multiple reference data, performing statistical value calculation on a preset number of reference data corresponding to the first model input data to obtain statistical reference data; and performing a similarity analysis on the statistical reference data and the reference data corresponding to the first target model input data to obtain a first similarity.

[0093] In the embodiments of this disclosure, statistical value calculation may include, but is not limited to, average calculation, median calculation, mode calculation, and weighted calculation. However, it should be clarified that this statement is not intended to limit the statistical value calculation to the above-mentioned methods; other calculation methods may also be used. The embodiments of this disclosure do not limit the method for calculating the first similarity.

[0094] Differentiated processing methods are provided for the reference data corresponding to different preset quantities of first model input data. When the preset quantity is one, direct comparison is performed. When the preset quantity is multiple, statistical calculation is used to reduce individual fluctuations, thereby improving the accuracy of first similarity determination.

[0095] For example, when performing the model offset analysis result based on the first similarity, it can be implemented in the following ways, but not limited to: determining that there is a model offset between the first dataset and the second dataset in response to the first similarity being greater than a preset similarity threshold; or determining that there is no model offset between the first dataset and the second dataset in response to the first similarity being less than or equal to the preset similarity threshold.

[0096] In the embodiments of this disclosure, the specific value of the preset similarity threshold is not limited.

[0097] The preset similarity threshold can be flexibly adjusted according to actual needs and application scenarios, which improves the flexibility of model offset analysis.

[0098] For example, when performing the model offset analysis result based on the first similarity, it can be implemented in the following ways, but is not limited to: determining the second target model input data in the first dataset; the second target model input data is the model input data corresponding to the maximum value in the second similarity; determining a preset number of second model input data in the first dataset based on the third similarity between the remaining model input data in the first dataset and the second target model input data; performing a similarity analysis on the reference data corresponding to the preset number of second model input data and the reference data corresponding to the first target model input data to obtain a fourth similarity; and determining that there is a model offset between the first dataset and the second dataset in response to the first similarity being less than the fourth similarity.

[0099] In the embodiments of this disclosure, the calculation methods for the third and fourth similarities are not limited. By introducing the input data of the second target model and calculating the fourth similarity through secondary sampling as a reference benchmark, the comparison relationship between the first and fourth similarities is used to determine whether model shift exists, reducing misjudgments caused by local data anomalies in the first dataset and improving the accuracy of model shift determination.

[0100] For example, when performing the execution of training an initial model using a first dataset and a second dataset respectively, obtaining training results, and determining the model offset analysis results based on the training results, the following methods can be used, but are not limited to: training the initial model using a first subset of data in the first dataset to obtain a first target model; and training the initial model using a second subset of data in the second dataset to obtain a second target model; testing the first target model using a third subset of data in the first dataset to obtain a first test result; and testing the second target model using a fourth subset of data in the second dataset to obtain a second test result; the training results include the first test result and the second test result; and the model offset analysis results are determined based on the consistency between the first test result and the second test result.

[0101] In the embodiments of this disclosure, the proportions of the first and third data subsets are not limited, nor are the proportions of the second and fourth data subsets. The embodiments of this disclosure do not limit the model results of the first and second target models, nor do they limit the training method of the initial model.

[0102] In the embodiments of this disclosure, the first test result is the model output data obtained after testing the first target model using a third data subset. The second test result is the model output data obtained after testing the second target model using a fourth data subset.

[0103] By training identical models on two datasets and comparing their test performance, the differences in conditional distribution are transformed into differences in model generalization ability, thus enriching the methods of model bias analysis.

[0104] For example, when determining the model offset analysis result based on the consistency between the first test result and the second test result, it can be implemented in the following ways, but not limited to: in response to the consistency between the first test result and the second test result, determining that there is no model offset between the first dataset and the second dataset; or in response to the lack of consistency between the first test result and the second test result, determining that there is a model offset between the first dataset and the second dataset.

[0105] In the embodiments of this disclosure, consistency between the first test result and the second test result can be defined as the difference between the first test result and the second test result being less than a preset difference threshold, while non-consistency between the first test result and the second test result can be defined as the difference between the first test result and the second test result being greater than or equal to the preset difference threshold. The embodiments of this disclosure do not limit the specific value of the preset difference threshold.

[0106] Using the consistency of model test results as the criterion, the impact of differences in the distribution of related data on model performance is considered, making the offset determination more in line with the actual model usage scenarios.

[0107] For example, when performing a first difference analysis of the model input data between the first dataset and the second dataset based on reference data, it can be implemented in the following ways, but is not limited to: determining the first reference data in the first dataset and the second reference data in the second dataset; the similarity between the first reference data and the second reference data is greater than a preset similarity threshold; and determining the first difference based on the distribution of the model input data of the first reference data and the distribution of the model input data of the second reference data.

[0108] In the embodiments of this disclosure, the specific value of the preset similarity threshold is not limited. The first reference data can be the label or annotation data corresponding to the model input data in the first dataset, and the second reference data can be the label or annotation data corresponding to the model input data in the second dataset.

[0109] In the embodiments of this disclosure, the distribution of the model input data of the first reference data can be the distribution of the model input data corresponding to the first reference data in the feature space or statistical dimension. The distribution of the model input data of the second reference data can be the distribution of the model input data corresponding to the second reference data in the feature space or statistical dimension. The embodiments of this disclosure do not limit the specific method for calculating the first difference.

[0110] The preset similarity threshold can be flexibly adjusted according to actual needs and application scenarios, which improves the flexibility of selecting the first and second reference data.

[0111] For example, when performing the model offset analysis result based on the first difference degree, it can be implemented in the following ways, but not limited to: determining that there is a model offset between the first dataset and the second dataset in response to the first difference degree being greater than a preset first difference degree threshold; or determining that there is no model offset between the first dataset and the second dataset in response to the first difference degree being less than or equal to the preset first difference degree threshold.

[0112] In the embodiments of this disclosure, the specific value of the preset first difference threshold is not limited. The preset first difference threshold can be flexibly adjusted according to actual needs and application scenarios, thereby improving the flexibility of model offset determination.

[0113] For example, when performing covariate shift analysis on the first dataset and the second dataset, it can be implemented in, but is not limited to, the following ways, including any of the following: training a classification model using the first dataset and the second dataset, and determining the covariate shift analysis results based on the training results of the classification model; determining the covariate shift analysis results based on the first performance evaluation metric of the target model processing the first dataset, the second performance evaluation metric of the target model processing the second dataset, and the distribution density ratio of the model input data between the first dataset and the second dataset; determining the covariate shift analysis results based on the distribution of the model input data in the first dataset and the distribution of the model input data in the second dataset.

[0114] In embodiments of this disclosure, the classification model can be a discriminative model used to distinguish whether input data belongs to a first dataset or a second dataset. The training result can be the classification performance of the classification model on test data. The first performance evaluation metric can be the performance metric obtained by the target model after processing the first dataset. The second performance evaluation metric can be the performance metric obtained by the target model after processing the second dataset.

[0115] In the embodiments of this disclosure, the distribution density ratio is the ratio of the probability density of the model input data between the first dataset and the second dataset. The first performance evaluation metric can be the accuracy of the target model in processing the first dataset, and the second performance evaluation metric can be the accuracy of the target model in processing the second dataset. However, it should be clarified that this statement is not intended to limit the first performance evaluation metric to the aforementioned metrics; other metrics are also possible. The distribution of the model input data can refer to the distribution of the model input data in the feature space or statistical dimensions.

[0116] It provides three different methods for covariate shift analysis, which can be flexibly selected according to actual needs and application scenarios.

[0117] For example, when training a classification model using the first and second datasets, and determining the covariate shift analysis results based on the training results of the classification model, the following methods can be used, but are not limited to: training an initial classification model using the fifth data subset in the first dataset and the sixth data subset in the second dataset to obtain a classification model; testing the classification model using the seventh data subset in the first dataset and the eighth data subset in the second dataset to obtain a third test result; the classification model is used to determine whether the data input into the classification model belongs to the first dataset, or whether the classification model is used to determine whether the data input into the classification model belongs to the second dataset; and the covariate shift analysis results are determined based on the third performance evaluation index of the third test result.

[0118] In the embodiments of this disclosure, the proportions of the seventh and fifth data subsets are not limited, nor are the proportions of the eighth and sixth data subsets. The training and testing processes of the classification model are not limited in the embodiments of this disclosure.

[0119] In the embodiments of this disclosure, the model results of the initial classification model are not limited. The third test result can be the model output data obtained after testing the classification model using the seventh and eighth data subsets. The third performance evaluation index can be a metric extracted from the third test result to measure the performance of the classification model. The third performance evaluation index can be the recognition accuracy of the classification model in identifying the seventh and eighth data subsets. However, it should be clear that this statement is not intended to limit the third performance evaluation index to the above-mentioned index; other indices are also possible.

[0120] By training a classification model that can distinguish between two datasets, covariate shift detection is transformed into a classification task. The generalization ability of the classification model is used to indirectly reflect the degree of difference in input distribution, thus enriching the methods of covariate shift analysis.

[0121] For example, when determining the covariate offset analysis results based on the third performance evaluation index of the third test results, the following methods may be used, but are not limited to: determining that there is a covariate offset between the first dataset and the second dataset in response to the third performance evaluation index meeting the preset conditions; or determining that there is no covariate offset between the first dataset and the second dataset in response to the third performance evaluation index not meeting the preset conditions.

[0122] In the embodiments of this disclosure, the preset conditions can be a predefined range of performance indicators, and the specific values ​​of the performance indicator range are not limited in the embodiments of this disclosure. If the third performance evaluation indicator is within the performance indicator range, it is determined that there is a covariate offset between the first dataset and the second dataset; if the third performance evaluation indicator is not within the performance indicator range, it is determined that there is no covariate offset between the first dataset and the second dataset.

[0123] The preset conditions can be flexibly adjusted according to actual needs and application scenarios, which improves the flexibility of covariate shift analysis.

[0124] For example, when determining the covariate shift analysis results by performing a first performance evaluation index on the first dataset processed by the target model, a second performance evaluation index on the second dataset processed by the target model, and the distribution density ratio of the model input data between the first dataset and the second dataset, the following methods can be used, but are not limited to: performing a product calculation on the first performance evaluation index and the distribution density ratio to obtain a product-calculated first performance evaluation index; and determining the covariate shift analysis results based on the consistency between the product-calculated first performance evaluation index and the second performance evaluation index.

[0125] In the embodiments of this disclosure, the first performance evaluation index is corrected by combining the distribution density ratio, thereby reducing the performance comparison bias caused by distribution differences and improving the accuracy of covariate offset judgment.

[0126] For example, when determining the covariate offset analysis results based on the consistency between the first performance evaluation index and the second performance evaluation index calculated by the product, the following methods can be used, but are not limited to: determining that there is a covariate offset between the first dataset and the second dataset in response to the consistency between the first performance evaluation index and the second performance evaluation index calculated by the product; or determining that there is no covariate offset between the first dataset and the second dataset in response to the lack of consistency between the first performance evaluation index and the second performance evaluation index calculated by the product.

[0127] In the embodiments of this disclosure, consistency between the first and second performance evaluation indicators of the product calculation can be defined as the difference between the first and second performance evaluation indicators being less than a preset difference threshold. Inconsistency between the first and second performance evaluation indicators of the product calculation can be defined as the difference between the first and second test results being greater than or equal to the preset difference threshold. The embodiments of this disclosure do not limit the specific value of the preset difference threshold.

[0128] By judging the consistency between the corrected first performance evaluation index and the second performance evaluation index, the difference in distribution density is included in the reference range for performance comparison, which improves the ability of covariate offset determination to distinguish between true distribution offset and performance fluctuation caused by sampling density difference, thereby improving the accuracy of covariate offset determination.

[0129] For example, when determining the covariate shift analysis result based on the distribution of model input data in the first dataset and the distribution of model input data in the second dataset, the following methods can be used, but are not limited to: mapping the model input data in the first dataset to a preset feature space to obtain a first mapping vector; and mapping the model input data in the second dataset to a preset feature space to obtain a second mapping vector; and determining the covariate shift analysis result based on the distribution of the first mapping vector and the distribution of the second mapping vector.

[0130] In embodiments of this disclosure, the preset feature space can be a predefined multidimensional space used to characterize data features. The first mapping vector and the second mapping vector are the coordinate representations of the model input data in the first dataset and the second dataset, respectively, in the preset feature space. Embodiments of this disclosure do not limit the dimension of the preset feature space.

[0131] In the embodiments of this disclosure, all model input data in the first dataset can be transformed into a preset feature space using a feature extraction function or mapping model, forming multiple first mapping vectors. Similarly, all model input data in the second dataset can be transformed into a preset feature space, forming multiple second mapping vectors. The distribution of the first and second mapping vectors in the feature space is analyzed, and the covariate shift analysis results are determined based on a comparison of the distributions of the first and second mapping vectors.

[0132] By mapping model input data to a predefined feature space for distribution analysis, the interference of differences in the form of model input data on distribution comparison can be reduced, thereby improving the accuracy of covariate shift analysis.

[0133] For example, when determining the covariate offset analysis result based on the distribution of the first mapping vector and the distribution of the second mapping vector, the following methods can be used, but are not limited to: determining a second degree of difference between the first mapping vector and the second mapping vector based on the distribution of the first mapping vector and the distribution of the second mapping vector; and determining the covariate offset analysis result based on the second degree of difference.

[0134] In the embodiments of this disclosure, the specific calculation method of the second difference degree is not limited. By quantifying the distribution comparison into the second difference degree, the covariate shift analysis is quantified into a numerical comparison, thereby improving the accuracy of the covariate shift analysis.

[0135] For example, when performing the covariate offset analysis based on the second dissimilarity, the following methods may be used, but are not limited to: determining that there is a covariate offset between the first dataset and the second dataset in response to the second dissimilarity being greater than a preset second dissimilarity threshold; or determining that there is no covariate offset between the first dataset and the second dataset in response to the second dissimilarity being less than or equal to the preset second dissimilarity threshold.

[0136] In the embodiments of this disclosure, the specific value of the second difference threshold is not limited. The preset second difference threshold can be flexibly adjusted according to actual needs and application scenarios, thereby improving the flexibility of covariate shift analysis.

[0137] For example, the performance analysis results include the reasons for the performance degradation of the target model. When determining the performance analysis results of the target model based on the model shift analysis results and the covariate shift analysis results, the following methods can be used, but are not limited to: responding to the model shift analysis results indicating that there is a model shift between the first dataset and the second dataset, and the covariate shift analysis results indicating that there is no covariate shift between the first dataset and the second dataset, determining that the performance degradation is caused by model shift; or, responding to the model shift analysis results indicating that there is no model shift between the first dataset and the second dataset, and the covariate shift analysis results indicating that there is a covariate shift between the first dataset and the second dataset, determining that the performance degradation is caused by both model shift and covariate shift.

[0138] In the embodiments of this disclosure, by cross-judging the analysis results of both model shift and covariate shift, it is possible to distinguish between performance degradation caused by a single factor or a combination of factors, providing optimization directions for subsequent targeted model retraining or data adjustment.

[0139] In embodiments of this disclosure, the present invention is used to analyze the reasons for performance degradation when a model is migrated from dataset A (i.e., the first dataset) to dataset B (i.e., the second dataset). This invention is compatible with different tasks such as classification, regression, and ranking, and is compatible with both model-based and model-free analysis workflows. It can serve as a general diagnostic tool in R&D, validation, admission, regression testing, and version evolution. The process involves: acquiring dataset A and dataset B; analyzing model shift based on dataset A and dataset B; analyzing covariate shift based on dataset A and dataset B; and outputting the attribution for the performance degradation of the target model on dataset B based on the model shift analysis results and the covariate shift analysis results.

[0140] In the embodiments of this disclosure, it is assumed that an embodiment first obtains dataset A and dataset B; wherein dataset A may be a model training set, simulation set, historical version dataset, or reference domain dataset, and dataset B may be a real-world application set, field set, new version test set, or target domain dataset. The input variables are defined as follows: (i.e., model input data), output variables are (i.e., reference data); among which It can be discrete labels, continuous values, scores, or outputs from any task. Model offset analysis determines whether the conditional distribution between dataset A and dataset B has changed, i.e., whether the mapping from input to label has changed.

[0141] In embodiments of this disclosure, covariate shift analysis is used to determine whether the input distribution between dataset A and dataset B has changed, i.e., whether the data coverage and sampling distribution have changed. Based on the combined results of these analyses, an attribution conclusion is output. This conclusion includes at least: only model shift exists, only covariate shift exists, and both exist simultaneously. It may also include: when both exist simultaneously, the weights of their respective effects.

[0142] In the embodiments of this disclosure, model shift is also called concept shift or label shift. Essentially, it means that even if input samples A and B are very similar or even identical, the corresponding labels or output patterns still change. In wireless communication scenarios, this change can stem from implementation deviations between the algorithm link and the actual hardware link, changes in labeling mechanisms, changes in version rules, changes in system timing, changes in protocol stack behavior, or changes in measurement standards, etc.

[0143] In the embodiments of this disclosure, the model offset is expressed as:

[0144] in, In the dataset Given model input variables Corresponding model output variables The probability of the conditional distribution, In the dataset Given model input variables Corresponding model output variables The probability of the conditional distribution.

[0145] In embodiments of this disclosure, the embodiments may include, but are not limited to, the following methods or metrics for analyzing model shifts: proximity consistency test (preferred), for each sample in dataset B. Retrieve one or more nearest neighbor samples in dataset A. The system compares the consistency of discrete labels and the proximity of continuous outputs. These nearest neighbors can be constructed in the original feature space, normalized space, metric learning space, or a space weighted by prior domain knowledge. Numerical metrics for evaluating nearest neighbors include Euclidean distance and cosine similarity. When using multiple nearest neighbors, the labels can be averaged, voted, or weighted by an inverse distance. To determine model shift, the consistency check can first calculate a reference consistency baseline within dataset A (i.e., for each sample in dataset A). Retrieve one or more nearest neighbor samples in dataset A. This involves comparing whether discrete labels are consistent or whether continuous outputs are sufficiently close to obtain a reference result. Subsequently, for each sample in dataset B... Retrieve one or more nearest neighbor samples in dataset A. The model is then compared to see if discrete labels are consistent or if continuous outputs are sufficiently close. If the nearest neighbors of sample B in A generally have low consistency, or show a significant decrease compared to the reference consistency baseline, it indicates a model shift.

[0146] In the embodiments of this disclosure, the model consistency test involves training two target models with the same structure using dataset A and dataset B respectively, and comparing the consistency of the outputs of the two models on the same test set. If the consistency is low, it indicates that there is a model shift between the two datasets. This method can serve as supplementary evidence to the proximity consistency test, but because its conclusions are affected by factors such as model selection and training strategies, its reliability is lower than that of the proximity consistency test.

[0147] In the embodiments of this disclosure, the Maximum Mean Discrepancy (MMD) test can compare the conditional distribution differences between datasets A and B under different label conditions. When the labels are discrete, the conditional distributions of datasets A / B under different labels can be statistically analyzed; when the labels are continuous, the continuous labels can be divided into intervals, and the conditional distributions of datasets A / B can be statistically analyzed within the same label interval. The difference between the two distributions can be evaluated using either the distribution divergence or MMD. Specifically, MMD maps the distributions to a high-dimensional kernel space and calculates the mean distance between the two distributions in the high-dimensional space as an indicator of distribution difference. A large distribution difference indicates the existence of model shift. This method is suitable for use in conjunction with other high-confidence methods to supplement statistical significance analysis.

[0148] In the embodiments of this disclosure, covariate shift refers to a change in the input distribution, i.e., dataset B contains a large number of sample regions not fully covered by dataset A. In wireless communication scenarios, this shift typically arises from incomplete scenario configuration, insufficient use case coverage, inadequate parameter traversal, terminal state space expansion, network configuration changes, or the introduction of new input combinations from the external environment. Even if the original mapping remains unchanged, the model may experience performance degradation due to "not having seen this type of input before."

[0149] In embodiments of this disclosure, covariate offset is expressed as:

[0150] in, In the dataset Input variables of the model The probability distribution, In the dataset Input variables of the model The probability distribution.

[0151] In embodiments of this disclosure, the following methods or metrics for analyzing covariate shifts are included, but not limited to: domain classification (preferred), where samples from dataset A are labeled as domain label 0, and samples from dataset B are labeled as domain label 1, and a domain classifier is trained to determine whether any sample is more likely to come from A or B. The domain classifier can be a neural network. If the classification accuracy of the domain classifier on the test set is close to 0.5, it indicates that the input distributions of A and B are difficult to separate, and the covariate shift is not significant; if the accuracy is significantly higher than 0.5 and close to 1, it can be confirmed that there is a significant covariate shift between A and B. In imbalanced data scenarios, metrics such as recall and precision can be used in combination.

[0152] In embodiments of this disclosure, a covariate shift test based on density ratio is used if the assumed conditional distribution is... Keep it unchanged, but only input distribution Changes can be assessed by using density-weighted methods to transform evaluation metrics from dataset A into approximate estimates of dataset B, thereby determining whether performance differences are solely due to covariate shifts. (Density ratio) Represented as:

[0153] in, You can input data into the model In the dataset The probability of it appearing in You can input data into the model In the dataset The probability of it appearing in the middle.

[0154] In the embodiments of this disclosure, it is assumed that a domain classifier has been trained. If a certain sample is obtained through the domain classifier... The probability estimate of belonging to domain B is: The density ratio can then be further written as:

[0155] in, For datasets Sample size For datasets The number of samples. After applying the above weights to the sample loss, accuracy, error rate, or other evaluation functions in A, if the reweighted result can approximate the true metric in B, it indicates that the covariate bias can effectively explain the performance degradation.

[0156] In embodiments of this disclosure, the maximum mean difference test can be performed using a kernel method to measure the distribution difference between A and B in the feature space. First, a kernel mapping is defined. Mean embedding of dataset A :

[0157] in, For mathematical expectation operators, It can be the first mapping vector.

[0158] Then define the maximum mean difference between A and B. :

[0159] Among them, the mean embedding of dataset B .

[0160] In the embodiments of this disclosure, when the distributions of A and B are completely identical, MMD approaches 0; the larger the MMD, the greater the difference between the two.

[0161] In the embodiments of this disclosure, the embodiments are applicable as statistical auxiliary quantities, but their conclusions depend on the choice of kernel function and sample size, and are generally used as evidence of medium to low priority. In addition to the maximum mean difference, indicators such as the divergence of the difference distribution can also be used to assess the distributional difference between A and B.

[0162] Corresponding to the above-described model performance analysis method, this invention also proposes a model performance analysis apparatus. Since the apparatus embodiments of this invention correspond to the above-described method embodiments, details not disclosed in the apparatus embodiments can be referred to the above-described method embodiments, and will not be repeated here.

[0163] Figure 3 This is a schematic diagram of a model performance analysis device provided in an embodiment of the present disclosure. The model performance analysis device includes: The acquisition unit 41 is used to acquire the first dataset and the second dataset; the first dataset is the training dataset of the target model during the training phase, and the second dataset is the inference dataset of the target model during the inference phase. Analysis unit 42 is used to perform model shift analysis and covariate shift analysis on the first dataset and the second dataset to obtain model shift analysis results and covariate shift analysis results. The model shift analysis results are used to characterize the difference between the conditional distribution of the first dataset and the conditional distribution of the second dataset. The conditional distribution is the mapping relationship between the model input data and the reference data corresponding to the model input data. The covariate shift analysis results are used to characterize the difference between the model input data of the first dataset and the model input data of the second dataset. Unit 43 is used to determine the performance analysis results of the target model based on the model offset analysis results and the covariate offset analysis results.

[0164] According to the model performance analysis apparatus proposed in this disclosure, the apparatus includes acquiring a first dataset and a second dataset; the first dataset is the training dataset of the target model during the training phase, and the second dataset is the inference dataset of the target model during the inference phase; performing model shift analysis and covariate shift analysis on the first dataset and the second dataset to obtain model shift analysis results and covariate shift analysis results; the model shift analysis results are used to characterize the difference between the conditional distribution of the first dataset and the conditional distribution of the second dataset, where the conditional distribution is the mapping relationship between the model input data and the corresponding reference data; the covariate shift analysis results are used to characterize the difference between the model input data of the first dataset and the model input data of the second dataset; and determining the performance analysis results of the target model based on the model shift analysis results and the covariate shift analysis results. Model shift analysis can determine the change in the conditional distribution between the model input data and the reference data, and covariate shift analysis can determine the shift change in the distribution of the model input data; determining the performance analysis results based on the model shift analysis results and the covariate shift analysis results allows for the identification of the causes of model performance degradation from two dimensions: the change in the distribution of the model input data and the mapping relationship between the model input data and the reference data, thereby improving the efficiency and accuracy of model optimization.

[0165] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: Using the model input data as a benchmark, the first similarity between the first dataset and the second dataset on the reference data is analyzed, and the model offset analysis results are determined based on the first similarity. The initial model was trained using the first dataset and the second dataset respectively, the training results were obtained, and the model offset analysis results were determined based on the training results. Using the reference data as a benchmark, we analyze the first degree of difference between the first dataset and the second dataset in terms of the model input data, and determine the model offset analysis results based on the first degree of difference.

[0166] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: Based on the second similarity between the model input data in the first dataset and the first target model input data, a predetermined number of first model input data in the first dataset are determined; the first target model input data is at least one model input data in the second dataset. A similarity analysis is performed on the reference data corresponding to a preset number of first model input data and the reference data corresponding to the first target model input data to obtain the first similarity.

[0167] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: Based on the descending order of the second similarity, a preset number of first model input data are selected from the model input data in the first dataset.

[0168] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: In response to a preset quantity of one, a similarity analysis is performed on the reference data corresponding to the input data of the first model and the reference data corresponding to the input data of the first target model to obtain a first similarity score; or, In response to a preset number of reference data, statistical values ​​are calculated for the preset number of reference data corresponding to the first model input data to obtain statistical reference data; A similarity analysis is performed on the reference data corresponding to the statistical reference data and the input data of the first target model to obtain the first similarity.

[0169] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: In response to a first similarity score greater than a preset similarity threshold, a model bias is determined to exist between the first dataset and the second dataset; or, In response to a first similarity less than or equal to a preset similarity threshold, it is determined that there is no model offset between the first dataset and the second dataset.

[0170] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: Determine the input data of the second target model in the first dataset; the input data of the second target model is the model input data corresponding to the maximum value in the second similarity. Based on the third similarity between the remaining model input data in the first dataset and the input data of the second target model, a predetermined number of second model input data in the first dataset are determined. A fourth similarity score is obtained by performing a similarity analysis on the reference data corresponding to a preset number of second model input data and the reference data corresponding to the first target model input data. In response to the first similarity being less than the fourth similarity, a model shift is determined to exist between the first dataset and the second dataset.

[0171] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: The initial model is trained using the first subset of data in the first dataset to obtain the first target model; and the initial model is trained using the second subset of data in the second dataset to obtain the second target model. The first target model is tested using the third subset of data in the first dataset to obtain the first test result; and the second target model is tested using the fourth subset of data in the second dataset to obtain the second test result; the training result includes the first test result and the second test result. The model offset analysis results are determined based on the consistency between the first and second test results.

[0172] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: In response to the consistency between the first and second test results, it is determined that there is no model offset between the first and second datasets; or, In response to the lack of consistency between the first and second test results, it is determined that there is a model bias between the first and second datasets.

[0173] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: Determine the first reference data in the first dataset and the second reference data in the second dataset; the similarity between the first reference data and the second reference data is greater than a preset similarity threshold; The first degree of difference is determined based on the distribution of the model input data of the first reference data and the distribution of the model input data of the second reference data.

[0174] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: In response to a first difference value exceeding a preset first difference threshold, a model offset is determined to exist between the first dataset and the second dataset; or, In response to a first difference degree being less than or equal to a preset first difference threshold, it is determined that there is no model offset between the first dataset and the second dataset.

[0175] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: Using the first and second datasets, a classification model is trained, and the results of the covariate offset analysis are determined based on the training results of the classification model. Based on the first performance evaluation index of the target model processing the first dataset, the second performance evaluation index of the target model processing the second dataset, and the distribution density ratio of the model input data between the first dataset and the second dataset, determine the covariate shift analysis results; Based on the distribution of model input data in the first dataset and the distribution of model input data in the second dataset, the covariate offset analysis results are determined.

[0176] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: The initial classification model is trained using the fifth data subset from the first dataset and the sixth data subset from the second dataset to obtain the classification model. The classification model is tested using the seventh data subset from the first dataset and the eighth data subset from the second dataset to obtain the third test result; the classification model is used to determine whether the data input into the classification model belongs to the first dataset, or whether the classification model is used to determine whether the data input into the classification model belongs to the second dataset. Based on the third performance evaluation index of the third test results, the covariate offset analysis results are determined.

[0177] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: In response to the third performance evaluation metric meeting the preset conditions, it is determined that there is a covariate bias between the first dataset and the second dataset; or, In response to the fact that the third performance evaluation metric does not meet the preset conditions, it is determined that there is no covariate offset between the first dataset and the second dataset.

[0178] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: The product of the first performance evaluation index and the distribution density ratio is calculated to obtain the product-calculated first performance evaluation index. The covariate offset analysis results are determined based on the consistency between the first and second performance evaluation indices calculated by the product.

[0179] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: If the first and second performance evaluation metrics for product calculation are consistent, it is determined that there is a covariate bias between the first and second datasets; or, Since there is no consistency between the first and second performance evaluation metrics in response to the product calculation, it is determined that there is no covariate offset between the first and second datasets.

[0180] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: The model input data in the first dataset is mapped to a preset feature space to obtain the first mapping vector; and, The model input data in the second dataset is mapped to a preset feature space to obtain the second mapping vector; Based on the distribution of the first mapping vector and the distribution of the second mapping vector, the covariate offset analysis results are determined.

[0181] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: Based on the distribution of the first mapping vector and the distribution of the second mapping vector, determine the second degree of difference between the first mapping vector and the second mapping vector; The results of the covariate shift analysis are determined based on the second degree of difference.

[0182] In one possible implementation of this disclosure, the analysis unit 42 is further configured to: In response to a second dissimilarity level exceeding a preset second dissimilarity threshold, a covariate offset is determined to exist between the first and second datasets; or, In response to a second difference degree being less than or equal to a preset second difference degree threshold, it is determined that there is no covariate offset between the first dataset and the second dataset.

[0183] In one possible implementation of this disclosure, the performance analysis results include the reasons for the performance degradation of the target model, and the determining unit 43 is further configured to: In response to the model shift analysis showing a model shift between the first and second datasets, and the covariate shift analysis showing no covariate shift between the first and second datasets, the performance degradation is determined to be due to model shift; or, If the model offset analysis results show no model offset between the first and second datasets, and the covariate offset analysis results show a covariate offset between the first and second datasets, then the performance degradation is determined to be due to covariate offset; or, In response to the model offset analysis results showing a model offset between the first and second datasets, and the covariate offset analysis results showing a covariate offset between the first and second datasets, the performance degradation was determined to be caused by model offset and covariate offset.

[0184] Since the apparatus provided in this embodiment corresponds to the methods provided in the above embodiments, the implementation of the methods is also applicable to the apparatus provided in this embodiment, and will not be described in detail in this embodiment.

[0185] The methods and apparatus provided in the embodiments of this application have been described above. To implement the functions of the methods provided in the embodiments of this application, the electronic device may include a hardware structure and software modules, and may implement the above functions in the form of a hardware structure, software modules, or a hardware structure plus software modules. One of the above functions may be executed in the form of a hardware structure, software modules, or a hardware structure plus software modules.

[0186] Figure 4 This is a block diagram illustrating an electronic device 500 for implementing an analysis method for the performance of the above-described model, according to an exemplary embodiment. For example, the electronic device 500 may be applied to servers, cloud environments, operational service platforms, various computer platforms, terminal systems, and web page systems.

[0187] Reference Figure 4 The electronic device 500 may include one or more of the following components: processing component 502, memory 504, power supply component 506, multimedia component 508, audio component 510, input / output (I / O) interface 512, sensor component 514, and communication component 516.

[0188] Processing component 502 typically controls the overall operation of electronic device 500, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 502 may include one or more modules to facilitate interaction between processing component 502 and other components. For example, processing component 502 may include a multimedia module to facilitate interaction between multimedia component 508 and processing component 502.

[0189] Memory 504 is configured to store various types of data to support the operation of electronic device 500. Examples of this data include instructions for any application or method operating on electronic device 500, contact data, phonebook data, messages, pictures, videos, etc. Memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0190] Power supply component 506 provides power to various components of electronic device 500. Power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 500.

[0191] Multimedia component 508 includes a screen that provides an output interface between electronic device 500 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When electronic device 500 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0192] Audio component 510 is configured to output and / or input audio signals. For example, audio component 510 includes a microphone (MIC) configured to receive external audio signals when electronic device 500 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 504 or transmitted via communication component 516. In some embodiments, audio component 510 also includes a speaker for outputting audio signals.

[0193] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0194] Sensor assembly 514 includes one or more sensors for providing state assessments of various aspects of electronic device 500. For example, sensor assembly 514 may detect the on / off state of electronic device 500, the relative positioning of components such as the display and keypad of electronic device 500, changes in position of electronic device 500 or a component of electronic device 500, the presence or absence of user contact with electronic device 500, orientation or acceleration / deceleration of electronic device 500, and temperature changes of electronic device 500. Sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 514 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0195] Communication component 516 is configured to facilitate wired or wireless communication between electronic device 500 and other devices. Electronic device 500 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio), or combinations thereof. In one exemplary embodiment, communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 516 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0196] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0197] Embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in the above embodiments of this disclosure.

[0198] Embodiments of this disclosure also provide a computer program product comprising a computer program executable by a programmable device, the computer program having, when executed by the programmable device, the method described in the above embodiments of this disclosure.

[0199] For cases where electronic devices can be chips or chip systems, see [link to relevant documentation]. Figure 5 The diagram shows the structure of the chip. Figure 5 The chip shown includes a processor 601 and an interface 602. There can be one or more processors 601, and multiple interfaces 602.

[0200] Optionally, the chip also includes a memory 603 for storing necessary computer programs and data.

[0201] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

[0202] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0203] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0204] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0205] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (control method), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic device, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0206] It should be understood that various parts of the embodiments of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0207] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0208] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.

[0209] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for analyzing model performance, characterized in that, The method includes: Obtain a first dataset and a second dataset; the first dataset is the training dataset of the target model during the training phase, and the second dataset is the inference dataset of the target model during the inference phase. Model shift analysis and covariate shift analysis are performed on the first dataset and the second dataset to obtain model shift analysis results and covariate shift analysis results. The model shift analysis results are used to characterize the difference between the conditional distribution of the first dataset and the conditional distribution of the second dataset, where the conditional distribution is the mapping relationship between the model input data and the reference data corresponding to the model input data. The covariate shift analysis results are used to characterize the difference between the model input data of the first dataset and the model input data of the second dataset. Based on the model offset analysis results and the covariate offset analysis results, the performance analysis results of the target model are determined.

2. The method according to claim 1, characterized in that, Perform model offset analysis on the first dataset and the second dataset, including any of the following: Using the model input data as a benchmark, the first similarity between the first dataset and the second dataset on the reference data is analyzed, and the model offset analysis result is determined based on the first similarity. An initial model is trained using the first dataset and the second dataset respectively to obtain training results, and the model offset analysis results are determined based on the training results. Using the reference data as a benchmark, the first difference between the first dataset and the second dataset on the model input data is analyzed, and the model offset analysis result is determined based on the first difference.

3. The method according to claim 2, characterized in that, The step of analyzing the first similarity between the first dataset and the second dataset on the reference data, based on the model input data, includes: Based on the second similarity between the model input data in the first dataset and the first target model input data, a preset number of first model input data in the first dataset are determined; the first target model input data is at least one model input data in the second dataset. A similarity analysis is performed on the reference data corresponding to the preset number of first model input data and the reference data corresponding to the first target model input data to obtain the first similarity.

4. The method according to claim 3, characterized in that, The step of determining a preset number of first model input data in the first dataset based on the second similarity between the model input data in the first dataset and the first target model input data includes: According to the descending order of the second similarity, the preset number of first model input data are selected from the model input data in the first dataset.

5. The method according to claim 3, characterized in that, The step of performing similarity analysis on the reference data corresponding to the preset number of first model input data and the reference data corresponding to the first target model input data to obtain the first similarity includes: In response to the preset quantity being one, a similarity analysis is performed on the reference data corresponding to the first model input data and the reference data corresponding to the first target model input data to obtain the first similarity; or, In response to the preset quantity being multiple, statistical values ​​are calculated for the reference data corresponding to the preset quantity of first model input data to obtain statistical reference data; A similarity analysis is performed on the statistical reference data and the reference data corresponding to the input data of the first target model to obtain the first similarity.

6. The method according to claim 2, characterized in that, Determining the model offset analysis result based on the first similarity includes: In response to the first similarity being greater than a preset similarity threshold, it is determined that there is a model offset between the first dataset and the second dataset; or, In response to the first similarity being less than or equal to the preset similarity threshold, it is determined that there is no model offset between the first dataset and the second dataset.

7. The method according to claim 3, characterized in that, Determining the model offset analysis result based on the first similarity includes: Determine the second target model input data in the first dataset; the second target model input data is the model input data corresponding to the maximum value in the second similarity; Based on the third similarity between the remaining model input data in the first dataset and the second target model input data, the preset number of second model input data in the first dataset are determined. A fourth similarity score is obtained by performing a similarity analysis on the reference data corresponding to the preset number of second model input data and the reference data corresponding to the first target model input data; In response to the first similarity being less than the fourth similarity, a model offset is determined to exist between the first dataset and the second dataset.

8. The method according to claim 2, characterized in that, The step of training an initial model using the first dataset and the second dataset respectively, obtaining training results, and determining the model offset analysis results based on the training results includes: The initial model is trained using a first subset of data from the first dataset to obtain a first target model; and the initial model is trained using a second subset of data from the second dataset to obtain a second target model. The first target model is tested using a third subset of data in the first dataset to obtain a first test result; and the second target model is tested using a fourth subset of data in the second dataset to obtain a second test result; the training result includes the first test result and the second test result. The model offset analysis result is determined based on the consistency between the first test result and the second test result.

9. The method according to claim 8, characterized in that, The step of determining the model offset analysis result based on the consistency between the first test result and the second test result includes: In response to the consistency between the first test result and the second test result, it is determined that there is no model offset between the first dataset and the second dataset; or, In response to the lack of consistency between the first test result and the second test result, it is determined that there is a model offset between the first dataset and the second dataset.

10. The method according to claim 2, characterized in that, The step of analyzing the first difference between the first dataset and the second dataset on the model input data, based on the reference data, includes: Determine a first reference data in the first dataset and a second reference data in the second dataset; the similarity between the first reference data and the second reference data is greater than a preset similarity threshold; The first degree of difference is determined based on the distribution of the model input data of the first reference data and the distribution of the model input data of the second reference data.

11. The method according to claim 2, characterized in that, Determining the model offset analysis result based on the first difference includes: In response to the first difference being greater than a preset first difference threshold, it is determined that there is a model offset between the first dataset and the second dataset; or, In response to the first difference degree being less than or equal to the preset first difference degree threshold, it is determined that there is no model offset between the first dataset and the second dataset.

12. The method according to claim 1, characterized in that, Perform covariate offset analysis on the first dataset and the second dataset, including any of the following: Using the first dataset and the second dataset, a classification model is trained, and the covariate shift analysis results are determined based on the training results of the classification model. The covariate shift analysis results are determined based on the first performance evaluation index of the target model processing the first dataset, the second performance evaluation index of the target model processing the second dataset, and the distribution density ratio of the model input data between the first dataset and the second dataset. The covariate offset analysis results are determined based on the distribution of model input data in the first dataset and the distribution of model input data in the second dataset.

13. The method according to claim 12, characterized in that, The step of training a classification model using the first dataset and the second dataset, and determining the covariate shift analysis results based on the training results of the classification model, includes: The initial classification model is trained using the fifth data subset from the first dataset and the sixth data subset from the second dataset to obtain the classification model. The classification model is tested using the seventh data subset in the first dataset and the eighth data subset in the second dataset to obtain a third test result; the classification model is used to determine whether the data input into the classification model belongs to the first dataset, or whether the classification model is used to determine whether the data input into the classification model belongs to the second dataset. The covariate offset analysis results are determined based on the third performance evaluation index of the third test results.

14. The method according to claim 13, characterized in that, The determination of the covariate shift analysis result based on the third performance evaluation index of the third test result includes: In response to the third performance evaluation metric meeting a preset condition, it is determined that there is a covariate bias between the first dataset and the second dataset; or, In response to the fact that the third performance evaluation metric does not meet the preset conditions, it is determined that there is no covariate offset between the first dataset and the second dataset.

15. The method according to claim 12, characterized in that, The step of determining the covariate shift analysis result based on the first performance evaluation index of the target model processing the first dataset, the second performance evaluation index of the target model processing the second dataset, and the distribution density ratio of the model input data between the first dataset and the second dataset includes: The product of the first performance evaluation index and the distribution density ratio is calculated to obtain the first performance evaluation index calculated by the product. The covariate offset analysis result is determined based on the consistency between the first performance evaluation index and the second performance evaluation index calculated by the product.

16. The method according to claim 15, characterized in that, The determination of the covariate shift analysis result based on the consistency between the first performance evaluation index calculated based on the product and the second performance evaluation index includes: In response to the consistency between the first performance evaluation metric and the second performance evaluation metric in the product calculation, it is determined that there is a covariate offset between the first dataset and the second dataset; or, In response to the lack of consistency between the first performance evaluation metric and the second performance evaluation metric in the product calculation, it is determined that there is no covariate offset between the first dataset and the second dataset.

17. The method according to claim 12, characterized in that, The step of determining the covariate shift analysis results based on the distribution of model input data in the first dataset and the distribution of model input data in the second dataset includes: The model input data in the first dataset is mapped to a preset feature space to obtain a first mapping vector; and, The model input data in the second dataset is mapped to a preset feature space to obtain a second mapping vector; The covariate offset analysis results are determined based on the distribution of the first mapping vector and the distribution of the second mapping vector.

18. The method according to claim 17, characterized in that, The step of determining the covariate offset analysis result based on the distribution of the first mapping vector and the distribution of the second mapping vector includes: Based on the distribution of the first mapping vector and the distribution of the second mapping vector, a second degree of difference between the first mapping vector and the second mapping vector is determined; The covariate offset analysis results are determined based on the second degree of difference.

19. The method according to claim 18, characterized in that, The step of determining the covariate shift analysis result based on the second degree of difference includes: In response to the second difference being greater than a preset second difference threshold, it is determined that there is a covariate offset between the first dataset and the second dataset; or, In response to the second difference degree being less than or equal to the preset second difference degree threshold, it is determined that there is no covariate offset between the first dataset and the second dataset.

20. The method according to claim 1, characterized in that, The performance analysis results include the reasons for the performance degradation of the target model. Determining the performance analysis results of the target model based on the model shift analysis results and the covariate shift analysis results includes: If the model offset analysis result indicates that there is a model offset between the first dataset and the second dataset, and the covariate offset analysis result indicates that there is no covariate offset between the first dataset and the second dataset, then the performance degradation is determined to be due to model offset; or, If the model offset analysis result indicates that there is no model offset between the first dataset and the second dataset, and the covariate offset analysis result indicates that there is a covariate offset between the first dataset and the second dataset, then the performance degradation is determined to be caused by covariate offset; or, In response to the model offset analysis result indicating that there is a model offset between the first dataset and the second dataset, and the covariate offset analysis result indicating that there is a covariate offset between the first dataset and the second dataset, the performance degradation is determined to be caused by model offset and covariate offset.

21. A device for analyzing model performance, characterized in that, The device includes: The acquisition unit is used to acquire a first dataset and a second dataset; the first dataset is the training dataset of the target model during the training phase, and the second dataset is the inference dataset of the target model during the inference phase. The analysis unit is used to perform model shift analysis and covariate shift analysis on the first dataset and the second dataset to obtain model shift analysis results and covariate shift analysis results. The model shift analysis results are used to characterize the difference between the conditional distribution of the first dataset and the conditional distribution of the second dataset, where the conditional distribution is the mapping relationship between the model input data and the reference data corresponding to the model input data. The covariate shift analysis results are used to characterize the difference between the model input data of the first dataset and the model input data of the second dataset. The determining unit is used to determine the performance analysis results of the target model based on the model offset analysis results and the covariate offset analysis results.

22. The apparatus according to claim 21, characterized in that, The analysis unit is also used for: Using the model input data as a benchmark, the first similarity between the first dataset and the second dataset on the reference data is analyzed, and the model offset analysis result is determined based on the first similarity. An initial model is trained using the first dataset and the second dataset respectively to obtain training results, and the model offset analysis results are determined based on the training results. Using the reference data as a benchmark, the first difference between the first dataset and the second dataset on the model input data is analyzed, and the model offset analysis result is determined based on the first difference.

23. The apparatus according to claim 21, characterized in that, The analysis unit is also used for: Using the first dataset and the second dataset, a classification model is trained, and the covariate shift analysis results are determined based on the training results of the classification model. The covariate shift analysis results are determined based on the first performance evaluation index of the target model processing the first dataset, the second performance evaluation index of the target model processing the second dataset, and the distribution density ratio of the model input data between the first dataset and the second dataset. The covariate offset analysis results are determined based on the distribution of model input data in the first dataset and the distribution of model input data in the second dataset.

24. The apparatus according to claim 21, characterized in that, The performance analysis results include the reasons for the performance degradation of the target model, and the determining unit is further configured to: In response to the model offset analysis result indicating that there is a model offset between the first dataset and the second dataset, and the covariate offset analysis result indicating that there is no covariate offset between the first dataset and the second dataset, the performance degradation is determined to be caused by model offset. or, In response to the model offset analysis result indicating that there is no model offset between the first dataset and the second dataset, and the covariate offset analysis result indicating that there is a covariate offset between the first dataset and the second dataset, the cause of the performance degradation is determined to be covariate offset; or, In response to the model offset analysis result indicating that there is a model offset between the first dataset and the second dataset, and the covariate offset analysis result indicating that there is a covariate offset between the first dataset and the second dataset, the performance degradation is determined to be caused by model offset and covariate offset.

25. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-20.

26. A computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-20.

27. A program product, characterized in that, Includes computer instructions for causing a computer to perform the method of any one of claims 1-20.

Images