Method and device for training accuracy evaluation model, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses a method and device for training an accuracy evaluation model, a storage medium and electronic equipment, and the method comprises the steps: obtaining a first sample data set corresponding to an original data set according to the original data set corresponding to a target model; obtaining first output distribution of the target model about the original data set and second output distribution of the target model about the first sample data set; obtaining first distribution offset information between the second output distribution and the first output distribution, and determining first prediction accuracy of the target model about the first sample data set according to label information corresponding to the original data set; an accuracy evaluation model is trained according to the first distribution offset information and the first prediction accuracy, a trained accuracy evaluation model is obtained, and the first prediction accuracy serves as label information corresponding to the first distribution offset information.

Description

Technical Field

[0001] The present invention relates to computer technology, and in particular to a method, device, storage medium and electronic device for training an accuracy assessment model. Background Art

[0002] In smart credential products and risk control systems for various businesses, various algorithm models such as machine learning, natural language processing, image recognition, and graph networks play a vital role. As the scale and complexity of models continue to increase, it is necessary not only to address business risks, but also to ensure the smooth operation of the models and avoid various potential risks caused by model performance degradation. This requires that the effects of various models need to be monitored to ensure that the model effect evaluation can be fed back in a timely manner.

[0003] However, it is not easy to evaluate the effectiveness of a model. It often requires relying on manually annotated data labels, which means higher labor costs, time costs, and money costs. When there are many models, the cost of evaluating the model effects increases exponentially, and manual labeling has serious lags and cannot meet real-time requirements. Summary of the invention

[0004] The purpose of the embodiments of this specification is to provide a method, device, storage medium and electronic device for training an accuracy assessment model.

[0005] The embodiments of this specification provide a method for training an accuracy evaluation model, and propose a method for evaluating the performance of a trained model on an unlabeled test set. By training an accuracy evaluation model to predict the accuracy of the model output corresponding to the unlabeled test set, it can be used to evaluate the current performance of the trained model, solve the real-time effect evaluation and monitoring problem of the online model, and construct a regression model between the distribution difference and the model effect accuracy by learning the distribution difference between different data sets as features. By using the distribution difference between the unlabeled test data and the original model training data as input, the accuracy evaluation model is used to directly predict the model accuracy of the model on the test set, thereby achieving the purpose of real-time evaluation of the current performance of the model, which can reduce the human cost, time cost, and money cost of model performance evaluation, and can meet the real-time requirements. The method includes:

[0006] According to the original data set corresponding to the target model, a first sample data set corresponding to the original data set is obtained;

[0007] Obtaining a first output distribution of the target model with respect to the original data set and a second output distribution of the target model with respect to the first sample data set;

[0008] Obtaining first distribution offset information between the second output distribution and the first output distribution, and determining a first prediction accuracy of the target model with respect to the first sample data set according to label information corresponding to the original data set;

[0009] An accuracy assessment model is trained according to the first distribution shift information and the first prediction accuracy to obtain a trained accuracy assessment model, wherein the first prediction accuracy is used as label information corresponding to the first distribution shift information, and the accuracy assessment model is used to output a second prediction accuracy of the target model with respect to the test data set according to the second distribution shift information corresponding to the input unlabeled test data set.

[0010] Furthermore, obtaining a first sample data set corresponding to the original data set according to the original data set corresponding to the target model includes:

[0011] An original data set corresponding to the target model is randomly sampled to obtain a first sample data set corresponding to the original data set.

[0012] Furthermore, randomly extracting the original data set corresponding to the target model to obtain a first sample data set corresponding to the original data set includes:

[0013] Randomly extracting the original data set corresponding to the target model to obtain a corresponding second sample data set;

[0014] Transform the second sample data set according to a preset transformation rule to obtain a first sample data set corresponding to the original data set;

[0015] The step of determining the first prediction accuracy of the target model with respect to the first sample data set according to the label information corresponding to the original data set includes:

[0016] A first prediction accuracy of the target model with respect to the first sample data set is determined according to the label information corresponding to the original data set and the preset transformation rule.

[0017] Further, the obtaining first distribution offset information between the second output distribution and the first output distribution includes:

[0018] Performing a binning operation on the first output distribution to obtain corresponding first binning sequence information, wherein the first binning sequence information includes a plurality of first bins, each of which includes at least one first output;

[0019] Performing a binning operation on the second output distribution to obtain corresponding second binning sequence information, wherein the second binning sequence information includes a plurality of second bins, each second bin includes at least one second output, and the number of the first bins is the same as the number of the second bins;

[0020] Constructing third distribution information according to a ratio feature of the first binning sequence information relative to the first output distribution, and constructing fourth distribution information according to a ratio feature of the second binning sequence information relative to the second output distribution;

[0021] First distribution offset information between the second output distribution and the first output distribution is determined according to the third distribution information and the fourth distribution information.

[0022] Further, before obtaining the first distribution offset information between the second output distribution and the first output distribution, the method further includes:

[0023] A sorting operation is performed on the first output distribution and the second output distribution respectively.

[0024] Further, performing a binning operation on the first output distribution to obtain corresponding first binning sequence information includes:

[0025] An equal-frequency binning operation is performed on the first output distribution to obtain corresponding first binning sequence information, wherein each first bin contains the same number of first outputs.

[0026] Further, performing a binning operation on the first output distribution to obtain corresponding first binning sequence information includes:

[0027] An equal-width binning operation is performed on the first output distribution to obtain corresponding first binning sequence information, wherein the length of the first output value interval corresponding to each first bin is the same.

[0028] Further, determining first distribution offset information between the second output distribution and the first output distribution according to the third distribution information and the fourth distribution information includes:

[0029] Distribution offset information between the second output distribution and the first output distribution is determined based on the difference between each third data in the third distribution information and the fourth data in the fourth distribution information corresponding to the same position as the third data, and the ratio of the third data to the fourth data.

[0030] Furthermore, constructing third distribution information according to a ratio feature of the first binning sequence information relative to the first output distribution includes:

[0031] The third distribution information is constructed according to a ratio of the first output quantity in each first bin in the first bin sequence information to the first output quantity in the first output distribution.

[0032] Further, obtaining a first output distribution of the target model with respect to the original data set and a second output distribution of the target model with respect to the first sample data set includes:

[0033] A first output distribution consisting of a first feature extracted by the target model for the original data set and a second output distribution consisting of a second feature extracted by the target model for the first sample data set are obtained.

[0034] Furthermore, both the first feature and the second feature are one-dimensional features that have been normalized and reduced in dimension.

[0035] Further, obtaining a first output distribution of the target model with respect to the original data set and a second output distribution of the target model with respect to the first sample data set includes:

[0036] A first output distribution composed of a first confidence level predicted by the target model for the original data set and a second output distribution composed of a second confidence level predicted by the target model for the first sample data set are obtained.

[0037] Furthermore, the method further comprises:

[0038] Acquire second distribution offset information between a third output distribution of the target model with respect to the test data set and the first output distribution;

[0039] The second distribution shift information is input into the trained accuracy evaluation model to obtain the output second prediction accuracy of the target model with respect to the test data set.

[0040] The present specification also provides a method for predicting accuracy, the method comprising:

[0041] Obtaining second distribution offset information between a third output distribution of the target model with respect to the unlabeled test data set and a first output distribution of the target model with respect to the original data set;

[0042] The second distribution shift information is input into a trained accuracy evaluation model to obtain a second prediction accuracy of the target model with respect to the test data set as an output.

[0043] The embodiment of this specification also provides a device for training an accuracy assessment model, including:

[0044] A sample data set acquisition module, used to acquire a first sample data set corresponding to the original data set according to the original data set corresponding to the target model;

[0045] An output distribution obtaining module, used to obtain a first output distribution of the target model with respect to the original data set and a second output distribution of the target model with respect to the first sample data set;

[0046] a first distribution offset obtaining module, configured to obtain first distribution offset information between the second output distribution and the first output distribution, and determine a first prediction accuracy of the target model with respect to the first sample data set according to label information corresponding to the original data set;

[0047] A model training module is used to train an accuracy assessment model according to the first distribution shift information and the first prediction accuracy to obtain a trained accuracy assessment model, wherein the first prediction accuracy is used as label information corresponding to the first distribution shift information, and the accuracy assessment model is used to output the second prediction accuracy of the target model with respect to the test data set according to the second distribution shift information corresponding to the input unlabeled test data set.

[0048] The embodiment of this specification also provides a device for predicting accuracy, including:

[0049] A second distribution shift obtaining module, used to obtain second distribution shift information between a third output distribution of the target model with respect to the unlabeled test data set and a first output distribution of the target model with respect to the original data set;

[0050] The model prediction module is used to input the second distribution shift information into the trained accuracy evaluation model to obtain the output second prediction accuracy of the target model with respect to the test data set.

[0051] The embodiments of the present specification also provide a storage medium, wherein the storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the above method.

[0052] An embodiment of the present specification also provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above method.

[0053] In an embodiment of the present specification, a method for evaluating the performance of a trained model on an unlabeled test set is proposed. By training an accuracy evaluation model to predict the accuracy of the model output corresponding to the unlabeled test set, it can be used to evaluate the current performance of the trained model, solving the problem of real-time effect evaluation and monitoring of the online model. By learning the distribution differences between different data sets as features, a regression model between the distribution differences and the model effect accuracy is constructed. The distribution differences between the unlabeled test data and the original model training data are used as input, and the accuracy evaluation model is used to directly predict the model accuracy of the model on the test set, thereby achieving the purpose of real-time evaluation of the current performance of the model, which can reduce the manpower cost, time cost, and money cost of model performance evaluation, and can meet the real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flowchart of a method for training an accuracy assessment model provided in an embodiment of this specification;

[0055] Figure 2 A flowchart of a method for predicting accuracy provided in an embodiment of this specification;

[0056] Figure 3 A flowchart of a method for training an accuracy assessment model provided as an example in an embodiment of this specification;

[0057] Figure 4 A flowchart of a method for predicting accuracy of an example provided in an embodiment of this specification;

[0058] Figure 5 A schematic diagram of the structure of a device for training an accuracy assessment model provided in an embodiment of this specification;

[0059] Figure 6 A schematic diagram of a device for predicting accuracy provided in an embodiment of this specification;

[0060] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0062] See also Figure 1 , is a flow chart of a method for training an accuracy assessment model provided in an embodiment of this specification. In an embodiment of this specification, the method for training an accuracy assessment model is applied to a device for training an accuracy assessment model (hereinafter referred to as "accuracy assessment model training device") or an electronic device equipped with an accuracy assessment model training device. Figure 1 The process shown is described in detail, and the method for training the accuracy evaluation model may specifically include the following steps:

[0063] S102: Obtain a first sample data set corresponding to the original data set according to the original data set corresponding to the target model.

[0064] In some embodiments, the target model refers to the model of the current performance of the model to be evaluated. The target model includes but is not limited to various types of deep learning models, such as text models (for example, document recognition models), image models (for example, face recognition models), multimodal models (for example, identity recognition models), etc. This specification does not limit this. It should be noted that the above-mentioned target models are only examples and not limitations. Those skilled in the art should understand that any type of target model can be included in the scope of protection of this specification.

[0065] In some embodiments, the target model is optimized and trained on the original data set, and this process produces a trained target model. The original data set may include only a labeled training data set, or may also include both a labeled training data set and a labeled validation data set.

[0066] In some embodiments, multiple first sample data sets can be obtained by randomly sampling the original data set multiple times, wherein the number of data extracted each time may be fixed or not fixed, and the number of data in the first sample data set is less than the number of data in the original data set. In some embodiments, multiple first sample data sets can be obtained by transforming each data in the original data set according to multiple different preset transformation rules, and here, this example embodiment does not specifically limit the specific data change method. In some embodiments, the original data set can also be randomly sampled to obtain a second sample data set, and then each data in the second sample data set is changed according to the preset change rule to obtain the first sample data set. In some embodiments, the number of first sample data sets obtained according to the original data set corresponding to the target model can be arbitrary, and this specification does not limit this.

[0067] S104, obtaining a first output distribution of the target model with respect to the original data set and a second output distribution of the target model with respect to the first sample data set.

[0068] In some embodiments, the first output distribution is used to characterize the distribution of specific output information of the target model with respect to the original data set, and the first output distribution is a set consisting of specific output information of the target model with respect to each data in the original data set, and the second output distribution is used to characterize the distribution of specific output information of the target model with respect to the first sample data set, and the second output distribution is a set consisting of specific output information of the target model with respect to each data in the first sample data set. In some embodiments, the specific output includes but is not limited to any one of the output result output by the target model for the input data (i.e., the data in the original data set or the data in the first sample data set), the feature information extracted by the target model for the input data, and the confidence information predicted by the target model for the input data. In some embodiments, the number of data in the first output distribution is the same as the number of data in the second output distribution. In some embodiments, the number of data in the second output distribution is less than the number of data in the first output distribution.

[0069] S106, obtaining first distribution offset information between the second output distribution and the first output distribution, and determining a first prediction accuracy of the target model with respect to the first sample data set according to label information corresponding to the original data set.

[0070] In some embodiments, the distribution offset information is used to characterize the degree of difference between two output distributions, i.e., the data distribution conditions of two sets, by numerical values. Any type of distance metric function (e.g., Jensen-Shannon divergence, total variation distance (TVD), Kullback-Leibler (KL) divergence, etc.) can be used to calculate the distribution offset information between the two output distributions. For example, the distribution offset information between the two output distributions can be obtained by performing KL divergence calculation on the two output distributions. For example, the divergence of the second output distribution relative to the first output distribution can be used as the first distribution offset information between the second output distribution and the first output distribution, or the divergence of the first output distribution relative to the second output distribution can be used as the first distribution offset information between the second output distribution and the first output distribution. For another example, the divergence of the second output distribution relative to the first output distribution and the divergence of the first output distribution relative to the second output distribution can be added together to form the first distribution offset information between the second output distribution and the first output distribution.

[0071] In some embodiments, if the original data set and the first sample data set only involve an extraction relationship, since the original data set is labeled, that is, each data in the first sample data set has a corresponding label, according to the label corresponding to each data in the first sample data set and the model output result of the target model on the data, the prediction accuracy of the target model on each data in the first sample data set can be obtained, and the first prediction accuracy of the target model on the first sample data set can be further obtained (for example, the prediction accuracy corresponding to each data in the first sample data set is averaged), and the first prediction accuracy is used as the label information corresponding to the first distribution shift information. In some embodiments, if the original data set and the first sample data set involve a transformation relationship, for each data in the first sample data set, according to the label corresponding to the original data in the original data set and the model output result of the target model on the original data, the prediction accuracy of the target model on the original data can be obtained, and the prediction accuracy can be used as the prediction accuracy of the target model on the data in the first sample data set, and the first prediction accuracy of the target model on the first sample data set can be further obtained (for example, the prediction accuracy corresponding to each data in the first sample data set is averaged), and the first prediction accuracy can be used as the label information corresponding to the first distribution shift information.

[0072] S108, training an accuracy assessment model according to the first distribution shift information and the first prediction accuracy to obtain a trained accuracy assessment model, wherein the first prediction accuracy is used as label information corresponding to the first distribution shift information, and the accuracy assessment model is used to output a second prediction accuracy of the target model with respect to the test data set according to the second distribution shift information corresponding to the input unlabeled test data set.

[0073] In some embodiments, the first distribution shift information between the second output distribution corresponding to each first sample data set and the first output distribution corresponding to the original data set and the label information corresponding to the first distribution shift information (i.e., the first prediction accuracy of the target model with respect to the first sample data set) can be subjected to regression training, that is, a trained accuracy assessment model can be obtained by performing regression training on multiple first distribution shift information and the label information corresponding to each first distribution shift information, that is, a regression model between the distribution shift information and the model prediction accuracy is constructed. In some embodiments, the input of the accuracy assessment model is the distribution shift information between the output distribution of the target model with respect to a data set and the first output distribution of the target model with respect to the original data set, and the output of the accuracy assessment model is the prediction accuracy corresponding to the distribution shift information, that is, the output of the accuracy assessment model is the prediction accuracy of the target model with respect to the data set, and this is used to further determine the current performance of the target model.

[0074] In some embodiments, an unlabeled test data set is used to evaluate the current performance of a trained target model. First, a third output distribution of the target model with respect to the unlabeled test data set and a first output distribution of the target model with respect to the original data set are obtained, and then second distribution offset information between the third output distribution and the first output distribution is obtained. By inputting the second distribution offset information into the trained accuracy evaluation model, a second prediction accuracy of the target model output by the model with respect to the test data set can be obtained, and this can be used to further determine the current performance of the target model.

[0075] In an embodiment of the present specification, a method for evaluating the performance of a trained model on an unlabeled test set is proposed. By training an accuracy evaluation model to predict the accuracy of the model output corresponding to the unlabeled test set, it can be used to evaluate the current performance of the trained model, solving the problem of real-time effect evaluation and monitoring of the online model. By learning the distribution differences between different data sets as features, a regression model between the distribution differences and the model effect accuracy is constructed. The distribution differences between the unlabeled test data and the original model training data are used as input, and the accuracy evaluation model is used to directly predict the model accuracy of the model on the test set, thereby achieving the purpose of real-time evaluation of the current performance of the model, which can reduce the manpower cost, time cost, and money cost of model performance evaluation, and can meet the real-time requirements.

[0076] In some embodiments, obtaining the first sample data set corresponding to the original data set according to the original data set corresponding to the target model includes: randomly extracting the original data set corresponding to the target model to obtain the first sample data set corresponding to the original data set. In some embodiments, multiple first sample data sets can be obtained by randomly extracting the original data set multiple times, wherein the amount of data extracted each time may be fixed or not fixed, and the amount of data in the first sample data set is less than the amount of data in the original data set.

[0077] In some embodiments, the randomly extracting the original data set corresponding to the target model to obtain the first sample data set corresponding to the original data set includes: randomly extracting the original data set corresponding to the target model to obtain the corresponding second sample data set; transforming the second sample data set according to a preset transformation rule to obtain the first sample data set corresponding to the original data set; wherein, determining the first prediction accuracy of the target model with respect to the first sample data set according to the label information corresponding to the original data set includes: determining the first prediction accuracy of the target model with respect to the first sample data set according to the label information corresponding to the original data set and the preset transformation rule. In some embodiments, the original data set may be first randomly extracted to obtain the second sample data set, and then each data in the second sample data set may be changed according to the preset change rule to obtain the first sample data set. In some embodiments, for each data in the first sample data set, the original data corresponding to the data in the original data set can be determined based on a preset transformation rule, and then the prediction accuracy of the target model with respect to the original data is obtained based on the label corresponding to the original data and the model output result of the target model with respect to the original data, and the prediction accuracy is used as the prediction accuracy of the target model with respect to the data in the first sample data set, the first prediction accuracy of the target model with respect to the first sample data set, and the first prediction accuracy is used as the prediction accuracy corresponding to the first distribution offset information.

[0078] In some embodiments, obtaining the first distribution offset information between the second output distribution and the first output distribution includes: performing a binning operation on the first output distribution to obtain corresponding first binning sequence information, wherein the first binning sequence information includes multiple first bins, each first bin includes at least one first output; performing a binning operation on the second output distribution to obtain corresponding second binning sequence information, wherein the second binning sequence information includes multiple second bins, each second bin includes at least one second output, and the number of first bins is the same as the number of second bins; constructing third distribution information based on the proportional characteristics of the first binning sequence information relative to the first output distribution, and constructing fourth distribution information based on the proportional characteristics of the second binning sequence information relative to the second output distribution; determining the first distribution offset information between the second output distribution and the first output distribution based on the third distribution information and the fourth distribution information. In some embodiments, a binning operation is performed on the first output distribution to obtain corresponding first binning sequence information, the first binning sequence information is an ordered sequence consisting of a plurality of first bins, that is, the first output distribution is divided into a plurality of first bins according to a preset number, and each first bin includes at least one first output (i.e., data in the first output distribution), and a binning operation is performed on the second output distribution to obtain corresponding second binning sequence information, the second binning sequence information is an ordered sequence consisting of a plurality of second bins, that is, the second output distribution is divided into a plurality of second bins according to a preset number, and each second bin includes at least one second output (i.e., data in the second output distribution). In some embodiments, after performing the binning operation, it is necessary to make the number of first bins in the first binning sequence information the same as the number of second bins in the second binning sequence information. In some embodiments, there is no intersection between different first bins, and there is no intersection between different second bins.In some embodiments, each first bin is converted into a new distribution element according to the proportion feature of each first bin in the first bin sequence information relative to the first output distribution, so as to construct the third distribution information. For example, a new ordered sequence (i.e., the third distribution information) is constructed according to the proportion of the number of data in each first bin relative to the number of data in the first output distribution. For example, the first data in the third distribution information is the proportion of the number of data in the first bin in the first bin sequence information relative to the number of data in the first output distribution, the second data in the third distribution information is the proportion of the number of data in the second bin in the first bin sequence information relative to the number of data in the first output distribution, and so on. The last data in the third distribution information is the proportion of the number of data in the last bin in the first bin sequence information relative to the number of data in the first output distribution. For another example, the third distribution information is constructed according to the proportion of the average value of the data in each first bin relative to the average value of the data in the first output distribution. For another example, the third distribution information is constructed according to the proportion of the total number of data in each first bin relative to the total number of data in the first output distribution. In some embodiments, the method of constructing the fourth distribution information according to the proportional characteristics of the second binning sequence information relative to the second output distribution is similar to the method of constructing the third distribution information described above and will not be repeated here. Since the number of first bins in the first binning sequence information is the same as the number of second bins in the second binning sequence information, the third distribution information and the fourth distribution information have the same amount of data. In some embodiments, in the process of using a preset distance metric function to calculate the first distribution offset information between the second output distribution and the first output distribution, since the distance metric function requires that the two output distributions have the same number of data (for example, the KL divergence requires that the two distributions have the same number of data), when the number of data in the second output distribution is different from the number of data in the first output distribution, that is, when the number of data in the second output distribution is less than the number of data in the first output distribution, it is necessary to convert the first output distribution into a new third distribution information, and convert the second output distribution into a new fourth distribution information, the third distribution information and the fourth distribution information have the same number of data, and when calculating the KL divergence between the second output distribution and the first output distribution, the KL divergence between the second output distribution and the first output distribution is indirectly calculated through the third distribution information and the fourth distribution information, that is, two output distributions corresponding to different numbers of data (the first output distribution and the second output distribution) are converted into two distribution information corresponding to the same number of data (the third distribution information and the fourth distribution information) for distribution offset calculation.In some embodiments, there is no clear relative relationship between the first output distribution and the second output distribution, the distribution offset between the two is symmetrical, while the KL divergence is asymmetric (in the KL divergence, the KL divergence of distribution A with respect to distribution B is usually not equal to the KL divergence of distribution B with respect to distribution A). In order to achieve distance symmetry, the distribution offset is represented by the characteristic that the sum of the divergence of the first output distribution with respect to the second output distribution and the divergence of the second output distribution with respect to the first output distribution is equal to the sum of the divergence of the second output distribution with respect to the first output distribution and the divergence of the first output distribution with respect to the second output distribution. The divergence of the second output distribution with respect to the first output distribution and the divergence of the first output distribution with respect to the second output distribution are added as the first distribution offset information between the second output distribution and the first output distribution. The specific calculation process is as follows:

[0079] Dis(U r ,U s )=KL(U r ||U s )+KL(U s ||U r )

[0080] Among them, U r It refers to the first output distribution of the target model about the original data set, U s It refers to the second output distribution of the target model with respect to the first sample data set, Dis(U r ,U s ) refers to the first distribution offset information between the second output distribution and the first output distribution, KL(U r ||U s ) refers to the KL divergence of the first output distribution relative to the second output distribution, KL(U s ||U r ) refers to the KL divergence of the second output distribution with respect to the first output distribution.

[0081] In some embodiments, before obtaining the first distribution offset information between the second output distribution and the first output distribution, the method further includes: performing a sorting operation on the first output distribution and the second output distribution. In some embodiments, the first output distribution and the second output distribution are sorted in the order of the size of the data values ​​to obtain the sorted first output distribution and the sorted second output distribution, and then the first distribution offset information between the sorted second output distribution and the sorted first output distribution is obtained, for example, the first output distribution and the second output distribution are sorted in the order of the corresponding data values ​​from large to small, and for example, the first output distribution and the second output distribution are sorted in the order of the corresponding data values ​​from small to large.

[0082] In some embodiments, the performing of the binning operation on the first output distribution to obtain the corresponding first binning sequence information includes: performing an equal-frequency binning operation on the first output distribution to obtain the corresponding first binning sequence information, wherein each first bin contains the same number of first outputs. In some embodiments, performing an equal-frequency binning operation on the first output distribution to obtain the corresponding first binning sequence information, the first binning sequence information is an ordered sequence composed of a plurality of first bins, that is, the first output distribution is divided into a plurality of first bins with equal frequency according to a preset number, each first bin contains the same number of first outputs (i.e., data in the first output distribution), wherein if the number of first outputs contained in a first bin is less than the number of first outputs contained in other first bins, the first bin is discarded. In some embodiments, the equal-frequency binning operation performed on the second output distribution is similar to the equal-frequency binning operation performed on the first output distribution described above, and will not be described in detail here.

[0083] In some embodiments, the performing of the binning operation on the first output distribution to obtain the corresponding first binning sequence information includes: performing an equal-width binning operation on the first output distribution to obtain the corresponding first binning sequence information, wherein the length of the first output value interval corresponding to each first bin is the same. In some embodiments, the performing of the equal-width binning operation on the first output distribution to obtain the corresponding first binning sequence information, the first binning sequence information is an ordered sequence composed of a plurality of first bins, that is, the first output distribution is divided into a plurality of first bins with equal width, the interval length corresponding to the first output value interval corresponding to each first bin is the same, and the first output value interval corresponding to each first bin is determined according to the first output with the largest value (that is, the data in the first output distribution) and the first output with the smallest value in the first bin, that is, the two endpoints of the first output value interval are the value of the first output with the largest value in the first bin and the value of the first output with the smallest value in the first bin, respectively, wherein, if the interval length of the first output value interval corresponding to a first bin is less than the interval length of the first output value interval corresponding to other first bins, then the first bin is discarded. In some embodiments, the equal-width binning operation performed on the second output distribution is similar to the equal-width binning operation performed on the first output distribution described above, and will not be described in detail here.

[0084] In some embodiments, the determining of the first distribution offset information between the second output distribution and the first output distribution according to the third distribution information and the fourth distribution information includes: determining the distribution offset information between the second output distribution and the first output distribution according to the difference between each third data in the third distribution information and the fourth data in the fourth distribution information corresponding to the same position as the third data and the ratio of the third data to the fourth data. In some embodiments, the divergence of the second output distribution relative to the first output distribution and the divergence of the first output distribution relative to the second output distribution are added to calculate the first distribution offset information between the second output distribution and the first output distribution. In the calculation process, the first output distribution is replaced by the third distribution information, and the second output distribution is replaced by the fourth distribution information to perform the calculation, so as to calculate the first distribution offset information between the second output distribution and the first output distribution based on the difference between each third data in the third distribution information and the fourth data in the fourth distribution information corresponding to the same position as the third data and the ratio of the third data to the fourth data. The specific calculation process is as follows:

[0085] Dis(U r ,U s )=KL(U r ||U s )+KL(U s ||U r )

[0086] =∑U r log(U r / U s )+∑U s log(U s / U r )

[0087] = >∑U a log(U a / U b )+∑U b log(U b / U a )

[0088] =∑(U a -U b )log(U a / U b )

[0089] Among them, U r It refers to the first output distribution of the target model about the original data set, U s It refers to the second output distribution of the target model with respect to the first sample data set, Dis(U r ,U s) refers to the first distribution offset information between the second output distribution and the first output distribution, KL(U r ||U s ) refers to the KL divergence of the first output distribution relative to the second output distribution, KL(U s ||U r ) refers to the KL divergence of the second output distribution relative to the first output distribution, U a It refers to the third distribution information constructed by transforming the first output distribution, U b It refers to the fourth distribution information constructed by transforming the second output distribution.

[0090] In some embodiments, the third distribution information is constructed according to the ratio feature of the first bin sequence information relative to the first output distribution, including: constructing the third distribution information according to the ratio of the first output quantity in each first bin in the first bin sequence information to the first output quantity in the first output distribution. In some embodiments, each first bin is converted into a new distribution element according to the ratio of the number of data in each first bin (i.e., the first output quantity) relative to the number of data in the first output distribution (i.e., the first output quantity), so as to construct a new ordered sequence, i.e., the third distribution information. For example, the first data in the third distribution information is the ratio of the number of data in the first bin in the first bin sequence information relative to the number of data in the first output distribution, the second data in the third distribution information is the ratio of the number of data in the second bin in the first bin sequence information relative to the number of data in the first output distribution, and so on, the last data in the third distribution information is the ratio of the number of data in the last bin in the first bin sequence information relative to the number of data in the first output distribution. In some embodiments, the method of constructing the fourth distribution information according to the ratio feature of the second bin sequence information relative to the second output distribution is similar to the method of constructing the third distribution information described above, and will not be repeated here.

[0091] In some embodiments, the obtaining of the first output distribution of the target model with respect to the original data set and the second output distribution of the target model with respect to the first sample data set includes: obtaining a first output distribution consisting of the first features extracted by the target model for the original data set and a second output distribution consisting of the second features extracted by the target model for the first sample data set. In some embodiments, the first output distribution of the target model with respect to the original data set refers to a set consisting of the first features extracted by the target model for each data in the original data set, and each data in the set is a first feature corresponding to a data in the original data set; the second output distribution of the target model with respect to the first sample data set refers to a set consisting of the second features extracted by the target model for each data in the first sample data set, and each data in the set is a second feature corresponding to a data in the first sample data set.

[0092] In some embodiments, the first feature and the second feature are both one-dimensional features that have been normalized and reduced in dimension. In some embodiments, the first feature and the second feature extracted by the target model for the original data set and the first sample data set are multidimensional features, and the first feature and the second feature need to be normalized and reduced in dimension. For example, the features of the first dimension in the multidimensional features are normalized, and then the features of the second dimension are reduced in dimension. The specific calculation process is as follows:

[0093] F=F–MEAN([F] 0 ) / STD([F] 0 )

[0094] F=MEAN([F] 1 )+MAX([F] 1 )+MIN([F] 1 )

[0095] Where F refers to the first feature extracted by the target model for the original data set or the second feature extracted by the target model for the first sample data set, MEAN, STD, MAX and MIN represent the mean, standard deviation, maximum and minimum values ​​respectively, [F] 0 refers to the features of the first dimension in the first feature or the features of the first dimension in the second feature, [F] 1 It refers to the feature of the second dimension in the first feature or the feature of the second dimension in the second feature.

[0096] In some embodiments, the obtaining of the first output distribution of the target model with respect to the original data set and the second output distribution of the target model with respect to the first sample data set includes: obtaining a first output distribution consisting of a first confidence predicted by the target model for the original data set and a second output distribution of a second confidence predicted by the target model for the first sample data set. In some embodiments, the first output distribution of the target model with respect to the original data set refers to a set consisting of a first confidence predicted by the target model for each data in the original data set, each data in the set being a first confidence corresponding to a data in the original data set, and the second output distribution of the target model with respect to the first sample data set refers to a set consisting of a second confidence predicted by the target model for each data in the first sample data set, each data in the set being a second confidence corresponding to a data in the first sample data set.

[0097] In some embodiments, the method further includes: obtaining second distribution offset information between the third output distribution of the target model with respect to the test data set and the first output distribution; inputting the second distribution offset information into the trained accuracy assessment model to obtain the second prediction accuracy of the output target model with respect to the test data set. In some embodiments, an unlabeled test data set is used to evaluate the current performance of the trained target model, first obtaining the third output distribution of the target model with respect to the test data set and the first output distribution of the target model with respect to the original data set, and then obtaining the second distribution offset information between the third output distribution and the first output distribution. By inputting the second distribution offset information into the trained accuracy assessment model, the second prediction accuracy of the target model output by the model with respect to the test data set can be obtained, and the current performance of the target model can be further determined by this.

[0098] This example embodiment uses the Divide-and-Aggregate Learning (DAL, learning based on divide and conquer) method to measure distribution shift. Divide Learning refers to converting two output distributions corresponding to different data quantities (first output distribution and second output distribution) into two distribution information corresponding to the same data quantity (third distribution information and fourth distribution information) for distribution shift calculation, so as to overcome the problem that the distribution shift cannot be calculated due to the different data quantities in the two output distributions. Aggregate Learning means that since there is no clear relative relationship between the first output distribution and the second output distribution, the distribution shift between the two is symmetrical, which contradicts the asymmetry of the KL divergence. In order to achieve distance symmetry, Aggregate Learning uses the characteristic that the sum of the divergence of the first output distribution relative to the second output distribution and the divergence of the second output distribution relative to the first output distribution is equal to the sum of the divergence of the second output distribution relative to the first output distribution and the divergence of the first output distribution relative to the second output distribution to represent the distribution shift, and adds the divergence of the second output distribution relative to the first output distribution and the divergence of the first output distribution relative to the second output distribution as the first distribution shift information between the second output distribution and the first output distribution. This example embodiment proposes a label-free model evaluation method based on DAL divide-and-conquer, which can evaluate the current performance of the model without labels, reduce the manpower cost, time cost, and money cost of model performance evaluation, and meet real-time requirements.

[0099] Figure 2 A flowchart of a method for predicting accuracy provided in an embodiment of this specification is provided. In an embodiment of this specification, the method for predicting accuracy is applied to a device for predicting accuracy (hereinafter referred to as "accuracy prediction device") or an electronic device equipped with an accuracy prediction device. Figure 2 The process shown is described in detail, and the method for predicting accuracy may specifically include the following steps:

[0100] S202, obtaining second distribution offset information between a third output distribution of the target model with respect to the unlabeled test data set and a first output distribution of the target model with respect to the original data set. In some embodiments, the relevant operations have been described in detail above and will not be repeated here.

[0101] S204, inputting the second distribution shift information into the trained accuracy assessment model to obtain the output second prediction accuracy of the target model with respect to the test data set. In some embodiments, the relevant operations have been described in detail above and will not be repeated here. In some embodiments, the training method of the accuracy assessment model has been described in detail above and will not be repeated here.

[0102] Figure 3 A flowchart of a method for training an accuracy assessment model is provided as an example in an embodiment of this specification.

[0103] like Figure 3 As shown, by inputting the original data set into the target model, a first output distribution of the target model with respect to the original data set is obtained, by inputting the first sample data set corresponding to the original data set into the target model, a second output distribution of the target model with respect to the first sample data set is obtained, and first distribution offset information between the second output distribution and the first output distribution is obtained. According to the label information corresponding to the original data set, a first prediction accuracy of the target model with respect to the first sample data set is determined, and the first prediction accuracy is used as the label information corresponding to the first distribution offset information. Regression training is performed according to the first distribution offset information and its corresponding label information to obtain a trained accuracy evaluation model.

[0104] Figure 4 A flowchart of a method for predicting accuracy of an example provided in an embodiment of this specification.

[0105] like Figure 4 As shown, by inputting the original data set into the target model, the first output distribution of the target model with respect to the original data set is obtained, by inputting the unlabeled test data set into the target model, the third output distribution of the target model with respect to the test data set is obtained, the second distribution offset information between the third output distribution and the first output distribution is obtained, the second distribution offset information is input into the trained accuracy evaluation model, and the second prediction accuracy of the output target model with respect to the test data set is obtained.

[0106] Figure 5 A schematic diagram of the structure of a device for training an accuracy assessment model provided in an embodiment of the present specification, the device for training an accuracy assessment model (hereinafter referred to as "accuracy assessment model training device 1") can be implemented as all or part of an electronic device through software, hardware or a combination of both. According to some embodiments, the accuracy assessment model training device 1 includes a sample data set acquisition module 11, an output distribution acquisition module 12, a first distribution offset acquisition module 13 and a model training module 14.

[0107] The sample data set acquisition module 11 is used to acquire a first sample data set corresponding to the original data set according to the original data set corresponding to the target model;

[0108] An output distribution obtaining module 12, used to obtain a first output distribution of the target model with respect to the original data set and a second output distribution of the target model with respect to the first sample data set;

[0109] A first distribution offset obtaining module 13 is used to obtain first distribution offset information between the second output distribution and the first output distribution, and determine a first prediction accuracy of the target model with respect to the first sample data set according to label information corresponding to the original data set;

[0110] The model training module 14 is used to train the accuracy assessment model according to the first distribution shift information and the first prediction accuracy to obtain a trained accuracy assessment model, wherein the first prediction accuracy is used as the label information corresponding to the first distribution shift information, and the accuracy assessment model is used to output the second prediction accuracy of the target model with respect to the test data set according to the second distribution shift information corresponding to the input unlabeled test data set.

[0111] In some embodiments, obtaining a first sample data set corresponding to the original data set according to the original data set corresponding to the target model includes: randomly sampling the original data set corresponding to the target model to obtain the first sample data set corresponding to the original data set.

[0112] In some embodiments, randomly extracting the original data set corresponding to the target model to obtain a first sample data set corresponding to the original data set includes: randomly extracting the original data set corresponding to the target model to obtain a corresponding second sample data set; transforming the second sample data set according to a preset transformation rule to obtain a first sample data set corresponding to the original data set; wherein, determining the first prediction accuracy of the target model with respect to the first sample data set based on the label information corresponding to the original data set includes: determining the first prediction accuracy of the target model with respect to the first sample data set based on the label information corresponding to the original data set and the preset transformation rule.

[0113] In some embodiments, obtaining the first distribution offset information between the second output distribution and the first output distribution includes: performing a binning operation on the first output distribution to obtain corresponding first binning sequence information, wherein the first binning sequence information includes multiple first bins, each first bin includes at least one first output; performing a binning operation on the second output distribution to obtain corresponding second binning sequence information, wherein the second binning sequence information includes multiple second bins, each second bin includes at least one second output, and the number of first bins is the same as the number of second bins; constructing third distribution information based on the proportional characteristics of the first binning sequence information relative to the first output distribution, and constructing fourth distribution information based on the proportional characteristics of the second binning sequence information relative to the second output distribution; determining the first distribution offset information between the second output distribution and the first output distribution based on the third distribution information and the fourth distribution information.

[0114] In some embodiments, before obtaining the first distribution offset information between the second output distribution and the first output distribution, the method further includes: performing a sorting operation on the first output distribution and the second output distribution respectively.

[0115] In some embodiments, performing a binning operation on the first output distribution to obtain corresponding first binning sequence information includes: performing an equal-frequency binning operation on the first output distribution to obtain corresponding first binning sequence information, wherein each first bin contains the same number of first outputs.

[0116] In some embodiments, performing a binning operation on the first output distribution to obtain corresponding first binning sequence information includes: performing an equal-width binning operation on the first output distribution to obtain corresponding first binning sequence information, wherein the length of the first output value interval corresponding to each first bin is the same.

[0117] In some embodiments, determining the first distribution offset information between the second output distribution and the first output distribution based on the third distribution information and the fourth distribution information includes: determining the distribution offset information between the second output distribution and the first output distribution based on the difference between each first output in the first binning sequence information and the second output at the same position corresponding to the first output in the second binning sequence information, and the ratio of the first output to the second output.

[0118] In some embodiments, constructing the third distribution information based on the proportional characteristics of the first binning sequence information relative to the first output distribution includes: constructing the third distribution information based on the ratio of the first output quantity in each first bin in the first binning sequence information to the first output quantity in the first output distribution.

[0119] In some embodiments, obtaining the first output distribution of the target model with respect to the original data set and the second output distribution of the target model with respect to the first sample data set includes: obtaining a first output distribution composed of a first feature extracted by the target model for the original data set and a second output distribution composed of a second feature extracted by the target model for the first sample data set.

[0120] In some embodiments, both the first feature and the second feature are one-dimensional features that have been normalized and dimensionally reduced.

[0121] In some embodiments, obtaining the first output distribution of the target model with respect to the original data set and the second output distribution of the target model with respect to the first sample data set includes: obtaining a first output distribution composed of a first confidence level predicted by the target model for the original data set and a second output distribution composed of a second confidence level predicted by the target model for the first sample data set.

[0122] In some embodiments, the accuracy assessment model training device 1 is also used to: obtain second distribution offset information between the third output distribution of the target model with respect to the test data set and the first output distribution; input the second distribution offset information into the trained accuracy assessment model to obtain the output second prediction accuracy of the target model with respect to the test data set.

[0123] Figure 6 A schematic diagram of a device for predicting accuracy provided in an embodiment of the present specification, the device for predicting accuracy (hereinafter referred to as "accuracy prediction device 2") can be implemented as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the accuracy prediction device 2 includes a second distribution offset acquisition module 21 and a model prediction module 22.

[0124] A second distribution shift obtaining module 21 is used to obtain second distribution shift information between a third output distribution of the target model with respect to the unlabeled test data set and a first output distribution of the target model with respect to the original data set;

[0125] The model prediction module 22 is used to input the second distribution shift information into the trained accuracy evaluation model to obtain the output second prediction accuracy of the target model with respect to the test data set.

[0126] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0127] The embodiment of the present specification also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor to execute the method of the embodiment of the present specification.

[0128] The embodiments of the present specification also provide a computer program product, which stores at least one instruction, and the at least one instruction is loaded by the processor to execute the method of the embodiments of the present specification.

[0129] The embodiments of this specification also provide Figure 7 The structural diagram of the electronic device shown in FIG. Figure 7 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the method of the embodiment of this specification.

[0130] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0131] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0132] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0133] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0135] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0136] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0137] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0138] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for training an accuracy assessment model, comprising: According to the original data set corresponding to the target model, a first sample data set corresponding to the original data set is obtained; Obtaining a first output distribution of the target model with respect to the original data set and a second output distribution of the target model with respect to the first sample data set; Obtaining first distribution offset information between the second output distribution and the first output distribution, and determining a first prediction accuracy of the target model with respect to the first sample data set according to label information corresponding to the original data set; An accuracy assessment model is trained according to the first distribution shift information and the first prediction accuracy to obtain a trained accuracy assessment model, wherein the first prediction accuracy is used as label information corresponding to the first distribution shift information, and the accuracy assessment model is used to output a second prediction accuracy of the target model with respect to the test data set according to the second distribution shift information corresponding to the input unlabeled test data set.

2. According to the method of claim 1, obtaining a first sample data set corresponding to the original data set according to the original data set corresponding to the target model comprises: An original data set corresponding to the target model is randomly sampled to obtain a first sample data set corresponding to the original data set.

3. According to the method of claim 2, the step of randomly extracting the original data set corresponding to the target model to obtain a first sample data set corresponding to the original data set comprises: Randomly extracting the original data set corresponding to the target model to obtain a corresponding second sample data set; Transform the second sample data set according to a preset transformation rule to obtain a first sample data set corresponding to the original data set; The step of determining the first prediction accuracy of the target model with respect to the first sample data set according to the label information corresponding to the original data set includes: A first prediction accuracy of the target model with respect to the first sample data set is determined according to the label information corresponding to the original data set and the preset transformation rule.

4. The method according to claim 1, wherein obtaining first distribution offset information between the second output distribution and the first output distribution comprises: Performing a binning operation on the first output distribution to obtain corresponding first binning sequence information, wherein the first binning sequence information includes a plurality of first bins, each of which includes at least one first output; Performing a binning operation on the second output distribution to obtain corresponding second binning sequence information, wherein the second binning sequence information includes a plurality of second bins, each second bin includes at least one second output, and the number of the first bins is the same as the number of the second bins; Constructing third distribution information according to a ratio feature of the first binning sequence information relative to the first output distribution, and constructing fourth distribution information according to a ratio feature of the second binning sequence information relative to the second output distribution; First distribution offset information between the second output distribution and the first output distribution is determined according to the third distribution information and the fourth distribution information.

5. The method according to claim 4, before obtaining the first distribution offset information between the second output distribution and the first output distribution, further comprising: A sorting operation is performed on the first output distribution and the second output distribution respectively.

6. The method according to claim 4, wherein the step of performing a binning operation on the first output distribution to obtain corresponding first binning sequence information comprises: An equal-frequency binning operation is performed on the first output distribution to obtain corresponding first binning sequence information, wherein each first bin contains the same number of first outputs.

7. The method according to claim 4, wherein the step of performing a binning operation on the first output distribution to obtain corresponding first binning sequence information comprises: An equal-width binning operation is performed on the first output distribution to obtain corresponding first binning sequence information, wherein the length of the first output value interval corresponding to each first bin is the same.

8. The method according to claim 4, wherein determining first distribution offset information between the second output distribution and the first output distribution according to the third distribution information and the fourth distribution information comprises: Distribution offset information between the second output distribution and the first output distribution is determined based on the difference between each third data in the third distribution information and the fourth data in the fourth distribution information corresponding to the same position as the third data, and the ratio of the third data to the fourth data.

9. The method according to claim 4, wherein constructing the third distribution information according to the ratio characteristic of the first binning sequence information relative to the first output distribution comprises: The third distribution information is constructed according to a ratio of the first output quantity in each first bin in the first bin sequence information to the first output quantity in the first output distribution.

10. The method according to claim 1, wherein obtaining a first output distribution of the target model with respect to the original data set and a second output distribution of the target model with respect to the first sample data set comprises: A first output distribution consisting of a first feature extracted by the target model for the original data set and a second output distribution consisting of a second feature extracted by the target model for the first sample data set are obtained.

11. According to the method of claim 10, the first feature and the second feature are both one-dimensional features after normalization and dimensionality reduction.

12. The method according to claim 1, wherein obtaining a first output distribution of the target model with respect to the original data set and a second output distribution of the target model with respect to the first sample data set comprises: A first output distribution composed of a first confidence level predicted by the target model for the original data set and a second output distribution composed of a second confidence level predicted by the target model for the first sample data set are obtained.

13. The method according to any one of claims 1 to 12, further comprising: Acquire second distribution offset information between a third output distribution of the target model with respect to the test data set and the first output distribution; The second distribution shift information is input into the trained accuracy evaluation model to obtain the output second prediction accuracy of the target model with respect to the test data set.

14. A method for predicting accuracy, comprising: Obtaining second distribution offset information between a third output distribution of the target model with respect to the unlabeled test data set and a first output distribution of the target model with respect to the original data set; The second distribution offset information is input into a trained accuracy assessment model to obtain a second prediction accuracy of the output target model with respect to the test data set, wherein the trained accuracy assessment model is trained by the method described in any one of claims 1 to 12.

15. A device for training an accuracy assessment model, comprising: A sample data set acquisition module, used to acquire a first sample data set corresponding to the original data set according to the original data set corresponding to the target model; An output distribution obtaining module, used to obtain a first output distribution of the target model with respect to the original data set and a second output distribution of the target model with respect to the first sample data set; a first distribution offset obtaining module, configured to obtain first distribution offset information between the second output distribution and the first output distribution, and determine a first prediction accuracy of the target model with respect to the first sample data set according to label information corresponding to the original data set; A model training module is used to train an accuracy assessment model according to the first distribution shift information and the first prediction accuracy to obtain a trained accuracy assessment model, wherein the first prediction accuracy is used as label information corresponding to the first distribution shift information, and the accuracy assessment model is used to output the second prediction accuracy of the target model with respect to the test data set according to the second distribution shift information corresponding to the input unlabeled test data set.

16. A device for predicting accuracy, comprising: A second distribution shift obtaining module, used to obtain second distribution shift information between a third output distribution of the target model with respect to the unlabeled test data set and a first output distribution of the target model with respect to the original data set; A model prediction module is used to input the second distribution offset information into a trained accuracy assessment model to obtain the output second prediction accuracy of the target model with respect to the test data set, wherein the trained accuracy assessment model is trained by the method described in any one of claims 1 to 12.

17. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

18. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method as claimed in any one of claims 1 to 14.

19. A computer program product having at least one instruction stored thereon, characterized in that: When the at least one instruction is executed by the processor, the steps of the method described in any one of claims 1 to 14 are implemented.

Images