A label model verification method, device and equipment

Abstract

The application relates to a label model verification method, device and equipment, the method comprising: acquiring full object data, determining a label category distribution corresponding to label information of a target label in the full object data; randomly extracting a preset sample amount of object data items from the full object data to generate object data samples; based on the label category distribution and a target label model corresponding to the target label, performing sample structure adjustment on the object data samples to obtain target data samples, and determining a target confusion matrix obtained by predicting label information of the target label for each object in the target data samples; and based on the target confusion matrix, verifying the accuracy of label information predicted by the target label model. The application can make the verification result obtained based on the extracted sample data more objective, avoid the deviation of the verification result caused by biased sampling, and reduce the acquisition cost of sample data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data mining, and in particular to a label model verification method, device and equipment. BACKGROUND

[0002] A user label system is the basis of data operation and an indispensable feature module in a big data model; high label accuracy can help products achieve thousands of faces and win users; poor label accuracy can have a negative impact on business. Taking advertisement targeting as an example, poor label accuracy mainly has two effects. Firstly, it affects the user experience of the traffic side. If a user who has already graduated from a master's program always sees adult undergraduate examination advertisement when using a product, this inaccurate advertisement targeting makes the user experience unsatisfactory, and the user may even uninstall the product and permanently lose it. Secondly, it affects customer relationships and even leads to enterprise public relations crises and legal risks. The People's Republic of China Advertising Law limits the content of advertisements on mass media for minors. If the accuracy of age labels is poor, alcohol and tobacco advertisements are targeted at minors, which may result in a warning from the advertiser, or even lead the enterprise into a public relations crisis and legal risk. As can be seen, in order to provide support for accurate advertisement targeting, fine operation, personalized recommendation, intelligent marketing, big data risk control, etc., an effective label accuracy verification means is needed to ensure the quality of labels before they go online. SUMMARY

[0003] The technical problem to be solved by the present application is to provide a label model verification method, device and equipment, which can solve the problem that sample data extracted from full data does not have sampling representativeness, so that the verification result obtained based on the extracted sample data is more objective, avoiding the deviation of the verification result caused by biased sampling, and reducing the cost of obtaining sample data.

[0004] To solve the above technical problems, on the one hand, the present application provides a label model verification method, which comprises:

[0005] Obtaining full object data, determining the label category distribution corresponding to the label information of the target label in the full object data;

[0006] Randomly extracting a preset sample size of object data items from the full object data to generate object data samples; wherein the preset sample size is obtained by multiple sampling of the preset object data and analysis of the sampling results of the multiple sampling;

[0007] based on the label category distribution corresponding to the label information of the target label in the full object data and the target label model corresponding to the target label, performing sample structure adjustment on the object data sample to obtain a target data sample, and determining a target confusion matrix obtained by predicting the label information of the target label for each object in the target data sample;

[0008] based on the target confusion matrix, verifying the accuracy of the target label model in predicting label information.

[0009] In another aspect, the present application provides a label model verification device, the device comprising:

[0010] a full object data acquisition module configured to acquire full object data and determine a label category distribution corresponding to the label information of a target label in the full object data;

[0011] an object data sample generation module configured to randomly extract a preset sample amount of object data items from the full object data to generate an object data sample; wherein the preset sample amount is obtained by performing multiple sampling on preset object data and analyzing the sampling results of the multiple sampling;

[0012] a target confusion matrix generation module configured to, based on the label category distribution corresponding to the label information of the target label in the full object data and the target label model corresponding to the target label, perform sample structure adjustment on the object data sample to obtain a target data sample, and determine a target confusion matrix obtained by predicting the label information of the target label for each object in the target data sample;

[0013] an accuracy verification module configured to, based on the target confusion matrix, verify the accuracy of the target label model in predicting label information.

[0014] In another aspect, the present application provides a device comprising a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the label model verification method as described above.

[0015] In another aspect, the present application provides a computer storage medium, the storage medium storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by a processor to implement the label model verification method as described above.

[0016] Implementing the embodiments of the present application has the following beneficial effects:

[0017] The application generates object data samples by randomly extracting a preset sample size of object data items from the acquired full amount of object data; adjusts the sample structure of the object data samples based on the label category distribution corresponding to the label information of the target label in the full amount of object data and the target label model corresponding to the target label, to obtain target data samples, and determines the target confusion matrix obtained by predicting the label information of the target label for each object in the target data samples; and then calculates the relevant performance indicators based on the target confusion matrix to verify the accuracy of the label information predicted by the target label model. In the application, the sampling sample size when random sampling is obtained by sampling the preset object data multiple times and analyzing the sampling results of the multiple samplings. In addition, when the object data samples are sampled from the full amount of object data, the sample structure of the object data samples is adjusted, so that by determining the appropriate sampling sample size and adjusting the sample structure of the sampled samples, the problem that the sampled data extracted from the full amount of data is not representative of sampling can be solved, the verification result obtained based on the extracted sample data is more objective, and the deviation of the verification result caused by biased sampling is avoided. After the sampling sample size is determined, only the corresponding sample size of data items needs to be obtained to achieve the sampling effect of obtaining more than the sample size of data items, without the need to additionally obtain more data items, thereby reducing the cost of obtaining sample data and saving system resources. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, and the advantages thereof, a brief introduction will be given to the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on these drawings.

[0019] Figure 1 is an implementation environment schematic diagram provided by an embodiment of the present application;

[0020] Figure 2 is a label model verification method flowchart provided by an embodiment of the present application;

[0021] Figure 3 is a sample structure adjustment method flowchart provided by an embodiment of the present application by using a weighted proportioning method;

[0022] Figure 4 is a label information prediction method flowchart provided by an embodiment of the present application;

[0023] Figure 5 is a target data sample generation method flowchart provided by an embodiment of the present application;

[0024] Figure 6 is a target confusion matrix generation method flowchart provided by an embodiment of the present application;

[0025] Figure 7 is a sample structure adjustment method flowchart provided by an embodiment of the present application through stratified sampling method;

[0026] Figure 8 is another target data sample generation method flowchart provided by an embodiment of the present application;

[0027] Figure 9 is a confusion matrix schematic diagram of real results and predicted results provided by an embodiment of the present application;

[0028] Figure 10 is a label model optimization method flowchart provided by an embodiment of the present application;

[0029] Figure 11 is an accuracy change trend analysis method flowchart provided by an embodiment of the present application;

[0030] Figure 12 is a label model evaluation optimization method schematic diagram provided by an embodiment of the present application;

[0031] Figure 13 is an illustration schematic diagram of evaluating real sample relationship of the first period and the second period provided by an embodiment of the present application;

[0032] Figure 14 is an age distribution schematic diagram provided by an embodiment of the present application;

[0033] Figure 15 is a process schematic diagram of biased sampling provided by an embodiment of the present application;

[0034] Figure 16 is a weighted proportion adjustment process schematic diagram provided by an embodiment of the present application;

[0035] Figure 17 is a stratified sampling adjustment process schematic diagram provided by an embodiment of the present application;

[0036] Figure 18 is a label model verification device schematic diagram provided by an embodiment of the present application;

[0037] Figure 19 is a device structure schematic diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the objects, technical solutions and advantages of the present application clearer, the following further describes the present application with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0039] It should be noted that the terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in other than the order illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units need not be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0040] First, the related terms involved in the embodiments of the present application are explained as follows:

[0041] Large dish: the total number of users of the business;

[0042] Real sample: the sample obtaining the real label is called a real sample;

[0043] Evaluation panel library: the sample library composed of real samples obtained each time the label is evaluated;

[0044] Bad case: in the label accuracy verification, the sample whose prediction result is inconsistent with the real result;

[0045] Good case: in the label accuracy verification, the sample whose prediction result is consistent with the real result.

[0046] Please refer to Figure 1It shows an implementation environment schematic diagram provided by the embodiment of the application, which can include a data sample library 110, a first terminal 120 and a second terminal 130. Specifically, when the second terminal 130 needs to obtain full-amount object data, the full-amount object data can be obtained through various channels, for example, the second terminal 130 can obtain self-built stable sample data from the data sample library 110, or can obtain sample data in the form of a questionnaire from the first terminal 120, and the like. When the second terminal 130 obtains the full-amount object data, the second terminal 130 extracts object data items of a preset sample amount from the full-amount object data to generate object data samples, and verifies the prediction accuracy of a target label model based on a label category distribution corresponding to label information of a target label in the full-amount data, the target label model corresponding to the target label, and the object data samples.

[0047] The first terminal 110 can communicate with the second terminal 120 based on a browser / server mode (B / S) or a client / server mode (C / S). The first terminal 110 can include an entity device such as a smart phone, a tablet computer, a notebook computer, a digital assistant, a smart wearable device, a vehicle-mounted terminal, a server, and the like, or can include software such as an application program running in the entity device. The operating system running on the first terminal 110 in the embodiment of the application can include, but is not limited to, an Android system, an IOS system, linux, windows, and the like.

[0048] The second terminal 130 can establish a communication connection with the first terminal 120 through a wire or wireless. The second terminal 130 can be an independent physical server, or can be a server cluster or a distributed system composed of multiple physical servers, or can be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, and big data and artificial intelligence platforms.

[0049] Big data refers to a collection of data that cannot be captured, managed and processed within a certain time range by conventional software tools, and is a massive, high-growth and diversified information asset that requires new processing modes to have stronger decision-making, insight discovery and process optimization capabilities. With the advent of the cloud era, big data has attracted more and more attention. Big data requires special technology to effectively process large amounts of data over time. The technologies suitable for big data include large-scale parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the Internet and scalable storage systems. The embodiment of the application can mine user labels to establish a user portrait through big data mining technology.

[0050] To address the problem in existing technologies where object data samples extracted from the full object dataset lack representativeness and cannot represent the verification results based on the full object dataset when predicting target label models, this application provides a label model verification method. The execution entity can be the aforementioned second terminal. (See also...) Figure 2 The method may include:

[0051] S210. Obtain the full object data and determine the label category distribution corresponding to the label information of the target label in the full object data.

[0052] When acquiring full object data, this application embodiment can flexibly obtain real data for various tags through multiple channels, specifically including self-built sample libraries, questionnaire design, and offline surveys. These three methods complement each other, and the multi-channel real data acquisition solution can solve the problem of data acquisition difficulties caused by the lack of channels to contact real users, thus meeting the needs for evaluating and verifying various tags. Here, "object" specifically refers to business users. The full object data in this application embodiment can refer to user data of a preset number of users used for tag model verification. It can include real tag data of a preset number of users and other data, so that when predicting user tag information, the predicted tag information can be compared with the real tag information to determine whether the tag information prediction is correct.

[0053] In this embodiment of the application, the target tag information corresponds to at least two tag categories. The tag category distribution includes the proportion of the number of objects in each tag category to the total number of objects. For example, when the target tag is gender, for a certain user object, the tag information corresponding to the gender tag is either male or female, so the corresponding tag categories are divided into male and female. Specifically, the tag category distribution refers to the proportion of the number of objects with the gender tag information of male to the total number of objects in the full object data, and the proportion of the number of objects with the gender tag information of female to the total number of objects. For example, the proportion of the tag category of male is a, and the proportion of the tag category of female is (1-a).

[0054] S220. Randomly extract a preset number of object data items from the full object data to generate object data samples; wherein, the preset number of samples is obtained by sampling the preset object data multiple times and analyzing the sampling results of the multiple samplings.

[0055] When sampling the full object data, there is a high probability that the sample structure of the sample sample deviates from the structure in the large plate data. When there is a difference between the sampling structure in the real sample and the large plate structure, the verification result of the sampling may be seriously biased, so that the sampling accuracy cannot represent the large plate accuracy. The label model verification result in the embodiment of the application is obtained by sampling and estimation, and measures can be taken from two aspects to make the verification result of the label model based on the sampling sample consistent with the verification result of the label model based on the large plate data, that is, the determination of the sampling sample quantity and the structure adjustment of the sampling sample.

[0056] For the determination of the sampling sample quantity, the preset sample quantity here is obtained by sampling the preset object data multiple times and analyzing it. The preset sample quantity can be considered as an optimal sample quantity within a certain range. When the extracted sample quantity is less than the optimal sample quantity, the corresponding sampling effect is worse than that of the sample corresponding to the optimal sample quantity. When the extracted sample quantity is greater than the optimal sample quantity, the corresponding sampling effect is similar to that of the sample corresponding to the optimal sample quantity. Therefore, the sampling sample quantity in the embodiment of the application is determined as the optimal sample quantity, which can achieve a sampling effect similar to that of the sample corresponding to the optimal sample quantity without the need for more data item extraction, thereby reducing the acquisition cost of the sample data. In the specific determination of the sample quantity, the optimal sample quantity can be set as the actual sample quantity, or the actual sample quantity can be set to be greater than the optimal sample quantity, which is determined according to the implementation.

[0057] S230. Based on the label category distribution corresponding to the label information of the target label in the full object data, and the target label model corresponding to the target label, the sample structure of the object data sample is adjusted to obtain a target data sample, and a target confusion matrix is determined by predicting the label information of the target label of each object in the target data sample.

[0058] After sampling the object data sample from the full object data, the object data sample needs to be adjusted in structure. An appropriate sampling scheme with a representative structure of the large plate can be used, which can be realized by methods such as weighted proportioning, stratified sampling, Bootstrap sampling (self-help sampling method) and the like.

[0059] For adjusting the sample structure by the weighted proportioning method, reference can be made to Figure 3 , which can include:

[0060] S310. Based on the target label model corresponding to the target label, the label information of the target label of each object in the object data sample is predicted to obtain a first prediction result; the first prediction result includes the predicted object quantity corresponding to each label category of the target label.

[0061] For each target label, there is a target label model corresponding thereto, when predicting the label information of a target label of a user, the target label information of the user can be predicted based on other item information of the user, each object data item corresponds to an object, each object data item of an object includes a plurality of data corresponding to the object, and the plurality of data includes real label information of the target label; for details, see Figure 4 The method shown in FIG. 3 is a label information prediction method, which can include the following steps:

[0062] S410. For each object in the object data sample, data other than the real label information of the target label in the plurality of data corresponding to the object is taken as the input of the target label model, and a first label prediction result of the target label model for the target label of the object is obtained.

[0063] Here, the first label prediction result can be the predicted label information of the target label of each object in the object data sample.

[0064] S420. The first label prediction results of the objects are counted to obtain a first confusion matrix.

[0065] For each object, the real label information of the target label of the object can be determined from the object data sample, and the predicted label information of the target label of the object determined after target label prediction. Comparing the real label information with the predicted label information can determine whether the label information prediction for the object is correct. The number of objects whose label information prediction is accurate is counted to obtain a first confusion matrix. In the first confusion matrix, for a certain label category, the number of objects whose prediction is accurate and the number of objects whose prediction is incorrect in the label category are counted.

[0066] S430. The first confusion matrix is determined as the first prediction result.

[0067] S320. For each label category corresponding to the target label, a first proportion of the number of objects belonging to the label category in the total object data is determined based on the label category distribution, and a second proportion of the number of objects belonging to the label category in the object data sample is determined.

[0068] According to the above, the label category distribution includes the proportion of the number of objects of each label category in the total number of objects, so that according to the label category distribution of the target label in the full object data, the first proportion of the number of objects belonging to each label category in the total number of objects can be determined; since the object data sample is randomly sampled from the full object data, the proportions of the number of objects of each label category in the total number of objects in the object data sample are not consistent, so that the second proportion of the number of objects belonging to each label category in the sample in the object data sample needs to be recalculated.

[0069] S330. Determine the weight corresponding to each label category based on the first proportion and the second proportion corresponding to each label category.

[0070] For each label category, the ratio of the second proportion corresponding to the label category to the first proportion corresponding to the label category can be taken as the weight corresponding to the label category.

[0071] S340. Obtain the target data sample based on the weight corresponding to each label category and the object data sample.

[0072] Since the number of objects belonging to each label category classification can be determined according to the object data sample, for each label category classification, the product of the number of objects belonging to the label category classification in the object data sample and the weight of the label category is determined as the number of objects belonging to the label category classification in the target data sample.

[0073] Specifically, refer to Figure 5 which shows a target data sample generation method, which can include:

[0074] S510. Multiply the number of real objects corresponding to each label category in the object data sample by the weight corresponding to the corresponding label category to obtain the number of real objects corresponding to each label category after weighting.

[0075] S520. Generate the target data sample based on the objects of each label category and the number of real objects of the corresponding label category after weighting.

[0076] For example, for label category A, the weight corresponding to it is a, and the number of objects actually belonging to label category A in the object data sample is x, so the number of objects actually belonging to label category A in the target data sample can be determined as x*a; based on the same method, the number of objects belonging to different label categories in the target data sample can be determined.

[0077] S350. Adjust the predicted object number corresponding to each label category in the first prediction result based on the weight corresponding to each label category to obtain the target confusion matrix.

[0078] The first prediction result can be a first confusion matrix obtained after label information prediction on the object data sample. Similarly, in the first confusion matrix, the number of objects actually belonging to each label category is multiplied by the corresponding weight to obtain the number of objects actually belonging to the label category after adjustment. Figure 6 The method for generating a target confusion matrix is shown in FIG. 6, which can include the following steps.

[0079] S610. The number of predicted objects corresponding to each label category in the first confusion matrix is multiplied by the weight corresponding to each label category to obtain a second confusion matrix.

[0080] S620. The second confusion matrix is determined as the target confusion matrix.

[0081] The target confusion matrix obtained here is obtained by weighting and matching the sampled object data sample. By analyzing the target confusion matrix, the same or similar verification results as the full-disk data can be obtained.

[0082] For adjusting the sample structure by stratified sampling method, please refer to Figure 7 The method can include the following steps.

[0083] S710. A first number of object data items are extracted from the object data sample to generate the target data sample, wherein the label category distribution corresponding to the label information of the target label in the target data sample is consistent with the label category distribution corresponding to the label information of the target label in the full-disk object data.

[0084] For the target data sample extracted from the object data sample, it is necessary to ensure that the label category distribution corresponding to the label information of the target label is consistent with the label category distribution corresponding to the label information of the target label in the full-disk object data. In this way, the verification result obtained by analyzing the target data sample can represent the verification result obtained by analyzing the full-disk data.

[0085] Please refer to Figure 8 Another method for generating a target data sample is shown in FIG. 8, which can include the following steps.

[0086] S810. For each label category corresponding to the target label, a third proportion of the number of objects belonging to the label category in the full-disk object data is determined based on the label category distribution.

[0087] According to the above content, the label category distribution includes the proportion of the number of objects in each label category in the total number of objects.

[0088] S820. Determine the number of objects belonging to each label category in the target data sample based on the first number and a third proportion of the number of objects belonging to each label category in the total object data.

[0089] For each label category, the product of the first number and the third proportion corresponding to the label category is determined as the number of objects in the target data sample of the label category.

[0090] S830. Extract the corresponding object data items from the object data sample based on the number of objects belonging to each label category in the target data sample, and generate the target data sample.

[0091] Through the above calculation, the number of objects of each label category in the target data sample is determined, and the target data sample can be generated by extracting from the object data sample according to the corresponding label category and the corresponding number.

[0092] S720. For each object in the target data sample, the data in the multiple data corresponding to the object except the true label information of the target label is taken as the input of the target label model, and the second label prediction result of the target label model for the object is obtained.

[0093] S730. Statistically analyze the second label prediction results of the objects to obtain a third confusion matrix.

[0094] Here, the specific method of predicting label information for each object in the target data sample and generating a corresponding confusion matrix can be referred to the above steps S410-S430, which will not be repeated here.

[0095] S740. Determine the third confusion matrix as the target confusion matrix.

[0096] In the embodiments of the present application, the structure adjustment of the sampling sample is described in detail by using the two methods of weighted proportioning and stratified sampling. It should be noted that this is only an example, and any method that can adjust the structure of the sampling sample can be applied to the embodiments of the present application.

[0097] S240. Verify the accuracy of the label information predicted by the target label model based on the target confusion matrix.

[0098] After obtaining the target confusion matrix based on the object data sample, a series of index calculations can be performed based on the target confusion matrix. These indexes obtained by calculation can be used to evaluate the accuracy of the label information predicted by the target label model.

[0099] For details, please refer to Figure 9which shows the confusion matrix of real results and predicted results, the index parameters used in the embodiments of the application can include:

[0100] 1. Accuracy

[0101] The accuracy of the label refers to the proportion of users labeled correctly, and the accuracy is the most core index of the user label system. A label with very low accuracy has no application value. The accuracy calculation is shown in formula (1):

[0102]

[0103] 2. Precision, recall, f1

[0104] The precision, recall, and f1 of different label values are also important indexes, which can clearly indicate the label value to be focused on in the next step.

[0105] Precision (Precision) is also called precision, which is for the predicted results. Its meaning is the probability of actual positive samples in all predicted positive samples. The precision calculation is shown in formula (2):

[0106]

[0107] Recall (Recall) is also called recall, which is for the real results. Its meaning is the probability of predicted positive samples in actual positive (negative) samples. The recall calculation is shown in formula (3):

[0108]

[0109] The two indexes are usually mutually exclusive and difficult to achieve, and are mutually restricted in large-scale data sets. Therefore, comprehensive consideration is needed. The most common method is F-Measue, which is the weighted harmonic mean of Precision and Recall, as shown in formula (4):

[0110]

[0111] When the parameter a = 1, it is the most common f1, as shown in formula (5):

[0112]

[0113] f1 considers the results of precision and recall. When f1 is high, it indicates that the label mining effect of the target label model is relatively effective.

[0114] 3. Coverage rate

[0115] The label coverage refers to the proportion of users labeled in the total users, and the higher the coverage is, the better. However, the coverage and the accuracy are a pair of contradictory indexes, and the two need to be balanced. Generally, the coverage is improved as much as possible under the condition that the accuracy meets certain standards. The calculation formula of the coverage estimated by sampling data is shown in formula (6):

[0116]

[0117] The above related indexes of the embodiments of the present application can be used to measure the prediction ability of the target label model. It should be noted that the above indexes listed in the embodiments of the present application are only illustrative, and any index that can be used to evaluate the prediction ability of the target label model can be used in the embodiments of the present application.

[0118] The label model used for predicting the label information of the user in the embodiments of the present application is also trained based on related user parameters. While the quality of the label model is continuously evaluated, the label model is also continuously optimized. Specifically, please refer to Figure 10 which shows a label model optimization method. The method can include:

[0119] S1010. When the predicted label information of the target label of the target object obtained by performing target label information prediction on the target object based on the target label model is inconsistent with the real label information, determining that the object data item corresponding to the target object is an abnormal data item.

[0120] S1020. Backtracking the abnormal data item, and optimizing the target label model based on the abnormal data item.

[0121] The abnormal data item here refers to a bad case, that is, a data item whose prediction information is inconsistent with the real information after prediction by the label model. The related data items of the bad case are backtracked, and the original label model is optimized based on these related data items, so that the optimized label model can be obtained, thereby realizing the optimization of the label model while evaluating the prediction ability of the label model.

[0122] During the continuous evaluation of the accuracy of the label model, the change of the prediction accuracy before and after the optimization of the label model is analyzed. At this time, the accuracy evaluation conclusions of different channels may be inconsistent, for example, the real samples obtained by channels 1 and 2 are labeled and predicted by the models before and after optimization respectively. The analysis accuracy of channel 1 is improved, and the analysis accuracy of channel 2 is decreased. At this time, the accuracy change of the target label model needs to be determined based on the comprehensive analysis of the accuracy evaluation conclusions of the target label model of different channels. For details, please refer to Figure 11The method for analyzing the accuracy rate change trend can include:

[0123] S1110. Determine the object data group with a sampling coincidence rate higher than a first threshold value as a first group, and determine the object data group with a sampling coincidence rate lower than a second threshold value as a second group.

[0124] Specifically, when the sampling coincidence rate is higher than the first threshold value, for example, the sampling coincidence rate is higher than 60%, it can be determined that each time the object data group is extracted as the same group; when the sampling coincidence rate is lower than the second threshold value, for example, the sampling coincidence rate is approximately 0%, it can be determined that each time the object data group is extracted as a different group.

[0125] S1120. Randomly extract a first preset number of object data items from the first group to generate a first sample, and randomly extract a second preset number of object data items from the second group to generate a second sample.

[0126] According to the above content, the first sample extracted at the current time can be regarded as the same group as the first sample extracted at the previous time, and the second sample extracted at the current time can be regarded as a different group from the second sample extracted at the previous time.

[0127] S1130. Determine a first accuracy rate of the target label model for predicting label information based on a target confusion matrix corresponding to the first sample, and determine a second accuracy rate of the target label model for predicting label information based on a target confusion matrix corresponding to the second sample.

[0128] S1140. Determine the accuracy rate change trend of the target label model for predicting label information based on the first accuracy rate and the second accuracy rate.

[0129] For the specific determination of the accuracy rate change trend, refer to Table 1:

[0130] Table 1: Interpretation table of label model accuracy change

[0131]

[0132]

[0133] Why does the analysis of the overall change in accuracy rate need to be divided into two aspects of "the same group" and "different groups"? This can be explained through the following life examples:

[0134] A parent wants to know whether the child's math score has improved after attending a tutoring class. He summarizes from two angles:

[0135] 1.

Same paper

[0136] 2.

Different paper

[0137] If both scores are improved, parents can consider that the child's math score has improved overall; if the score of the same paper is improved, but the score of the different paper is not improved, then parents need to find another way to improve the child's thinking.

[0138] Based on the inspiration of multi-channel tag accuracy change evaluation, combined with the "same group" and "different group" angles, the overall change of tag accuracy in the big plate can be analyzed, and the application embodiment further provides a tag model evaluation optimization method. The flowchart of the method can be referred to Figure 12 , which shows the evaluation process and optimization process of the tag model at each period:

[0139] For the first period, specifically can include:

[0140] S1. Obtain new real samples of the first period.

[0141] S2. Predict the user tags in the new real samples by the tag model of the first period to obtain goodcase and badcase.

[0142] S3. Take goodcase as real samples of the next period, take the first part of badcase as real samples of the next period, and take the remaining part (second part) of badcase to optimize the tag model of the first period to generate a first optimized model.

[0143] For the second period, specifically can include:

[0144] S1. Obtain goodcase of the first period, first part of badcase of the first period, and new real samples.

[0145] S2. Predict the user tags in the obtained samples by the tag model of the second period (i.e. the first optimized model) to obtain goodcase and badcase.

[0146] S3. Take goodcase as real samples of the next period, take the first part of badcase as real samples of the next period, and take the remaining part (second part) of badcase to optimize the tag model of the second period to generate a second optimized model.

[0147] The operations of the third period and the fourth period and the like are similar to those of the second period, which will not be repeated here.

[0148] Wherein, the description of the evaluation of the real sample relationship in the first period and the second period can be referred to Figure 13 In the second period, the same group and different groups are divided for the evaluation of the first optimization model. Since part of the samples in the second period are the good cases and the first part of the bad cases in the first period, the part of the samples can be used as the same group to analyze the optimization effect of the model when the accuracy of the first optimization model is evaluated. The real samples newly obtained in the second period, which do not coincide with the samples in the first period, can be used as different groups to analyze the consistency of the optimization effect of the model. In addition, the bad cases in the first period for positioning problems and optimizing the model do not participate in the effect evaluation of the optimization model in the second period. For the good cases and the first part of the bad cases in the first period, if the latest state is not obtained, they do not participate in the effect evaluation of the optimization model in the second period.

[0149] In addition, in the same period, the accuracy of the analysis label is used to analyze the accuracy difference in different channels. The following aspects can be considered:

[0150] 1. Check the reliability of the real samples, check the structure of the sampled real samples, and ensure the accuracy of the analysis results in each channel;

[0151] 2. Different channels belong to "different groups", and the large accuracy difference indicates that the mining effect of the model is inconsistent, and the mining effect is good for part of the groups and general for part of the groups;

[0152] 3. Since the amount and type of original data generated by each user are different, the strength of data expression is also different, which will affect the accuracy of the mining effect. Therefore, the amount, type, and characteristic expression of the original data of different groups are analyzed to understand the reasons for the difference in the mining effect, and then the optimization model is inspired.

[0153] In the evaluation process of the label model, when the accuracy of the label model is found to be decreased, in addition to analyzing the reasons in the amount, type, and characteristic expression of the original data, we can also start from the basic attributes such as age, gender, and region, analyze the basic attribute dimensions, and narrow down the range of low-accuracy users to be positioned. For example, taking the parenting label of a certain evaluation as an example, please refer to Figure 14The age distribution of the parenting label is shown, and the parenting definition in this evaluation is: in the state of preparing for pregnancy, pregnancy, or children under 3 years old. The accuracy of this parenting is more than 10% lower than the last time. The analysis of the badcase age distribution found that the proportion of users less than 18 and more than 40 increased significantly (more than 20%), which is relatively abnormal (the age of the parenting user in the state of preparing for pregnancy or having a child under 3 years old is 18-40 years old, and the proportion is at least 90%). It is understood from the abnormality of the badcase age distribution that the parenting state of the user over 40 years old is not good, and the decrease of the accuracy of this parenting is likely to be caused by the inaccurate mining of the user over 40 years old. Through age analysis, the scope of the problem user is narrowed down, and the 40-year-old parenting user is focused on.

[0154] According to the above content, when implementing the label model verification method of the application, the following operation steps can be included: 1. Understand the label background; 2. Obtain real data; 3. Calculate the accuracy, precision, recall, f1 and coverage rate; 4. Analyze the accuracy trend; 5. Accuracy decrease analysis and the like.

[0155] 1. Understand the label background

[0156] Before verifying the accuracy of the label, it is very important to clearly understand the label background. The label background to be confirmed can refer to the content in Table 2:

[0157] Table 2 Label background to be confirmed

[0158]

[0159] 2. Obtain real data

[0160] After understanding the label background, three real data acquisition schemes can be used: Panel library technology, questionnaire design, and offline research collection. The three complement each other and can flexibly meet various label evaluation needs. For specific details of the three real data acquisition schemes, refer to Table 3:

[0161] Table 3 Three data acquisition schemes

[0162]

[0163] For the above real data acquisition scheme, five principles need to be met:

[0164] Reliability: Multi-party verification to ensure the authenticity of user information;

[0165] Flexibility: Support various evaluation needs;

[0166] Consistency: The real data obtained is consistent with the label definition;

[0167] Timeliness: Obtain the latest real data;

[0168] Stability: Support continuous evaluation, timely response to new label or label model update evaluation work.

[0169] 3. Calculate accuracy, precision, recall, f1 and coverage

[0170] For the real data obtained, there is a large possibility that the sample structure obtained is different from the mobile Internet user big plate structure. When the sampling structure in the real sample is different from the big plate structure, the verification result of the sample may be seriously biased, and the sampling accuracy cannot represent the big plate accuracy. Therefore, before calculating the related indicators, it is necessary to ensure that the sampling sample is representative. The above content in the embodiment of the application ensures that the sampling is representative by determining the appropriate sample size and adjusting the structure of the sampling sample.

[0171] (1) Determine the appropriate sample size

[0172] It is assumed that when the sampling accuracy and the real accuracy differ by ± 5%, the sampling accuracy is representative.

[0173] In different populations, the confidence level of the sampling accuracy and the real accuracy within ± 5% is:

[0174] In the same number of samples in the 100,000, 1,000,000, and 10,000,000 population, repeat 50 times, calculate the probability (confidence level) that satisfies the condition | sampling accuracy - real accuracy | ≤ ± 5%, and the conclusion is that the appropriate sampling sample size is independent of the size of the business population, and the sampling number does not need to increase with the increase of the business population.

[0175] In different sampling quantities, the confidence level of the sampling accuracy and the real accuracy within ± 5% is:

[0176] In the same population (10,000,000), different numbers of samples are extracted, repeated 50 times, and the probability (confidence level) that satisfies the condition | sampling accuracy - real accuracy | ≤ ± 5% is calculated. The conclusion is that when the sampling sample size reaches 400, the sampling accuracy is basically close to the real accuracy.

[0177] Therefore, the sample size of the object data sample extracted from the full object data in the embodiment of the application can be set to 400.

[0178] (2) Adjust the sample structure of the object data sample

[0179] When obtaining the real sample, there is a 99% possibility that the sampling sample structure obtained is different from the mobile Internet user big plate structure. The accuracy calculated by the sample structure with deviation is not representative of the big plate. Taking the gender label as an example for illustration:

[0180] First, it is assumed that the number of real users of male gender and female gender in the large data is known, and the prediction result of the user gender label in the large data is predicted by the label prediction model, to obtain the confusion matrix between the prediction result and the real data in the large data; real samples are extracted from the large data, and it is assumed that the proportion of male users extracted in the sampling process is increased by 15%, the distribution changes before and after sampling, and the extracted samples are directly predicted by the label prediction model to obtain the confusion matrix between the prediction result and the real result after sampling. The above biased sampling process can be referred to as Figure 15 .

[0181] The prediction accuracy of the label model on the large data and the prediction accuracy on the sampled samples are calculated respectively, and the comparison results are shown in Table 4:

[0182] Table 4: Comparison of accuracy of biased sampling

[0183] Prediction Market Accuracy Sample Accuracy Market vs. Sample Male 80 / (80+20)＝80％ 40 / (40+5)＝88.9％ ↑ Female 60 / (60+40)＝60％ 15 / (15+20)＝14.3％ ↓↓ Gender (80+60) / 200＝70％ (40+15) / 80＝68.8％ ↓

[0184] From Table 4, it can be seen that the accuracy of female before and after biased sampling differs by 46%, so when the sampling structure is different from the large data structure, the verification result of sampling may be seriously biased, and the sampling accuracy at this time cannot represent the large data accuracy.

[0185] The weighted ratio is used to adjust the sample structure, and the weighted ratio adjustment process can be referred to as Figure 16 The first weight of the proportion of the number of male users in the large data to the proportion of the number of male users in the sampled sample is calculated, and the second weight of the proportion of the number of female users in the large data to the proportion of the number of female users in the sampled sample is calculated. Based on the confusion matrix before the weighted ratio, the corresponding weights are multiplied, which can be that the number of users actually male in the sampled sample and the number of users predicted as male are multiplied by the first weight; the number of users actually female in the sampled sample and the number of users predicted as male are multiplied by the second weight, and the number of users predicted as female is multiplied by the second weight, to obtain the confusion matrix after the weighted ratio.

[0186] Based on the confusion matrix after the weighted ratio, the large data accuracy, the sampling accuracy before the weighted ratio, and the sampling accuracy after the weighted ratio are calculated, and the accuracy results after the weighted ratio are shown in Table 5:

[0187] Table 5: Accuracy results after weighted ratio

[0188] Prediction Market Accuracy Sample Accuracy Before Weighted Proportion Sample Accuracy After Weighted Proportion Male 80 / (80+20)＝80％ 40 / (40+5)＝88.9％ 40*0.8 / (40*0.8+5*1.6)＝80％ Female 60 / (60+40)＝60％ 15 / (15+20)＝14.3％ 15*1.6 / (15*1.6+20*0.8)＝60％ Gender (80+60) / 200＝70％ (40+15) / 80＝68.8％ (40*0.8+15*1.6) / 80＝70％

[0189] As can be seen from Table 5, the accuracy after the weighted ratio is consistent with the large data accuracy.

[0190] The sample structure is adjusted by stratified sampling, and the stratified sampling process can be referred to Figure 17 When the first data sample is extracted from the biased sampling of the large plate data, the second data sample is extracted from the first data sample, but the distribution of male and female users in the second data sample needs to be consistent with the distribution in the large plate data, and then the user gender in the second data sample is predicted by the label model to obtain the confusion matrix after stratified sampling.

[0191] The large plate accuracy, the sampling accuracy before stratified sampling, and the sampling accuracy after stratified sampling are calculated based on the confusion matrix after stratified sampling. The accuracy results after stratified sampling are shown in Table 6:

[0192] Table 6: Accuracy results after stratified sampling

[0193] Prediction Market Accuracy Sample Accuracy Before Stratification Sample Accuracy Before Stratification Male 80 / (80+20)＝80％ 40 / (40+5)＝88.9％ 20 / (20+5)＝80％ Female 60 / (60+40)＝60％ 15 / (15+20)＝14.3％ 15 / (15+10)＝60％ Gender (80+60) / 200＝70％ (40+15) / 80＝68.8％ (20+15) / 50＝70％

[0194] As shown in Table 6, the accuracy after stratified sampling is consistent with the large plate accuracy.

[0195] The label model verification method proposed in the embodiments of the present application can be applied to related business scenarios with label construction, such as advertisement label system, financial label system, product user portrait, etc. The continuous label evaluation scheme helps the label model to be effectively iterated and optimized. In the process of model iteration and optimization, the continuous label evaluation scheme is equivalent to the role of a referee, and makes two key decisions: whether the new label can be put online and handed over to the business for safe use, and whether the label in the user label system can be replaced by the old model. In addition to providing reference for key decisions, the continuous label evaluation scheme also provides direction for formulating label optimization strategies. Badcase is used to locate problems, optimize feature extraction, analyze the reasons for the decrease of label accuracy, and narrow the range of non-effective users mined by the model for targeted optimization.

[0196] The content disclosed in the embodiments of the present application solves three problems in label accuracy verification, forms a general, scientific and standardized continuous label evaluation scheme, and the solutions to the three problems are as follows:

[0197] 1. Using Panel library technology + questionnaire design + offline research collection, a scheme for contacting real users and flexibly obtaining real user information is formed to solve the problem of difficulty in obtaining real samples;

[0198] 2. A sampling scheme is formed to obtain an objective evaluation result by adjusting the sample structure through weighting, stratified sampling, etc. to solve the representativeness controversy problem caused by sampling evaluation;

[0199] 3. Forming an interpretation method for analyzing the overall optimization effect of a model, and proposing a Panel library construction scheme to support the method: analyzing the optimization effect of a model in the same group, analyzing the consistency of the optimization effect of a model in different groups, and then obtaining the overall optimization effect conclusion of the model.

[0200] The continuous label evaluation scheme provides effective decision-making for whether the label can be online for business use, helps the label model to be effectively iterated and optimized, and enables the business to use the label with confidence. Through the label, intelligent marketing, personalized recommendation, and fine operation business capabilities can be achieved.

[0201] The label model verification method disclosed by the embodiments of the present application can solve the problem that the sample data extracted from the full data does not have sampling representativeness, so that the verification result obtained based on the extracted sample data is more objective, and avoids the deviation of the verification result caused by biased sampling; and after determining the sample size, only the corresponding sample size of data items needs to be obtained to achieve the sampling effect of obtaining more than the sample size of data items, without the need to additionally obtain more data items, thereby reducing the cost of obtaining sample data and saving system resources.

[0202] The embodiments also provide a label model verification device, please refer to Figure 18 , the device can include:

[0203] The full object data acquisition module 1810 is configured to acquire full object data, and determine a label category distribution corresponding to label information of a target label in the full object data;

[0204] The object data sample generation module 1820 is configured to randomly extract a preset sample size of object data items from the full object data to generate an object data sample; wherein the preset sample size is obtained by multiple sampling of the preset object data and analysis of the sampling results of the multiple sampling;

[0205] The target confusion matrix generation module 1830 is configured to perform sample structure adjustment on the object data sample based on the label category distribution corresponding to the label information of the target label in the full object data and the target label model corresponding to the target label to obtain a target data sample, and determine a target confusion matrix obtained by predicting the label information of the target label for each object in the target data sample;

[0206] The accuracy verification module 1840 is configured to verify the accuracy of the label information predicted by the target label model based on the target confusion matrix.

[0207] Specifically, the label information of the target label corresponds to at least two label categories, and the label category distribution includes a proportion of an object quantity of each label category in a total object quantity; correspondingly, the target confusion matrix generation module 1830 includes:

[0208] a first prediction result generation module configured to predict label information of a target label of each object in the object data sample based on a target label model corresponding to the target label, to obtain a first prediction result; the first prediction result includes a predicted object quantity corresponding to each label category of the target label;

[0209] a proportion determination module configured to, for each label category corresponding to the target label, determine a first proportion of an object quantity belonging to the label category in the total object data based on the label category distribution, and determine a second proportion of the object quantity belonging to the label category in the object data sample;

[0210] a weight determination module configured to determine a weight corresponding to each label category based on the first proportion and the second proportion corresponding to the label category respectively;

[0211] a first generation module configured to obtain the target data sample based on the weight corresponding to each label category and the object data sample;

[0212] a first adjustment module configured to adjust the predicted object quantity corresponding to each label category in the first prediction result based on the weight corresponding to each label category, to obtain the target confusion matrix.

[0213] Specifically, each object data item corresponds to an object, and each object data item of each object includes a plurality of data corresponding to the object, and the plurality of data includes real label information of the target label; correspondingly, the first prediction result generation module includes:

[0214] a first prediction module configured to, for each object in the object data sample, take data other than the real label information of the target label in the plurality of data corresponding to the object as an input of the target label model, to obtain a first label prediction result of the target label model for the target label of the object;

[0215] a first statistics module configured to statistically process the first label prediction result of each object, to obtain a first confusion matrix;

[0216] a first determination module configured to determine the first confusion matrix as the first prediction result.

[0217] The first generation module includes:

[0218] The first calculation module is configured to multiply the number of real objects corresponding to each label category in the object data sample by the weight corresponding to the label category to obtain the number of real objects corresponding to each label category after weighting.

[0219] The second generation module is configured to generate the target data sample based on the objects of each label category and the number of real objects of each label category after weighting.

[0220] The first adjustment module includes:

[0221] The second calculation module is configured to multiply the number of predicted objects corresponding to each label category in the first confusion matrix by the weight corresponding to the label category to obtain a second confusion matrix.

[0222] The second determination module is configured to determine the second confusion matrix as the target confusion matrix.

[0223] The target confusion matrix generation module 1830 can further include:

[0224] The first extraction module is configured to extract a first number of object data items from the object data sample to generate the target data sample, wherein the label category distribution corresponding to the label information of the target label in the target data sample is consistent with the label category distribution corresponding to the label information of the target label in the full-amount object data.

[0225] The second prediction module is configured to, for each object in the target data sample, take the data in the multiple data corresponding to the object except the real label information of the target label as the input of the target label model to obtain the second label prediction result of the target label model for the target label of the object.

[0226] The second statistical module is configured to statistically analyze the second label prediction results of the objects to obtain a third confusion matrix.

[0227] The third determination module is configured to determine the third confusion matrix as the target confusion matrix.

[0228] The first extraction module includes:

[0229] The fourth determination module is configured to, for each label category corresponding to the target label, determine a third proportion of the number of objects belonging to the label category in the full-amount object data based on the label category distribution.

[0230] The fifth determination module is configured to determine the number of objects belonging to each label category in the target data sample based on the first number and the third proportion of the number of objects belonging to each label category in the full-amount object data.

[0231] The third generation module is configured to generate the target data sample by extracting corresponding object data items from the object data sample based on the number of objects belonging to each label category in the target data sample.

[0232] Further, the apparatus further comprises:

[0233] The group determination module is configured to determine an object data group with a sampling coincidence rate higher than a first threshold as a first group, and determine an object data group with a sampling coincidence rate lower than a second threshold as a second group.

[0234] The second extraction module is configured to randomly extract a first preset number of object data items from the first group to generate a first sample, and randomly extract a second preset number of object data items from the second group to generate a second sample.

[0235] The accuracy determination module is configured to determine a first accuracy of the target label model in predicting label information based on a target confusion matrix corresponding to the first sample, and determine a second accuracy of the target label model in predicting label information based on a target confusion matrix corresponding to the second sample.

[0236] The accuracy change trend determination module is configured to determine an accuracy change trend of the target label model in predicting label information based on the first accuracy and the second accuracy.

[0237] The apparatus can further comprise:

[0238] The abnormal data item determination module is configured to determine an object data item corresponding to a target object as an abnormal data item when the predicted label information of the target label of the target object obtained by performing target label information prediction on the target object based on the target label model is inconsistent with the real label information.

[0239] The model optimization module is configured to backtrack the abnormal data item, and optimize the target label model based on the abnormal data item.

[0240] The apparatus provided in the above embodiments can execute the method provided in any embodiment of the present application, and has the corresponding function modules and beneficial effects of executing the method. Technical details not described in the above embodiments can be referred to the method provided in any embodiment of the present application.

[0241] The present embodiment further provides a computer readable storage medium, the storage medium stores at least one instruction or at least one program, the at least one instruction or the at least one program is loaded and executed by a processor as any method described above in the present embodiment.

[0242] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the methods described in this embodiment.

[0243] This embodiment also provides a device, the structural diagram of which can be found in the following figure. Figure 19 The device 1900 can vary considerably depending on its configuration or performance, and may include one or more central processing units (CPUs) 1922 (e.g., one or more processors) and memory 1932, and one or more storage media 1930 (e.g., one or more mass storage devices) for storing applications 1942 or data 1944. The memory 1932 and storage media 1930 may be temporary or persistent storage. Programs stored in the storage media 1930 may include one or more modules (not shown), each module including a series of instruction operations on the device. Furthermore, the CPU 1922 may be configured to communicate with the storage media 1930 and execute the series of instruction operations in the storage media 1930 on the device 1900. The device 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input / output interfaces 1958, and / or one or more operating systems 1941, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Etc. Any of the methods described above in this embodiment can be based on... Figure 19 The equipment shown is used for implementation.

[0244] This specification provides the operational steps of the methods described in the embodiments or flowcharts, but more or fewer operational steps may be included based on conventional or non-inventive labor. The steps and order listed in the embodiments are merely one possible execution order among many steps and do not represent the only execution order. In actual system or interrupt product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0245] The structure shown in this embodiment is only a partial structure related to the solution of this application and does not constitute a limitation on the device to which the solution of this application is applied. Specific devices may include more or fewer components than shown, or combinations of certain components, or arrangements of different components. It should be understood that the methods, apparatuses, etc., disclosed in this embodiment can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or unit modules through some interfaces.

[0246] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0247] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0248] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A label model verification method, characterized in that, include: Obtain all object data and determine the tag category distribution corresponding to the tag information of the target tag in the all object data; the tag category corresponding to the tag information of the target tag is at least two, and the tag category distribution includes the proportion of the number of objects of each tag category in the total number of objects; A preset sample size of object data items is randomly extracted from the full object data to generate object data samples; wherein, the preset sample size is obtained by sampling the preset object data multiple times and analyzing the sampling results of the multiple samplings. Based on the target label model corresponding to the target label, the label information of the target label of each object in the object data sample is predicted to obtain a first prediction result; the first prediction result includes the number of predicted objects corresponding to each label category of the target label; For each tag category corresponding to the target tag, a first proportion of the number of objects belonging to the tag category in the full object data is determined based on the tag category distribution, and a second proportion of the number of objects belonging to the tag category in the object data sample is determined. The weight corresponding to each label category is determined based on the first ratio and the second ratio corresponding to each label category. Based on the weights corresponding to each label category and the object data samples, the target data samples are obtained; Based on the weights corresponding to each label category, the number of predicted objects corresponding to each label category in the first prediction result is adjusted to obtain the target confusion matrix; Based on the target confusion matrix, the accuracy of the target label model in predicting label information is verified.

2. The label model verification method according to claim 1, characterized in that, Each object data item corresponds to one object, and each object data item includes multiple data items corresponding to the object, including the actual tag information of the target tag; The step of predicting the label information of the target labels of each object in the object data sample based on the target label model corresponding to the target label, and obtaining the first prediction result, includes: For each object in the object data sample, the data in the multiple data corresponding to the object, excluding the real label information of the target label, is used as the input of the target label model to obtain the first label prediction result of the target label model for the object; The first label prediction results for each object are statistically analyzed to obtain the first confusion matrix; The first confusion matrix is ​​determined as the first prediction result.

3. The label model verification method according to claim 2, characterized in that, The process of obtaining the target data sample based on the weights corresponding to each label category and the object data sample includes: Multiply the number of real objects corresponding to each label category in the object data sample by the weight corresponding to the corresponding label category to obtain the weighted number of real objects corresponding to each label category. The target data sample is generated based on the objects in each tag category and the weighted number of real objects in the corresponding tag category. The step of adjusting the number of predicted objects corresponding to each label category in the first prediction result based on the weights corresponding to each label category to obtain the target confusion matrix includes: Multiply the number of predicted objects corresponding to each label category in the first confusion matrix by the weight corresponding to each label category to obtain the second confusion matrix; The second confusion matrix is ​​determined as the target confusion matrix.

4. The label model verification method according to claim 1, characterized in that, Each object data item corresponds to one object, and each object data item includes multiple data items corresponding to the object, including the actual tag information of the target tag; The method further includes: A first number of object data items are extracted from the object data sample to generate the target data sample, wherein the label category distribution corresponding to the label information of the target label in the target data sample is consistent with the label category distribution corresponding to the label information of the target label in the full object data. For each object in the target data sample, the data in the multiple data corresponding to the object, excluding the real label information of the target label, is used as the input of the target label model to obtain the second label prediction result of the target label model for the object. The second label prediction results for each object are statistically analyzed to obtain the third confusion matrix; The third confusion matrix is ​​determined as the target confusion matrix.

5. The label model verification method according to claim 4, characterized in that, The label category distribution includes the proportion of objects in each label category to the total number of objects; The step of extracting a first number of object data items from the object data sample to generate the target data sample includes: For each tag category corresponding to the target tag, a third proportion is determined based on the tag category distribution of the total number of objects in the full object data that belong to the tag category; Based on the first quantity and the third proportion of the number of objects belonging to each tag category in the full object data, the number of objects belonging to each tag category in the target data sample is determined. Based on the number of objects belonging to each label category in the target data sample, corresponding object data items are extracted from the object data sample to generate the target data sample.

6. The label model verification method according to claim 1, characterized in that, The method further includes: The data group with a sampling overlap rate higher than the first threshold is identified as the first group, and the data group with a sampling overlap rate lower than the second threshold is identified as the second group. A first preset number of object data items are randomly selected from the first group to generate a first sample; a second preset number of object data items are randomly selected from the second group to generate a second sample. Based on the target confusion matrix corresponding to the first sample, a first accuracy rate of the target label model in predicting label information is determined, and based on the target confusion matrix corresponding to the second sample, a second accuracy rate of the target label model in predicting label information is determined. Based on the first accuracy rate and the second accuracy rate, the trend of the accuracy change of the target label model in predicting label information is determined.

7. The label model verification method according to claim 1, characterized in that, The method further includes: When the predicted label information of the target object is not consistent with the real label information when the target label information is predicted based on the target label model, the object data item corresponding to the target object is determined to be an abnormal data item. The abnormal data items are traced back, and the target label model is optimized based on the abnormal data items.

8. A label model verification device, characterized in that, include: The full object data acquisition module is used to acquire full object data and determine the tag category distribution corresponding to the tag information of the target tag in the full object data; the tag category corresponding to the tag information of the target tag is at least two, and the tag category distribution includes the proportion of the number of objects of each tag category in the total number of objects; The object data sample generation module is used to randomly extract a preset number of object data items from the full object data to generate object data samples; wherein, the preset number of samples is obtained by sampling the preset object data multiple times and analyzing the sampling results of the multiple samplings. A target confusion matrix generation module is used to adjust the sample structure of the object data samples to obtain target data samples based on the label category distribution corresponding to the label information of the target labels in the full object data and the target label model corresponding to the target labels, and to determine the target confusion matrix obtained by predicting the label information of the target labels for each object in the target data samples; the target confusion matrix generation module includes: A first prediction result generation module is used to predict the label information of the target label of each object in the object data sample based on the target label model corresponding to the target label, and obtain a first prediction result; the first prediction result includes the number of predicted objects corresponding to each label category of the target label; a proportion determination module is used to determine, for each label category corresponding to the target label, a first proportion of the number of objects belonging to the label category in the full object data and a second proportion of the number of objects belonging to the label category in the object data sample based on the label category distribution; a weight determination module is used to determine the weight corresponding to the label category based on the first proportion and the second proportion corresponding to each label category; a first generation module is used to obtain the target data sample based on the weight corresponding to each label category and the object data sample; a first adjustment module is used to adjust the number of predicted objects corresponding to each label category in the first prediction result based on the weight corresponding to each label category, and obtain the target confusion matrix; The accuracy verification module is used to verify the accuracy of the target label model's prediction of label information based on the target confusion matrix.

9. The apparatus according to claim 8, characterized in that, Each object data item corresponds to an object, and each object's object data item includes multiple data items corresponding to the object, including the true label information of the target label; correspondingly, the first prediction result generation module includes: The first prediction module is used to take the data other than the real label information of the target label from the multiple data corresponding to the object in the object data sample as the input of the target label model for each object, and obtain the first label prediction result of the target label model for the target label of the object. The first statistical module is used to statistically analyze the first label prediction results of each object and obtain the first confusion matrix; The first determining module is used to determine the first confusion matrix as the first prediction result.

10. The apparatus according to claim 9, characterized in that, The first generation module includes: The first calculation module is used to multiply the number of real objects corresponding to each label category in the object data sample by the weight corresponding to the corresponding label category to obtain the weighted number of real objects corresponding to each label category. The second generation module is used to generate the target data sample based on the objects of each tag category and the weighted number of real objects of the corresponding tag category; The first adjustment module includes: The second calculation module is used to multiply the number of predicted objects corresponding to each label category in the first confusion matrix by the weight corresponding to each label category to obtain the second confusion matrix. The second determining module is used to determine the second confusion matrix as the target confusion matrix.

11. The apparatus according to claim 8, characterized in that, Each object data item corresponds to one object, and each object data item includes multiple data items corresponding to the object, including the actual tag information of the target tag; The target confusion matrix generation module includes: The first extraction module is used to extract a first number of object data items from the object data sample to generate the target data sample, wherein the label category distribution corresponding to the label information of the target label in the target data sample is consistent with the label category distribution corresponding to the label information of the target label in the full object data. The second prediction module is used to take the data other than the real label information of the target label from the multiple data corresponding to the object as the input of the target label model for each object in the target data sample, and obtain the second label prediction result of the target label model for the target label of the object. The second statistical module is used to statistically analyze the second label prediction results of each object and obtain the third confusion matrix; The third determining module is used to determine the third confusion matrix as the target confusion matrix.

12. The apparatus according to claim 11, characterized in that, The first extraction module includes: The fourth determining module is used to determine, for each tag category corresponding to the target tag, a third proportion of the number of objects belonging to the tag category in the full object data based on the tag category distribution; The fifth determining module is used to determine the number of objects belonging to each tag category in the target data sample based on the first quantity and the third proportion of the number of objects belonging to each tag category in the full object data. The third generation module is used to extract corresponding object data items from the object data sample based on the number of objects belonging to each label category in the target data sample, and generate the target data sample.

13. The apparatus according to claim 8, characterized in that, The device further includes: The population determination module is used to determine the first population of object data groups whose sampling overlap rate is higher than a first threshold, and to determine the second population of object data groups whose sampling overlap rate is lower than a second threshold. The second extraction module is used to randomly extract a first preset number of object data items from the first group to generate a first sample, and to randomly extract a second preset number of object data items from the second group to generate a second sample. An accuracy determination module is used to determine a first accuracy of the target label model predicting label information based on the target confusion matrix corresponding to the first sample, and to determine a second accuracy of the target label model predicting label information based on the target confusion matrix corresponding to the second sample. The accuracy change trend determination module is used to determine the accuracy change trend of the target label model predicting label information based on the first accuracy and the second accuracy.

14. The apparatus according to claim 8, characterized in that, The device further includes: The abnormal data item determination module is used to determine the object data item corresponding to the target object as an abnormal data item when the predicted label information of the target object is not consistent with the real label information after predicting the target label information of the target object based on the target label model. The model optimization module is used to backtrack the abnormal data items and optimize the target label model based on the abnormal data items.

15. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the label model verification method as described in any one of claims 1 to 7.

16. A computer storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor according to any one of claims 1 to 7.

17. A computer program product or computer program, characterized in that, The computer program product or computer program includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the label model verification method as described in any one of claims 1 to 7.

Images