Method, device, computer equipment and storage medium for verifying labeled data

Abstract

The present application discloses a verification method, device, computer equipment and storage medium for annotated data, the method comprising: obtaining M groups of first emotion annotation data for the same annotated sample set; determining the first annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in N groups of second emotion annotation data; determining the second annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each group of other first emotion annotation data in M ​​groups of first emotion annotation data, the consistency weight according to which the weighted consistency score is based is determined by the correlation between the annotated emotion tags; verifying each group of first emotion annotation data based on the first annotation score and the second annotation score corresponding to each group of first emotion annotation data. The method can realize accurate verification of emotion annotation data.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a method, apparatus, computer equipment, and storage medium for verifying labeled data. Background Art

[0002] With the rapid advancement of science and technology, deep learning involving sentiment-labeled data has attracted significant research interest and is being deployed in numerous applications. For example, sentiment-labeled data is used to train speech dialogue models and image evaluation models. However, sentiment labeling of sample data is susceptible to the subjective and personalized nature of the labeler, which in turn affects the accuracy of the labeling data. Summary of the Invention

[0003] This application proposes a method, device, computer equipment and storage medium for verifying labeled data, which can achieve accurate verification of emotion labeled data.

[0004] In a first aspect, an embodiment of the present application provides a method for verifying labeled data, the method comprising: obtaining M groups of first emotion labeling data for the same labeled sample set, wherein the labeling users of each group of first emotion labeling data are different, and M is a positive integer; determining a first labeling score corresponding to each group of first emotion labeling data based on a weighted consistency score between each group of first emotion labeling data and each group of second emotion labeling data in N groups of second emotion labeling data, wherein the second emotion labeling data is labeling data in a standard labeling database, and there are at least partially identical labeling samples between the second emotion labeling data and the first emotion labeling data, and N is a positive integer; determining a second labeling score corresponding to each group of first emotion labeling data based on a weighted consistency score between each group of first emotion labeling data and each other group of first emotion labeling data in the M groups of first emotion labeling data, wherein the consistency weight according to which the weighted consistency score is determined is determined by the correlation between the labeled emotion tags; verifying each group of first emotion labeling data based on the first labeling score and the second labeling score corresponding to each group of first emotion labeling data.

[0005] In the second aspect, an embodiment of the present application provides a verification device for labeled data, the device comprising: a data acquisition module, a first scoring module, a second scoring module and a data verification module, wherein the data acquisition module is used to obtain M groups of first emotion labeling data for the same labeled sample set, wherein each group of first emotion labeling data is labeled by a different user, and M is a positive integer; the first scoring module is used to determine the first annotation score corresponding to each group of first emotion labeling data based on the weighted consistency score between each group of first emotion labeling data and each group of second emotion labeling data in N groups of second emotion labeling data, and the second emotion labeling data is a label in a standard annotation database. annotation data, and there are at least partially identical annotation samples between the second emotion annotation data and the first emotion annotation data, and N is a positive integer; the second scoring module is used to determine the second annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each group of other first emotion annotation data in the M groups of first emotion annotation data, and the consistency weight based on which the weighted consistency score is determined by the correlation between the annotated emotion tags; the second scoring module is used to verify each group of first emotion annotation data based on the first annotation score and the second annotation score corresponding to each group of first emotion annotation data.

[0006] In a third aspect, an embodiment of the present application provides a computer device comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the verification method for the labeled data provided in the first aspect above.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which program code is stored. The program code can be called by a processor to execute the verification method of the labeled data provided in the first aspect above.

[0008] The solution provided by the present application is to obtain M groups of first emotion annotation data for the same annotation sample set, and then determine the first annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in the N groups of second emotion annotation data in the standard annotation database, and there are at least partially identical annotation samples between the second emotion annotation data and the first emotion annotation data, and determine the second annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each other group of first emotion annotation data in the M groups of first emotion annotation data, and the consistency weight based on which the weighted consistency score is determined is determined by the correlation between the labeled emotion tags, and then verify each group of first emotion annotation data based on the first annotation score and the second annotation score corresponding to each group of first emotion annotation data. Since the first annotation score and the second annotation score are both obtained by determining the weighted consistency score based on the consistency weight, and the weighted consistency weight is determined by the correlation between the emotion tags, the accuracy of the first annotation score and the second annotation score can be improved, so that when the emotion annotation data is verified using the first annotation score and the second annotation score, the verification accuracy can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A flow chart of a method for verifying labeled data according to an embodiment of the present application is shown.

[0011] Figure 2 A flow chart of a method for verifying labeled data according to another embodiment of the present application is shown.

[0012] Figure 3 A flow chart of a method for verifying labeled data according to another embodiment of the present application is shown.

[0013] Figure 4 A flow chart of a method for verifying labeled data according to yet another embodiment of the present application is shown.

[0014] Figure 5 A flow chart of a method for verifying labeled data according to yet another embodiment of the present application is shown.

[0015] Figure 6 A block diagram of a device for verifying labeled data according to an embodiment of the present application is shown.

[0016] Figure 7 It is a block diagram of a computer device for executing the verification method of labeled data according to an embodiment of the present application.

[0017] Figure 8 It is a storage unit for storing or carrying program codes for implementing the verification method for the labeled data according to the embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0019] Subjective data annotation, which involves collecting data related to people's subjective feelings, attitudes, and opinions, is fundamental to the research of algorithms such as visual understanding and semantic understanding, and is key to ensuring that the algorithm's results meet users' subjective satisfaction. Subjective data annotation tasks are highly subjective and personalized, and there is no single standard answer. This poses challenges to controlling the quality of the labeled data. Quality control for subjective annotation tasks must be able to avoid mislabeling and misclassification while preserving reasonable subjective differences.

[0020] In traditional methods of verifying labeled data, the labeled data is usually matched with expert labeled data to determine whether the labeled data is approved, or the label with the most annotations is selected as the final annotation result for the same labeled sample. However, in subjective annotation tasks for images, videos, and voices, labeling samples with emotional component labels, that is, the subjective emotions evoked by the samples, is an important annotation scenario. Emotional classification labels cannot be converted into quantitative data for analysis, nor can they be simply regarded as independent unordered categorical variables (for example, the two emotion categories of excitement and happiness are closer in semantic distance than the two labels of happiness and anger). Therefore, traditional verification schemes cannot handle emotional annotation scenarios well, resulting in insufficient verification accuracy of emotional annotation data.

[0021] In response to the above problems, the inventors have proposed the verification method, device, computer equipment, and storage medium for the labeled data provided in the embodiments of the present application. Since the first and second labeling scores obtained during the verification process are obtained by determining weighted consistency scores based on consistency weights, and the weighted consistency weights are determined by the correlation between the sentiment tags, the accuracy of the first and second labeling scores can be improved, thereby improving the verification accuracy when the sentiment labeling data is verified using the first and second labeling scores. The specific verification method for labeled data will be described in detail in the subsequent embodiments.

[0022] The following is a detailed introduction to the verification method of the annotation data provided in the embodiment of the present application in conjunction with the accompanying drawings.

[0023] See also Figure 1 , Figure 1 A flow chart of a method for verifying the labeled data provided by an embodiment of the present application is shown. In a specific embodiment, the method for verifying the labeled data is applied to Figure 6 The verification device 400 for the labeled data and the computer device 100 ( Figure 7 ). The following will take a computer device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the computer device used in this embodiment can be a server, PC, laptop, etc., and is not limited here. Figure 1 The process shown is described in detail. The verification method of the labeled data may specifically include the following steps:

[0024] Step S110: Obtain M groups of first emotion annotation data for the same annotated sample set, wherein each group of first emotion annotation data is annotated by a different user, and M is a positive integer.

[0025] In an embodiment of the present application, a computer device can obtain M groups of first emotion annotation data for the same annotation sample set by an annotation user, and the annotation user of each group of first emotion annotation data is different. The sample data of the annotation sample set can be sample data of the image, audio and other types, and the emotion annotation data is the data obtained after the sample data in the annotation sample set are annotated with emotion labels. Annotating emotion labels is to label the emotion or mood evoked by the sample data with an emotion label, that is, the emotion label is used to characterize the emotion or mood evoked by the sample data. For example, the emotion label may include: excitement, happiness, sadness, depression and other emotion-related labels. Optionally, the emotion annotation in the embodiment of the present application may also be an annotation of other types of subjective classification labels, such as synesthesia labeling, intention labeling, scene labeling and preference labeling.

[0026] In some embodiments, the computer device may divide the sample data to be labeled into multiple data packets, each data packet being a labeled sample set; then generate a labeling task based on the labeled sample set, and send the labeling task to the clients corresponding to multiple labeling users; after the labeling user completes the labeling of the labeled sample set, the labeling user's client then feeds back to the computer device, so that the computer device can obtain the labeled sample set from the labeling user. Each labeled sample set is labeled by M labeling users, that is, the labeling task corresponding to the same labeled sample set is sent to the clients corresponding to M labeling users, and thus, for the same labeled sample set, the computer device can obtain M groups of first emotion labeling data. It should be noted that a labeling task can include one labeled sample set or multiple labeled sample sets, that is, each labeling user can label one labeled sample set or multiple labeled sample sets. It only needs to ensure that the same labeled sample set is labeled by multiple labeling users, and this is not limited here.

[0027] In some embodiments, M can be greater than a preset number of users. That is, the number of groups of sentiment annotation data obtained by the computer device for the same annotated sample set should be greater than the preset number of users. This ensures that the subsequent determination of the consistency score of the sentiment annotation data for the same annotated sample set will result in an accurate annotation score. The preset number of users can be 4, 5, 6, 7, etc., and the specific number is not limited.

[0028] Step S120: Determine the first annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in N groups of second emotion annotation data, where the second emotion annotation data is annotation data in a standard annotation database, and there are at least partially identical annotation samples between the second emotion annotation data and the first emotion annotation data, and N is a positive integer.

[0029] In an embodiment of the present application, second emotion annotation data can be stored in advance in a standard annotation database, and the annotation samples corresponding to the second emotion annotation data are at least partially identical to the annotation samples corresponding to the first emotion annotation data. Among them, the emotion annotation data in the standard annotation database is the annotation data that has been pre-annotated and verified for the annotation samples. For example, it can be the annotation data that has been annotated and verified by a qualified annotator. Since there are at least partially identical annotation samples between the second emotion annotation data and the first emotion annotation data, the first emotion annotation data can be verified based on the second emotion annotation data to achieve the purpose of random inspection. It can be understood that in the scenario of crowdsourcing emotion labeling, the number of samples to be annotated is huge, and it is impossible to recruit qualified annotators for all samples to be annotated and annotate all samples under controllable conditions. Therefore, multiple groups of second emotion annotation data can be stored in the standard annotation database in advance, and the two emotion annotation data can be mixed into all samples to be annotated and distributed to other annotation users. The annotation samples corresponding to the second emotion annotation data in the standard annotation database are equivalent to random inspection questions with standard answers, which can check the annotation quality problems of the annotation users.

[0030] Among them, based on multiple groups of second emotion annotation data in the standard annotation database, when verifying the first emotion annotation data annotated by different users on the annotated sample set, the first annotation score corresponding to each group of first emotion annotation data can be determined based on the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in the N groups of second emotion annotation data. It can be understood that the consistency score between each group of first emotion annotation data and any group of second emotion annotation data represents the consistency of the annotation opinions of the corresponding two people. The larger the consistency score, the more unified the annotation opinions of the two people, that is, the first emotion annotation data to be verified is more matched with the standard second emotion annotation data, and the accuracy of the first emotion annotation data is also higher; in addition, since the emotion labels may have a strong correlation in emotion annotation, for example, the emotion label of excitement has a high correlation with the emotion label of happiness, while excitement has a low correlation with disgust. Therefore, when calculating the consistency score between each group of first emotion annotation data and each group of second emotion annotation data, the consistency weight can be determined according to the correlation between the emotion labels, and the consistency score calculated according to the consistency weight is more suitable for the emotion annotation scenario.

[0031] In some embodiments, when determining the weighted consistency score between each set of first emotion annotation data and each set of second emotion annotation data, the computer device may calculate a weighted consistency score based on the emotion tags tagged in each set of first emotion annotation data and each set of second emotion annotation data, and the consistency weights between the emotion tags. The weighted consistency score may be a weighted Kendall rank correlation coefficient, a weighted Kappa correlation coefficient, or the like.

[0032] In some embodiments, the consistency weight should ensure that the confusion between emotion labels with different correlations has different effects on the consistency calculation, wherein the higher the correlation of the emotion label, the smaller the effect of the confusion between the labels on the consistency calculation, that is, the lower the consistency weight. It can be understood that the higher the correlation between different emotion labels, the easier it is to confuse them, so the weight should be reduced during the consistency calculation to improve the accuracy of the determined weighted consistency score. For example, the emotion label corresponding to happiness has a higher correlation with the emotion label corresponding to excitement, and the emotion label corresponding to happiness has a lower correlation with the emotion label corresponding to disgust. Then, labeling the happy sample as excitement has a lower effect on the consistency, while labeling it as disgust has a higher effect on the consistency. Therefore, the consistency weight between the emotion label corresponding to happiness and the emotion label corresponding to excitement is smaller than the consistency weight between the emotion label corresponding to happiness and the emotion label corresponding to disgust.

[0033] In some embodiments, when determining the first annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in N groups of second emotion annotation data, the average of multiple weighted consistency scores corresponding to each group of first emotion annotation data can be obtained as the first annotation score; or the sum of multiple weighted consistency scores corresponding to each group of first emotion annotation data can be obtained as the first annotation score. The specific calculation method may not be limited.

[0034] Step S130: Based on the weighted consistency score between each group of first emotion annotation data and each other group of first emotion annotation data in the M groups of first emotion annotation data, determine the second annotation score corresponding to each group of first emotion annotation data, and the consistency weight based on which the weighted consistency score is determined by the correlation between the annotated emotion tags.

[0035] In an embodiment of the present application, when verifying the above-mentioned M groups of first emotion annotation data, when obtaining the corresponding second annotation score for any group of first emotion annotation data, the weighted consistency score between the group of first emotion annotation data and each group of other first emotion annotation data in the M groups of first emotion annotation data except the group of first emotion annotation data can be obtained. For example, the M groups of first emotion annotation data include: group a emotion annotation data, group b emotion annotation data, group c emotion annotation data, group d emotion annotation data and group e emotion annotation data. Then, when obtaining the corresponding second annotation score for group a emotion annotation data, the weighted consistency score between group a emotion annotation data and group b emotion annotation data, the weighted consistency score between group a emotion annotation data and group c emotion annotation data, the weighted consistency score between group a emotion annotation data and group d emotion annotation data, and the weighted consistency score between group a emotion annotation data and group e emotion annotation data can be obtained; then, based on the weighted consistency score between each group of first emotion annotation data and each group of other first emotion annotation data, the second annotation score corresponding to each group of first emotion annotation data is determined.

[0036] It can be understood that the consistency score between each group of first emotion annotation data and any other group of first emotion annotation data represents the consistency of the corresponding two people's annotation opinions. The larger the consistency score, the more unified the annotation opinions of the two people, that is, the first emotion annotation data to be verified is more matched with the standard second emotion annotation data. If the consistency score between any group of first emotion annotation data and other first emotion annotation data is large, the annotation opinions corresponding to the group of first emotion annotation data are more unified with the opinions of the majority of users, and the accuracy is higher; in addition, the same as step S120, when calculating the consistency score between each group of first emotion annotation data and each group of other first emotion annotation data, the consistency weight can be determined according to the correlation between the emotion tags, and the consistency score calculated according to the consistency weight is more suitable for the emotion annotation scenario.

[0037] In some embodiments, when determining the second annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each group of other first emotion annotation data, the average of the weighted consistency scores between each group of first emotion annotation data and each group of other first emotion annotation data can be obtained as the second annotation score; or the sum of the weighted consistency scores between each group of first emotion annotation data and each group of other first emotion annotation data can be obtained as the second annotation score. The specific calculation method may not be limited.

[0038] Step S140: verifying each set of first emotion annotation data based on the first annotation score and the second annotation score corresponding to each set of first emotion annotation data.

[0039] In an embodiment of the present application, the first annotation score corresponding to each group of first emotion annotation data is determined based on the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in the N groups of second emotion annotation data in the standard annotation database, so the first annotation score can reflect the accuracy of the standard annotation data for the verification of each group of first emotion annotation data; and the second annotation score is determined based on the weighted consistency score between each group of first emotion annotation data and each group of other first emotion annotation data corresponding to the same annotation sample set, so the second annotation score can reflect the accuracy of the verification of the group of first emotion annotation data by other first emotion annotation data for the same annotation sample set. Therefore, after obtaining the first annotation score and the second annotation score corresponding to each group of first emotion annotation data, the computer device can determine whether each group of first emotion annotation data is qualified based on the first annotation score and the second annotation score corresponding to each group of first emotion annotation data.

[0040] In some embodiments, if the first annotation score corresponding to the target emotion annotation data is greater than the first threshold, and the second annotation score is greater than the second threshold, the target emotion annotation data is determined to be qualified, and the target emotion annotation data is any group of emotion annotation data in the M groups of first emotion annotation data; if the first annotation score corresponding to the target emotion annotation data is less than or equal to the first threshold, and / or the second annotation score is less than or equal to the second threshold, the target emotion annotation data is determined to be unqualified.

[0041] In this embodiment, the computer device can determine whether each set of first emotion annotation data is qualified in the above manner. It can be understood that if the first annotation score is greater than the first threshold and the second annotation score is greater than the second threshold, it means that the annotation opinions of the target emotion annotation data and the emotion annotation data in the standard annotation database are relatively consistent, and the annotation opinions of the target emotion annotation data and other emotion annotation data for the same annotation sample set are also relatively consistent, so it can be determined to be qualified; otherwise, it can be determined to be unqualified.

[0042] In this embodiment, the first threshold value can be determined according to the calculation method of the first annotation score, and similarly, the second threshold value can be determined according to the calculation method of the second annotation score. Specifically, if the first annotation score is the average of multiple weighted consistency scores, the first threshold value is determined according to the empirical value of the weighted consistency score. For example, when the weighted consistency score is the Kappa correlation coefficient, -1 indicates complete inconsistency, 0 indicates accidental consistency, 0.0 to 0.20 indicates extremely low consistency, 0.21 to 0.40 indicates general consistency, 0.41 to 0.60 indicates moderate consistency, 0.61 to 0.80 indicates high consistency, and 0.81 to 1 indicates almost complete consistency. The first threshold value can be 0.41 to 0.60; similarly, if the second annotation score is the first annotation score, If the first annotation score is the average of multiple weighted consistency scores, the second threshold can also be determined according to the empirical value of the weighted consistency score; if the first annotation score is the sum of multiple weighted consistency scores, the first threshold can be the product of the number of multiple weighted consistency scores and the first threshold in the above-mentioned average case. For example, when the weighted consistency score is the Kappa correlation coefficient, the number of multiple weighted consistency scores is N, and the first threshold can be N*0.5; if the first annotation score is the sum of multiple weighted consistency scores, the second threshold can be the product of the number of multiple weighted consistency scores and the second threshold in the above-mentioned average case.

[0043] The verification method for labeled data provided in the embodiment of the present application determines the uniformity of the labeling opinions between each group of first emotion labeling data and each group of second emotion labeling data in the standard labeling database (i.e., the first labeling score) when verifying M groups of first emotion labeling data of the same labeled sample set, and also determines the uniformity of the labeling opinions between each group of first emotion labeling data and other emotion labeling data for the same labeled sample set (i.e., the second labeling score). Therefore, the first labeling score and the second labeling score are used simultaneously to effectively judge the labeling quality, which can improve the verification accuracy and obtain subjective labeling data that is objectively representative and universal; and since the first labeling score and the second labeling score are both obtained by determining the weighted consistency score based on the consistency weight, and the weighted consistency weight is determined by the correlation between the emotion tags, the accuracy of the first labeling score and the second labeling score can be improved, so that when the first labeling score and the second labeling score are used to verify the emotion labeling data, the verification accuracy can be further improved.

[0044] See also Figure 2 , Figure 2 A flow chart of a method for verifying the labeled data provided by another embodiment of the present application is shown. The method for verifying the labeled data is applied to the above-mentioned computer device. Figure 2The process shown is described in detail. The verification method of the labeled data may specifically include the following steps:

[0045] Step S210: Obtain M groups of first emotion annotation data for the same annotated sample set, wherein each group of first emotion annotation data is annotated by a different user, and M is a positive integer.

[0046] In the embodiment of the present application, step S210 can refer to the content of the aforementioned embodiment and will not be repeated here.

[0047] Step S220: Based on the second emotion annotation data in the standard annotation database, determine the number of times each sample is annotated with each emotion tag.

[0048] In an embodiment of the present application, when determining the consistency weight between emotion labels, the number of times each sample is labeled with each emotion label can be determined based on the second emotion labeling data in the standard labeling database, so as to determine the correlation coefficient between each two emotion labels based on the number of times each sample is labeled with each emotion label. It can be understood that for the same sample, since the emotion label is subjective, it may be labeled as label A or label B in the standard labeling database, but label A and label B are relatively related (relatively similar). For example, the emotion labels corresponding to "happy" and "excited" are relatively related. Therefore, for each sample, the frequency of each emotion label being selected can be counted so as to determine the correlation coefficient between each two emotion labels based on this.

[0049] Step S230: determining the consistency weight between each two emotion tags based on the number of times each emotion tag corresponding to each sample is labeled.

[0050] In some embodiments, after obtaining the number of times each sample is labeled with each emotion tag, the correlation coefficient between each two emotion tags can be determined based on the number of times each sample is labeled with each emotion tag; then, based on the correlation coefficient between each two emotion tags, the consistency weight between each two emotion tags is determined, and the consistency weight is negatively correlated with the correlation coefficient.

[0051] Among them, for emotion label i and emotion label j, when the same samples can be counted, the number of times emotion label i is marked is recorded as e i , the number of times the sentiment label j is marked is recorded as e j , and then calculate the correlation coefficient between each two emotional tags based on the correlation coefficient formula. The correlation coefficient formula is as follows:

[0052]

[0053] r ijis the correlation coefficient between emotion label i and emotion label j, cov(e i , e j ) is e i With e j The covariance of for e i The standard deviation of for e j The standard deviation of .

[0054] After obtaining the correlation coefficient between the emotion tag i and the emotion tag j, the consistency weight between the emotion tag i and the emotion tag j can be determined by the correlation coefficient. ij The calculation formula is as follows:

[0055]

[0056] It can be understood that since the consistency weight is negatively correlated with the correlation coefficient, the consistency weight obtained by this formula can ensure that the confusion between emotional labels with different correlations has different effects on the consistency calculation. Among them, the higher the correlation of the emotional label, the smaller the impact of the confusion between the labels on the consistency calculation, that is, the lower the consistency weight. For example, the emotional label corresponding to happiness has a high correlation with the emotional label corresponding to excitement, and the emotional label corresponding to happiness has a low correlation with the emotional label corresponding to disgust. Then, labeling the happy sample as excitement has a low impact on consistency, while labeling it as disgust has a high impact on consistency. Therefore, the consistency weight between the emotional label corresponding to happiness and the emotional label corresponding to excitement is smaller than the consistency weight between the emotional label corresponding to happiness and the emotional label corresponding to disgust.

[0057] Step S240: Determine the first annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in N groups of second emotion annotation data, where the second emotion annotation data is annotation data in a standard annotation database, and there are at least partially identical annotation samples between the second emotion annotation data and the first emotion annotation data, and N is a positive integer.

[0058] Step S250: Based on the weighted consistency score between each group of first emotion annotation data and each other group of first emotion annotation data in the M groups of first emotion annotation data, determine the second annotation score corresponding to each group of first emotion annotation data, and the consistency weight based on which the weighted consistency score is determined by the correlation between the annotated emotion tags.

[0059] Step S260: verifying each set of first emotion annotation data based on the first annotation score and the second annotation score corresponding to each set of first emotion annotation data.

[0060] In the embodiment of the present application, steps S240 to S260 can refer to the contents of other embodiments and will not be repeated here.

[0061] The verification method for labeled data provided in the embodiment of the present application provides a method for obtaining the consistency weight between different emotion tags of emotion classification, by determining the number of times each emotion tag corresponding to each sample in the standard labeling database is labeled, and based on this, determining the correlation coefficient between each two emotion tags, and then determining the consistency weight between each two emotion tags based on the correlation coefficient between each two emotion tags, so that the obtained consistency weight can ensure that the confusion between emotion tags with different correlations has different effects on the consistency calculation, so that when the weighted consistency score is further determined according to the determined consistency weight to obtain the first labeling score and the second labeling score, the accuracy of the first labeling score and the second labeling score can be improved, and then when the first labeling score and the second labeling score are used to verify the emotion labeling data, the verification accuracy can be further improved.

[0062] See also Figure 3 , Figure 3 A flow chart of a method for verifying the labeled data provided by another embodiment of the present application is shown. The method for verifying the labeled data is applied to the above-mentioned computer device. Figure 3 The process shown is described in detail. The verification method of the labeled data may specifically include the following steps:

[0063] Step S310: Obtain M groups of first emotion annotation data for the same annotated sample set, wherein each group of first emotion annotation data is annotated by a different user, and M is a positive integer.

[0064] In the embodiment of the present application, step S310 can refer to the contents of other embodiments and will not be repeated here.

[0065] Step S320: respectively obtaining a weighted consistency score between each group of the first emotion annotation data and each group of the second emotion annotation data in the N groups of the second emotion annotation data, and obtaining N consistency scores corresponding to each group of the first emotion annotation data.

[0066] The second emotion annotation data is annotation data in a standard annotation database, and there are at least partially identical annotation samples between the second emotion annotation data and the first emotion annotation data, and N is a positive integer.

[0067] In an embodiment of the present application, when the computer device determines the first annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in N groups of second emotion annotation data, it can respectively obtain the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in N groups of second emotion annotation data, thereby obtaining N consistency scores corresponding to each group of first emotion annotation data.

[0068] In some embodiments, the weighted consistency score can be a weighted Kappa correlation coefficient, and the computer device can generate N contingency tables corresponding to each group of first emotion annotation data based on each group of first emotion annotation data and each group of second emotion annotation data in N groups of second emotion annotation data; based on the N contingency tables corresponding to each group of first emotion annotation data, and the consistency weight, obtain the weighted Kappa correlation coefficient between each group of first emotion annotation data and each group of second emotion annotation data, and obtain N consistency scores corresponding to each group of first emotion annotation data, and the consistency weight is determined by the correlation between the labeled emotion tags.

[0069] The contingency table is a frequency distribution table of the cross-classification of two sets of observation data. For example, each set of the first emotion annotation data and the second emotion annotation data contains 20 pictures, and the emotion labels include: excitement, happiness, and sadness. The generated contingency table example is:

[0070]

[0071] Assuming the row labels correspond to user A, and the column labels correspond to user B, then the number 1 in the first column of the second row indicates that there is only one image out of the 20 that was labeled "sad" by user A and "excited" by user B. Similarly, the number 3 in the third row, third column indicates that there are three images that were labeled "happy" by both users. Furthermore, the sum of the numbers on the diagonal represents the total number of images for which the two users agree.

[0072] For a crosstab with r rows, let x ij is the element in row i and column j, x i+ is the sum of the elements in row i of the cross table, x +j is the sum of the elements in the jth column of the cross table, w ij is the consistency weight between the sentiment label in the i-th row and the sentiment label in the j-th column, n is the sum of all elements in the cross table, and the weighted Kappa correlation coefficient r is calculated based on the contingency table. k The formula is as follows:

[0073]

[0074] in,

[0075] Step S330: Obtain the mean of the N consistency scores corresponding to each group of first emotion annotation data to obtain the first annotation score corresponding to each group of first emotion annotation data.

[0076] In an embodiment of the present application, after obtaining N consistency scores corresponding to each group of first emotion annotation data, the average of the N consistency scores corresponding to each group of first emotion annotation data can be calculated to obtain the first annotation score corresponding to each group of first emotion annotation data.

[0077] For example, the calculation formula of the first annotation score is as follows:

[0078]

[0079] Among them, R s is the first annotation score, r k is the weighted Kappa correlation coefficient (i.e., consistency score) corresponding to the first sentiment annotation data, and N is the number of consistency scores, i.e., the number of groups of the second sentiment annotation data.

[0080] Step S340: Obtain the weighted consistency score between each group of first emotion annotation data and each other group of first emotion annotation data in the M groups of first emotion annotation data except each group of first emotion annotation data, and obtain M-1 consistency scores corresponding to each group of first emotion annotation data.

[0081] Similarly, for the M groups of first emotion annotation data of the same annotation sample set, the weighted consistency score between each group of first emotion annotation data and each group of data in the other M-1 groups of first emotion annotation data can be obtained to obtain M-1 consistency scores corresponding to each group of first emotion annotation data.

[0082] In some embodiments, the weighted consistency score can be a weighted Kappa correlation coefficient. The computer device can generate M-1 contingency tables corresponding to each group of first emotion annotation data based on obtaining each group of first emotion annotation data and each group of other first emotion annotation data; based on the M-1 contingency tables corresponding to each group of first emotion annotation data and the consistency weight, obtain the weighted Kappa correlation coefficient between each group of first emotion annotation data and each group of other first emotion annotation data, and obtain M-1 consistency scores corresponding to each group of first emotion annotation data, where the consistency weight is determined by the correlation between the labeled emotion tags. Among them, the method of obtaining the weighted Kappa correlation coefficient in step S340 can refer to the content of step S330 and will not be repeated here.

[0083] Step S350: Obtain the mean of the M-1 consistency scores corresponding to each group of first emotion annotation data to obtain the second annotation score corresponding to each group of first emotion annotation data.

[0084] In an embodiment of the present application, after obtaining M-1 consistency scores corresponding to each group of first emotion annotation data, the average of the M-1 consistency scores corresponding to each group of first emotion annotation data can be calculated to obtain the second annotation score corresponding to each group of first emotion annotation data.

[0085] Step S360: verifying each set of first emotion annotation data based on the first annotation score and the second annotation score corresponding to each set of first emotion annotation data.

[0086] In some embodiments, if the first annotation score corresponding to the target emotion annotation data is greater than the first threshold, and the second annotation score is greater than the second threshold, then the target emotion annotation data is determined to be qualified, and the target emotion annotation data is any group of emotion annotation data in the M groups of first emotion annotation data; if the first annotation score corresponding to the target emotion annotation data is less than or equal to the first threshold, and / or the second annotation score is less than or equal to the second threshold, then the target emotion annotation data is determined to be unqualified. In which, since the first annotation score and the second annotation score are the average of multiple weighted consistency scores, the first threshold and the second threshold can be determined based on the empirical value of the weighted consistency score. Optionally, when the weighted consistency score is a Kappa correlation coefficient, -1 indicates complete inconsistency, 0 indicates accidental consistency, 0.0-0.20 indicates extremely low consistency, 0.21-0.40 indicates general consistency, 0.41-0.60 indicates moderate consistency, 0.61-0.80 indicates high consistency, and 0.81-1 indicates almost complete consistency. The first threshold and the second threshold can be 0.41-0.60. For example, the first threshold and the second threshold are both 0.5.

[0087] The verification method for labeled data provided in an embodiment of the present application determines the uniformity of the labeling opinions between each group of first emotion labeling data and each group of second emotion labeling data (i.e., the first labeling score) according to the average of the weighted consistency scores between each group of first emotion labeling data and each group of second emotion labeling data in the standard labeling database when verifying M groups of first emotion labeling data of the same labeled sample set. It also determines the uniformity of the labeling opinions between each group of first emotion labeling data and other emotion labeling data for the same labeled sample set (i.e., the second labeling score) according to the average of the weighted consistency scores between each group of first emotion labeling data and other emotion labeling data for the same labeled sample set. Therefore, the first labeling score and the second labeling score are used simultaneously to effectively judge the labeling quality, which can improve the verification accuracy and obtain subjective labeling data that is objectively representative and universal. Moreover, since the first labeling score and the second labeling score are both obtained by taking the average value after determining the weighted consistency score based on the consistency weight, and the weighted consistency weight is determined by the correlation between the sentiment tags, the accuracy of the first labeling score and the second labeling score can be improved, so that when the first labeling score and the second labeling score are used to verify the sentiment labeling data, the verification accuracy can be further improved.

[0088] See also Figure 4 , Figure 4 A flow chart of a method for verifying the labeled data provided by another embodiment of the present application is shown. The method for verifying the labeled data is applied to the above-mentioned computer device. Figure 4 The process shown is described in detail. The verification method of the labeled data may specifically include the following steps:

[0089] Step S410: Acquire K groups of third emotion annotation data for the standard sample set, where K is a positive integer.

[0090] In an embodiment of the present application, the computer device can send the standard sample set in the standard sample library to the client corresponding to the annotation user who has the annotation qualification of the standard sample library. After the user annotation is completed, the client will feedback to the computer device, wherein the standard sample set can be sent to the clients corresponding to K annotation users who have the annotation qualification, thereby obtaining K groups of third emotion annotation data for the standard sample set. Among them, the annotation qualification of the standard sample library can be determined based on the number, number of times, accuracy, etc. of sample annotations previously made by the annotation user. Optionally, K can be greater than the above-mentioned N to ensure that the number of groups of second emotion annotation data in the subsequently obtained standard annotation database is greater than or equal to N.

[0091] Step S420: Obtain the consistency score between every two groups of third emotion annotation data in the K groups of third emotion annotation data.

[0092] In an embodiment of the present application, after the computer device obtains K groups of sample data for the standard sample set, it can obtain the consistency score between every two groups of third emotion annotation data for the K groups of third emotion annotation data. It should be noted that since the computer device does not obtain the standard annotation data and cannot accurately determine the consistency weight, the consistency score can be calculated when constructing the standard annotation database.

[0093] In some embodiments, a contingency table may be generated between each two sets of third sentiment annotation data, and a simple Kappa correlation coefficient may be calculated between each two sets of third sentiment annotation data. Specifically, the calculation formula of the Kappa correlation coefficient is as follows:

[0094]

[0095] For the r-row crosstab, let x ii is the diagonal element of the cross table, x i+ is the sum of the elements in row i of the cross table, x +i is the sum of the elements in the i-th column of the cross table, and n is the sum of all the elements in the cross table.

[0096] Step S430: Based on the consistency scores between each two groups of third emotion annotation data, the K groups of third emotion annotation data are screened, and the screened third emotion annotation data are stored as the second emotion annotation data in the standard annotation database.

[0097] In an embodiment of the present application, after obtaining the consistency score between each two groups of third emotion annotation data, the annotation data of the annotation users whose consistency scores between the annotation results and the annotation results of the vast majority of annotators are less than a preset threshold can be deleted according to the consistency score between each two groups of third emotion annotation data, and the remaining data can be stored in the standard annotation database. The annotation data in the standard annotation database is the gold standard data (i.e., the second emotion annotation data). Among them, according to the consistency score between each two groups of third emotion annotation data, all consistency scores corresponding to each group of third emotion annotation data can be determined, that is, K-1 consistency scores between each group of third emotion annotation data and each group of other third emotion annotation data, and then the number of consistency scores less than the preset threshold in the K-1 consistency scores is determined. If the number is greater than the preset number, it means that the annotation opinions of the users differ too much from those of other users, so they are deleted; if the number is not greater than the preset number, it means that the annotation opinions of the users are consistent with those of the majority of users, so they can be retained. In this way, the K groups of third emotion annotation data can be screened to obtain the emotion annotation data in the standard annotation database.

[0098] Step S440: Obtain M groups of first emotion annotation data for the same annotated sample set, wherein each group of first emotion annotation data is annotated by a different user, and M is a positive integer.

[0099] Step S450: Based on the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in N groups of second emotion annotation data, determine the first annotation score corresponding to each group of first emotion annotation data, the second emotion annotation data is the annotation data in the standard annotation database, and there are at least partially identical annotation samples between the second emotion annotation data and the first emotion annotation data, and N is a positive integer.

[0100] Step S460: Based on the weighted consistency score between each group of first emotion annotation data and each other group of first emotion annotation data in the M groups of first emotion annotation data, determine the second annotation score corresponding to each group of first emotion annotation data, and the consistency weight based on which the weighted consistency score is determined by the correlation between the annotated emotion tags.

[0101] Step S470: verifying each set of first emotion annotation data based on the first annotation score and the second annotation score corresponding to each set of first emotion annotation data.

[0102] In the embodiment of the present application, steps S440 to S470 can refer to the contents of other embodiments and will not be repeated here.

[0103] The verification method for labeled data provided in the embodiment of the present application provides a method for obtaining sentiment labeling data in a standard labeling database, thereby providing a standard for subsequent sentiment labeling data, thereby verifying the subsequent sentiment labeling data to calculate the first labeling score and improve the verification accuracy.

[0104] Next, pass Figure 5 The verification method of the labeled data involved in the above embodiment is introduced.

[0105] like Figure 5 As shown, the verification method of the annotation data involves a management client, a server, and an annotation client. The management client is the client corresponding to the engineer, the server is a computer device that executes the verification method of the annotation data in the aforementioned embodiment, and the annotation client is the client corresponding to the annotation user.

[0106] The management client can upload samples to be labeled (sample data to be labeled) and define emotion labels for emotion classification, as well as system parameters (such as the above-mentioned first threshold, second threshold, etc.).

[0107] After the system backend obtains the samples to be labeled, the defined emotional labels, and the system parameters, it can randomly divide the samples to be labeled, extract a certain proportion (1%-10%) of the sample data from all the samples to be labeled and put them into the standard sample library, and put the remaining sample data into the sample library to be labeled; then the system backend distributes the sample data in the standard sample library to labelers with standard labeling qualifications for labeling. Among them, the system backend determines whether the labeler has the labeling qualifications for the standard sample library (that is, the above-mentioned standard labeling qualifications) based on the labeler's past labeling performance. Only labelers whose labeling experience of the same type of data reaches a certain threshold and whose labeling pass rate is higher than a certain threshold have standard labeling qualifications. This threshold is generated by the system backend by default and can also be modified by the engineer corresponding to the management client.

[0108] When responding to the operations of the annotation user, the annotation client can determine whether the user has the standard annotation qualifications. If so, it can determine whether the user has a standard annotation task (i.e., an annotation task generated based on the sample data in the above-mentioned standard sample library); if a standard annotation task exists, the standard annotation task can be displayed, and based on the annotation operation of the annotation user, annotation data for the sample data in the standard sample library can be generated.

[0109] After the system backend receives the labeled data for the sample data in the labeled sample library, it stores the emotional tags selected by the labeling user into the database; after each sample data in the standard sample library is labeled T times (T>10, the T value can be modified by the management client), the labeling task of the standard sample library is completed. Here, T can be understood as the number of labeling users, that is, each sample data is labeled by T different labeling users. For the T groups of data obtained, a contingency table is generated between each two groups of data, and the Kappa correlation coefficient between the labeled data of each two labeling users is calculated; according to the calculation results, the labeled data of the labeling users whose correlation coefficients with the vast majority of labelers (i.e., the Kappa correlation coefficient) are less than the preset threshold are deleted, and the remaining data are stored in the standard labeling database. The labeled data in the standard labeling database are the gold standard data (i.e., the second emotional labeling data mentioned in the aforementioned embodiment).

[0110] In addition, the system backend counts the frequency of selection of each emotion tag corresponding to each annotation data in the standard annotation database based on the annotation data in the standard annotation database, and generates the consistency weights between the emotion tags based on the frequency to obtain the weight matrix.

[0111] After the system backend completes the acquisition of the labeled data in the standard labeling database and the determination of the consistency weight, it can divide the samples to be labeled into multiple data packets for the sample library to be labeled, and obtain multiple data packets to be labeled (i.e., labeled sample sets). Among them, the number of samples contained in each data packet should be set to ensure that the labeling user can complete the labeling at one time without getting tired, and a certain proportion (5%-10%) of the samples to be tested in each data packet comes from the standard sample library, and the rest comes from the sample library to be labeled, thereby ensuring that each data packet to be labeled is consistent with the standard labeling database. The system backend distributes labeling tasks to the labeling client with the data packet to be labeled as the smallest unit. The labeling task is carried out with the data packet to be labeled as the smallest unit, and the order of sample presentation in the data packet is random each time labeling is performed. Among them, the same data packet to be labeled can be labeled at most once by the same labeling client.

[0112] After assigning the labeling task for sample data in the sample library, the system backend receives sentiment annotation data (i.e., the aforementioned first sentiment annotation data) uploaded by the annotation client for the data packets to be labeled in the labeling task. Using the aforementioned weight matrix and the sentiment annotation data in the standard annotation database, the system backend calculates the first and second annotation scores corresponding to the sentiment annotation data for each data packet to be labeled, and performs consistency verification on the sentiment annotation data for each data packet to be labeled. After verification, the system backend marks the verification result (pass or fail) for each data packet to be labeled, and stores the sentiment annotation data with a passing result in the database. Furthermore, the system backend can provide feedback to the annotation client based on the annotation results, thereby providing feedback to the annotation user. For example, after verifying the annotation data for each data packet to be labeled, the system backend provides textual feedback to the annotation client via a web interface: "Your annotation quality for this data packet is pass / fail." This feedback allows the annotation user to promptly understand their own annotation status, reflect on, and correct incorrect annotation behavior. The system backend can also provide feedback to the management client to assist engineers in data cleaning, data weight setting, and annotation result acceptance.

[0113] In the above process, for all samples to be labeled, the labeled data of the standard sample library and the labeled data of the data packets that are subsequently verified to be qualified can constitute the labeled data of all samples to be labeled, thereby realizing the emotional labeling of all samples to be labeled.

[0114] See also Figure 6, which shows a structural block diagram of a data verification device 400 provided in an embodiment of the present application. The data verification device 400 uses the above-mentioned computer device and includes: a data acquisition module 410, a first scoring module 420, a second scoring module 430, and a data verification module 440. Among them, the data acquisition module 410 is used to obtain M groups of first emotion annotation data for the same annotation sample set, wherein the annotation users of each group of first emotion annotation data are different, and M is a positive integer; the first scoring module 420 is used to determine the first annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in N groups of second emotion annotation data, the second emotion annotation data is the annotation data in the standard annotation database, and there are at least some identical annotation samples between the second emotion annotation data and the first emotion annotation data, and N is a positive integer; the second scoring module 430 is used to determine the second annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each other group of first emotion annotation data in the M groups of first emotion annotation data, the consistency weight of the weighted consistency score being determined by the correlation between the annotated emotion tags; the second scoring module 440 is used to verify each group of first emotion annotation data based on the first annotation score and the second annotation score corresponding to each group of first emotion annotation data.

[0115] In some embodiments, the verification device 400 for the labeled data may further include: a frequency acquisition module and a weight determination module. The frequency acquisition module is configured to determine the number of times each sample is labeled with each emotion tag based on the second emotion tag data in the standard labeling database before determining the first labeling score corresponding to each group of first emotion tag data based on the weighted consistency score between each group of first emotion tag data and each group of second emotion tag data in the N groups of second emotion tag data; and the weight determination module is configured to determine the consistency weight between each two emotion tags based on the number of times each sample is labeled with each emotion tag.

[0116] As a possible implementation method, the weight determination module can be specifically used to: determine the correlation coefficient between each two emotion tags based on the number of times each emotion tag corresponding to each sample is labeled; based on the correlation coefficient between each two emotion tags, determine the consistency weight between each two emotion tags, and the consistency weight is negatively correlated with the correlation coefficient.

[0117] In some embodiments, the first scoring module 420 can be used to: obtain the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in the N groups of second emotion annotation data, and obtain N consistency scores corresponding to each group of first emotion annotation data; obtain the average of the N consistency scores corresponding to each group of first emotion annotation data, and obtain the first annotation score corresponding to each group of first emotion annotation data.

[0118] As a possible implementation, the first scoring module 420 obtains the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in the N groups of second emotion annotation data, and obtains N consistency scores corresponding to each group of first emotion annotation data, which may include: generating N contingency tables corresponding to each group of first emotion annotation data based on each group of first emotion annotation data and each group of second emotion annotation data in the N groups of second emotion annotation data; obtaining the weighted Kappa correlation coefficient between each group of first emotion annotation data and each group of second emotion annotation data based on the N contingency tables corresponding to each group of first emotion annotation data, and the consistency weight, to obtain N consistency scores corresponding to each group of first emotion annotation data, wherein the consistency weight is determined by the correlation between the labeled emotion tags.

[0119] In some embodiments, the second scoring module 420 can be used to: obtain the weighted consistency score between each group of first emotion annotation data and each other group of first emotion annotation data in the M groups of first emotion annotation data except each group of first emotion annotation data, and obtain M-1 consistency scores corresponding to each group of first emotion annotation data; obtain the average of the M-1 consistency scores corresponding to each group of first emotion annotation data, and obtain the second annotation score corresponding to each group of first emotion annotation data.

[0120] As a possible implementation, the second scoring module 420 obtains the weighted consistency score between each group of first emotion annotation data and each group of other first emotion annotation data in the M groups of first emotion annotation data except the first emotion annotation data, and obtains M-1 consistency scores corresponding to each group of first emotion annotation data, which may include: based on obtaining each group of first emotion annotation data and each group of other first emotion annotation data, generating M-1 contingency tables corresponding to each group of first emotion annotation data; based on the M-1 contingency tables corresponding to each group of first emotion annotation data and the consistency weight, obtaining the weighted Kappa correlation coefficient between each group of first emotion annotation data and each group of other first emotion annotation data, and obtaining M-1 consistency scores corresponding to each group of first emotion annotation data, wherein the consistency weight is determined by the correlation between the labeled emotion tags.

[0121] In some embodiments, the annotated data verification device 400 may further include: a third scoring module and a data screening module. The data acquisition module 410 may also be used to obtain K groups of third emotion annotation data for the standard sample set, where K is a positive integer; the third scoring module may be used to obtain a consistency score between each two groups of third emotion annotation data in the K groups of third emotion annotation data; the data screening module may be used to screen the K groups of third emotion annotation data based on the consistency score between each two groups of third emotion annotation data, and store the screened third emotion annotation data as the second emotion annotation data in the standard annotation database.

[0122] In some embodiments, the data verification module 440 can be used to: if the first annotation score corresponding to the target emotion annotation data is greater than the first threshold, and the second annotation score is greater than the second threshold, then the target emotion annotation data is determined to be qualified, and the target emotion annotation data is any group of emotion annotation data in the M groups of first emotion annotation data; if the first annotation score corresponding to the target emotion annotation data is less than or equal to the first threshold, and / or the second annotation score is less than or equal to the second threshold, then the target emotion annotation data is determined to be unqualified.

[0123] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0124] In several embodiments provided in this application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0125] In addition, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or software functional modules.

[0126] In summary, the solution provided by the present application is to obtain M groups of first emotion annotation data for the same annotation sample set, and then determine the first annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in the N groups of second emotion annotation data in the standard annotation database, and there are at least partially identical annotation samples between the second emotion annotation data and the first emotion annotation data, and determine the second annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each other group of first emotion annotation data in the M groups of first emotion annotation data, and the consistency weight based on which the weighted consistency score is determined is determined by the correlation between the labeled emotion tags, and then verify each group of first emotion annotation data based on the first annotation score and the second annotation score corresponding to each group of first emotion annotation data. Since the first annotation score and the second annotation score are both obtained by determining the weighted consistency score based on the consistency weight, and the weighted consistency weight is determined by the correlation between the emotion tags, the accuracy of the first annotation score and the second annotation score can be improved, so that when the emotion annotation data is verified using the first annotation score and the second annotation score, the verification accuracy can be improved.

[0127] Please refer to Figure 7 , which shows a block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 100 can be a server, a PC, a laptop, or other device capable of running applications. The computer device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more applications are configured to execute the method described in the aforementioned method embodiment.

[0128] The processor 110 may include one or more processing cores. The processor 110 utilizes various interfaces and circuits to connect various components within the computer device 100. It executes instructions, programs, code sets, or instruction sets stored in the memory 120, and accesses data stored in the memory 120 to perform various functions and process data within the computer device 100. Optionally, the processor 110 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 110 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 110 and may be implemented separately via a communications chip.

[0129] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the computer device 100 during use (such as a phone book, audio and video data, chat history data), etc.

[0130] Please refer to Figure 8 , which shows a block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 800 stores program code, which can be called by a processor to execute the method described in the above method embodiment.

[0131] The computer-readable storage medium 800 can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 800 has storage space for program code 810 for executing any of the method steps described above. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in a suitable form.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for verifying labeled data, characterized in that: The method comprises: Obtaining M groups of first emotion annotation data for the same annotated sample set, wherein each group of first emotion annotation data is annotated by a different user, and M is a positive integer; Determine a first annotation score corresponding to each group of first emotion annotation data based on a weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in N groups of second emotion annotation data, wherein the second emotion annotation data is annotation data in a standard annotation database, and there are at least partially identical annotation samples between the second emotion annotation data and the first emotion annotation data, and N is a positive integer; Determine a second annotation score corresponding to each group of first emotion annotation data based on a weighted consistency score between each group of first emotion annotation data and each group of other first emotion annotation data in the M groups of first emotion annotation data, wherein the consistency weight according to which the weighted consistency score is based is determined by the correlation between the annotated emotion tags; Based on the first annotation score and the second annotation score corresponding to each group of first emotion annotation data, each group of first emotion annotation data is verified.

2. The method according to claim 1, characterized in that Before determining the first annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in the N groups of second emotion annotation data, the method further includes: Based on the second emotion annotation data in the standard annotation database, determining the number of times each sample is annotated with each emotion tag; Based on the number of times each sample corresponding to each emotion tag is labeled, a consistency weight between every two emotion tags is determined.

3. The method according to claim 2, characterized in that The step of determining the consistency weight between every two emotion tags based on the number of times each emotion tag is marked includes: Determine the correlation coefficient between every two emotion tags based on the number of times each emotion tag corresponding to each sample is marked; Based on the correlation coefficient between each two emotion tags, a consistency weight between each two emotion tags is determined, and the consistency weight is negatively correlated with the correlation coefficient.

4. The method according to claim 1, characterized in that: The determining, based on the weighted consistency score between each group of the first emotion annotation data and each group of the second emotion annotation data in the N groups of the second emotion annotation data, a first annotation score corresponding to each group of the first emotion annotation data comprises: Respectively obtain a weighted consistency score between each group of the first emotion annotation data and each group of the second emotion annotation data in the N groups of the second emotion annotation data, and obtain N consistency scores corresponding to each group of the first emotion annotation data; The average of the N consistency scores corresponding to each group of first emotion annotation data is obtained to obtain the first annotation score corresponding to each group of first emotion annotation data.

5. The method according to claim 4, characterized in that The obtaining of the weighted consistency score between each group of the first emotion annotation data and each group of the second emotion annotation data in the N groups of the second emotion annotation data to obtain N consistency scores corresponding to each group of the first emotion annotation data includes: Based on each group of the first emotion annotation data and each group of the second emotion annotation data in the N groups of the second emotion annotation data, generating N contingency tables corresponding to each group of the first emotion annotation data; Based on the N contingency tables corresponding to each group of first emotion annotation data and the consistency weight, the weighted Kappa correlation coefficient between each group of first emotion annotation data and each group of second emotion annotation data is obtained to obtain N consistency scores corresponding to each group of first emotion annotation data, and the consistency weight is determined by the correlation between the annotated emotion tags.

6. The method according to claim 1, characterized in that The determining, based on the weighted consistency score between each group of first emotion annotation data and other first emotion annotation data in the M groups of first emotion annotation data, a second annotation score corresponding to each group of first emotion annotation data comprises: Obtain a weighted consistency score between each group of first emotion annotation data and each other group of first emotion annotation data in the M groups of first emotion annotation data except each group of first emotion annotation data, and obtain M-1 consistency scores corresponding to each group of first emotion annotation data; The average of the M-1 consistency scores corresponding to each group of the first emotion annotation data is obtained to obtain the second annotation score corresponding to each group of the first emotion annotation data.

7. The method according to claim 6, characterized in that The step of obtaining a weighted consistency score between each group of the first emotion annotation data and each other group of the first emotion annotation data in the M groups of the first emotion annotation data except the each group of the first emotion annotation data, to obtain M-1 consistency scores corresponding to each group of the first emotion annotation data, includes: Based on obtaining each group of the first emotion annotation data and each group of other first emotion annotation data, generating M-1 contingency tables corresponding to each group of the first emotion annotation data; Based on the M-1 contingency tables corresponding to each group of first emotion annotation data and the consistency weight, the weighted Kappa correlation coefficient between each group of first emotion annotation data and each other group of first emotion annotation data is obtained to obtain M-1 consistency scores corresponding to each group of first emotion annotation data, and the consistency weight is determined by the correlation between the annotated emotion tags.

8. The method according to any one of claims 1 to 7, characterized in that: Before obtaining M groups of first emotion annotation data for the same annotated sample set, the method further includes: Obtain K groups of third emotion annotation data for the standard sample set, where K is a positive integer; Obtaining a consistency score between every two groups of third emotion annotation data in the K groups of third emotion annotation data; Based on the consistency scores between each two groups of the third emotion annotation data, the K groups of the third emotion annotation data are screened, and the screened third emotion annotation data are stored in the standard annotation database as the second emotion annotation data.

9. The method according to any one of claims 1 to 7, characterized in that: The verifying each group of first emotion annotation data based on the first annotation score and the second annotation score corresponding to each group of first emotion annotation data includes: If the first annotation score corresponding to the target emotion annotation data is greater than the first threshold, and the second annotation score is greater than the second threshold, the target emotion annotation data is determined to be qualified, and the target emotion annotation data is any group of emotion annotation data in the M groups of first emotion annotation data; If the first annotation score corresponding to the target sentiment annotation data is less than or equal to the first threshold, and / or the second annotation score is less than or equal to the second threshold, the target sentiment annotation data is determined to be unqualified.

10. A verification device for labeled data, characterized in that: The device comprises: a data acquisition module, a first scoring module, a second scoring module and a data verification module, wherein: The data acquisition module is used to acquire M groups of first emotion annotation data for the same annotated sample set, wherein each group of first emotion annotation data is annotated by a different user, and M is a positive integer; The first scoring module is used to determine the first annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each group of second emotion annotation data in N groups of second emotion annotation data, wherein the second emotion annotation data is annotation data in a standard annotation database, and there are at least partially identical annotation samples between the second emotion annotation data and the first emotion annotation data, and N is a positive integer; The second scoring module is used to determine the second annotation score corresponding to each group of first emotion annotation data based on the weighted consistency score between each group of first emotion annotation data and each group of other first emotion annotation data in the M groups of first emotion annotation data, wherein the consistency weight according to which the weighted consistency score is based is determined by the correlation between the annotated emotion tags; The second scoring module is used to verify each group of first emotion annotation data based on the first annotation score and the second annotation score corresponding to each group of first emotion annotation data.

11. A computer device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 9.

Images