A cross-validation method, device and storage medium for emotional data labeling

Abstract

The application provides a cross-validation method and device for emotional data labeling and a storage medium. The method comprises the following steps: obtaining initial data, copying the same initial data to generate multiple same data, and splitting the data; distributing the split data to labelers for labeling; collecting the labeled data and re-integrating to obtain labeling results of the multiple data; cross-validating the labeling results, putting single labeling data consistent in the labeling results into a file, and putting inconsistent labeling data into another file; re-splitting the inconsistent labeling data and distributing the same to the labelers for re-labeling; and cross-validating the re-labeled results again. The application repeatedly cross-validates the labeling results of the labelers to improve the accuracy and availability of data labeling. The data labeling and verification can be performed internally, improving the data processing efficiency and reducing the cost of data labeling and verification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of emotional data labeling, and more particularly, to a cross-validation method for emotional data labeling, a device and a storage medium. BACKGROUND

[0002] In recent years, computer technology based on deep learning is becoming more and more mature. In the training process of data mining, artificial intelligence and machine learning, the required training set data is also increasing. However, the obtained data is too extensive, and there are many useless data, that is, the so-called noise. These noises will affect the training effect of the model. A good model needs a large amount of high-quality data to support, and these good data come from data labeling. At present, the data labeling basically adopts the manual labeling method. The labeling result of the data depends on the emotional tendency and labeling level of the labeling personnel to a certain extent, and this situation is particularly evident for emotional data labeling in natural language processing. Therefore, in order to improve the quality of emotional data labeling and reduce the cost of obtaining accurate data, the present application adopts a data labeling method in the form of crowdsourcing and a cross-validation method to obtain high-quality emotional data. It is not necessary for multiple people to vote and elect the best label for the data. It is only necessary to cross-evaluate internally, re-label the controversial data, and cross-validate multiple times to obtain the best result. SUMMARY

[0003] The present application is based on some technical defects that already exist. The purpose is to design a cross-validation method for emotional data labeling. This method can improve the labeling efficiency and reduce the error of labeling accuracy.

[0004] To achieve the above purpose, the present application provides a cross-validation method for emotional data labeling, which includes the following steps,

[0005] Step 1. Obtain initial data, and distribute the same initial data to multiple labeling personnel after cutting for labeling. The initial data includes pictures and text descriptions.

[0006] Step 2. Collect and integrate the labeling results to obtain multiple different labeling results of a piece of data, and cross-validate the multiple different labeling results.

[0007] Step 3. Put the single labeling data consistent in the labeling result that passes the verification into a file, and name it as a passed file. Put the inconsistent data back into another file, and name it as a returned file.

[0008] Step 4. The re-cutting of the returned file is performed again, and the re-labelling and judging of the labellers are performed again until a plurality of passed files are obtained, all the passed files are integrated to obtain a final data file, and the text emotional clues of the final file are judged and selected;

[0009] Step 5. The text emotional clues of each data in the selected file are calculated;

[0010] Step 6. According to the calculated scores, the text emotional clues with the highest scores are selected, and the obtained text and picture data are spliced as the final complete labelling data result.

[0011] On the basis of the above technical solutions, the application can also be improved as follows.

[0012] Optionally, the cutting and distribution of the same initial data to multiple labellers for labelling comprises obtaining initial data, and copying the data into multiple same data;

[0013] Each data is cut, and the cutting size and proportion of each data need to be the same;

[0014] The cut multiple data are distributed to multiple labellers for labelling.

[0015] Optionally, the integration of the labelling results to obtain multiple different labelling results of one data comprises,

[0016] The small labelling data results labelled by the labellers are collected and spliced to obtain multiple same large data again, but the labelling results of the multiple data are different, and the multiple different labelling results are cross-validated.

[0017] Optionally, the cross-validation of the multiple different labelling results comprises judging the validity of the data, and after the data is valid, the data is sequentially subjected to the judgment of the text and picture emotion, the picture and the picture auxiliary emotion.

[0018] Optionally, the judgment of the validity of the data comprises judging whether the text and picture are valid, if not, the data is directly put into an invalid file, and the next data is judged, if the text and picture are valid, the subsequent steps are continued.

[0019] Optionally, the calculation of the text emotional clues of each data in the final data file comprises calculating the text emotional clues of each data in the selected file by using the ROUGE-L algorithm and the BLEU algorithm; wherein the calculation formula of the ROUGE-L algorithm is:

[0020]

[0021]

[0022]

[0023] where LCS(X,Y) is the length of the longest common subsequence of X and Y, m and n represent the length of the artificial standard summary and the machine automatic summary respectively, R lcs , P lcs represent the recall rate and the accuracy rate respectively, β represents a hyper parameter, and the final F lcs is ROUGE-L.

[0024] Optionally, the graphic-text sentiment judgment of the data comprises comparing the graphic-text sentiments of multiple data, judging whether the data pass according to the proportion of the similar comparison of the data, if the judgment of the majority of the data is consistent, it is indicated that the marking passes, otherwise, it is indicated that the verification fails, and the data is put into the inconsistent file.

[0025] Optionally, the judgment of the picture in the data comprises judging the text extraction of the picture, if the majority of the data indicates that the text extraction needs to be performed, and the minority of the data indicates that the text extraction does not need to be performed, it is indicated that the data passes, otherwise, it is indicated that the verification fails, and the data needs to be put back to be re-labeled.

[0026] According to a second aspect of the present application, an electronic device is provided, comprising a memory and a processor, the processor being configured to implement the steps of the cross-validation method for sentiment data labeling when executing a computer program stored in the memory.

[0027] According to a third aspect of the present application, a computer readable storage medium is provided, having a computer management program stored thereon, the computer program being configured to implement the steps of the cross-validation method for sentiment data labeling when executed by a processor.

[0028] Technical effects and advantages of the present application:

[0029] The ROUGE-L algorithm and the BLEU algorithm in the automatic abstract evaluation method are added in the present application, the text sentiment clues in the sentiment data are calculated by using the ROUGE-L algorithm and the BLEU algorithm, and finally the sentiment data labeling result is obtained.

[0030] The present application can significantly speed up the process of data labeling under the existing conditions, improve the efficiency of cross-validation of sentiment data labeling, improve the accuracy of data, and also can reduce the cost of cross-validation of data labeling.

[0031] Other features and advantages of the present application will be set forth in the following description, and in part will be apparent from the description, or can be learned by practice of the application. The objects and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a step flow chart of a cross-validation method for emotional data labeling provided by an embodiment of the present application;

[0033] Figure 2 is a cross-validation flow chart for multiple different labeling results in an embodiment of the present application;

[0034] Figure 3 is a flow chart of final results obtained by the Chinese text emotional clues according to the ROUGE-L algorithm and the BLEU algorithm. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0036] To solve the problems in the prior art, the present application discloses a cross-validation method for emotional data labeling, specifically as shown in Figure 1 , comprising the following steps,

[0037] Step 1. Obtain initial data, and distribute the same initial data after being cut into multiple parts to multiple labelers for labeling. In the present embodiment, the data source is mainly obtained by crawling microblog comments, and the content mainly includes pictures and text descriptions.

[0038] Specifically, the step of distributing the same initial data after being cut into multiple parts to multiple labelers for labeling comprises the following steps:

[0039] ①Obtain initial data, and copy the data into three identical data;

[0040] ②Cut each data, and the cut size and proportion of each data need to be the same;

[0041] ③Distribute the multiple small data obtained by cutting each data to multiple labelers for labeling.

[0042] Step 2. Collect and integrate the labeling results to obtain multiple different labeling results of one data, and perform cross-validation on the multiple different labeling results.

[0043] Specifically, after the small portion of the annotation data results annotated by the annotators are returned and spliced and integrated, three same large portions of data are obtained again, but the annotation results of the three portions of data are different, and the annotation results of the three portions of data are cross-verified;

[0044] The cross-verification includes: first, judging the validity of the data, and then judging the graphic-text sentiment, the text sentiment, the picture and the picture auxiliary sentiment of the data in sequence.

[0045] Specifically, as shown in Figure 2 The emotion data validity judgment includes: first, judging whether the graphic-text is valid, if not, the data is directly put into the invalid file, and the next data is judged, if the graphic-text is valid, the following steps are continued;

[0046] The graphic-text sentiment judgment of the data specifically includes: comparing the graphic-text sentiments of the three portions of data, if the comparison of the three portions of data is 3:0, that is, the selection of the graphic-text sentiments of the three portions of data is all positive or all neutral or all negative, or 2:1, that is, 2 selections of positive and 1 selection of neutral, or 2 selections of negative and 1 selection of neutral, it indicates that the annotation passes, otherwise it indicates that the verification fails, and the data is put into the inconsistent file;

[0047] The text sentiment judgment of the data is the same as the graphic-text sentiment judgment of the data; specifically including: comparing the text sentiments of the three portions of data, if the comparison of the three portions of data is 3:0, that is, the selection of the text sentiments of the three portions of data is all positive or all neutral or all negative, or 2:1, that is, 2 selections of positive and 1 selection of neutral, or 2 selections of negative and 1 selection of neutral, it indicates that the annotation passes, otherwise it indicates that the verification fails, and the data is put into the inconsistent file;

[0048] The judgment of the picture in the data specifically includes: first, judging whether the picture is OCR: for the judgment of whether the picture is OCR, 3:0 represents that all or none of the pictures are subjected to OCR, 2:1 represents that 2 pictures are subjected to OCR and 1 picture is not subjected to OCR, and the two cases represent that the verification passes; for the case that 2 pictures are not subjected to OCR and 1 picture is subjected to OCR, it indicates that the verification fails and needs to be returned for re-annotation. It should be noted that OCR means extracting the text in the picture.

[0049] In the embodiment of the present application, the picture auxiliary emotion judgment is the same as the data graphic text emotion judgment step; specifically comprising: comparing the graphic text auxiliary emotions of the three data, if the comparison of the three data is 3:0, that is, the selection of the graphic text auxiliary emotions of the three data is all positive or all neutral or all negative, or 2:1, that is, 2 are positive and 1 is neutral, or 2 are negative and 1 is neutral, it indicates that the labeling passes, otherwise it indicates that the verification fails, and the data is put into the inconsistent file.

[0050] Step 3. The single labeled data consistent with the verification passed in the labeling result is put into a file, named as passed file, and the inconsistent one is put back to another file, named as put back file;

[0051] Step 4. The put back file is re-divided and re-distributed to the labelers for labeling and judgment again, steps 3 and 4 are repeated multiple times, multiple passed files are obtained, all the passed files are integrated to obtain a final data file, and then the text emotion clues of the final file are judged and selected.

[0052] Step 5. The text emotion clues of each data in the selected file are calculated;

[0053] Specifically, the text emotion clues of each data in the final data file are calculated, specifically comprising: the text emotion clues of each data in the selected file are calculated by using ROUGE-L algorithm and BLEU algorithm;

[0054] In the embodiment of the present application, ROUGE algorithm (Recall-Oriented Understudy for Gisting Evaluation) is an index for evaluating Chinese text summary, which is a set of indexes for evaluating automatic summarization and machine translation. It compares the automatically generated summary or translation with a set of reference summaries, usually manually generated, to calculate the corresponding score to measure the "similarity" between the automatically generated summary or translation and the reference summary. L in ROUGE-L is the first letter of LCS, where LCS represents the longest common subsequence.

[0055] The calculation formula of the ROUGE-L algorithm is:

[0056]

[0057]

[0058]

[0059] Where LCS(X,Y) is the length of the longest common subsequence of X and Y, and m and n represent the lengths of the human-standard summary and the machine-automated summary, respectively, which are generally the number of words contained; R lcs P lcs These represent recall and precision, respectively. β is a hyperparameter that needs to be set manually. Since hyperparameter β is usually set very large, generally only R is considered. lcs The last F lcs This is ROUGE-L, which represents the Chinese text summarization index.

[0060] In this embodiment of the invention, the full name of the BLEU algorithm is Bilingual Evaluation Understudy. Examples and formula explanations are provided. BLEU scores range from 0 to 1; the closer the score is to 1, the higher the translation quality. BLEU is primarily based on precision. The overall formula for BLEU is shown below.

[0061] The formula for calculating BLEU is as follows:

[0062]

[0063] In the formula, P refers to the precision of the gram; gram represents a feature vector dimension; Wn refers to the weight of the gram, which is generally set to uniform weight, that is, for any n, Wn = 1 / N, with an upper limit of 4 for N; BP is a penalty factor, which is less than 1 if the length of the translation is less than the shortest reference translation. BLEU's 1-gram precision indicates the degree to which the translation is faithful to the original text, while the other n-grams indicate the fluency of the translation; exp represents a function, N is the total number of N functions, and ∑ represents the summation from n = 1 to n = N.

[0064] Step 6. Based on the calculated scores, select the text sentiment cue with the highest score, and combine the obtained text and image data to obtain the final complete labeled data result.

[0065] Specifically, based on the scores calculated by the ROUGE-L algorithm and the BLEU algorithm, the text sentiment cues with the highest scores are selected, and the resulting text and image data are concatenated to obtain the final complete labeled data result.

[0066] In the embodiment of the present application, for a piece of text, there are three different annotation results, namely three different text sentiment clues, one piece of data indicating invalid, which will be excluded from the total data; one piece of data indicating verification pass; one piece of data indicating that it needs to be re-annotated. The re-annotated data needs new annotators to re-annotate the data, and then cross-validation is performed again. After multiple verifications, the final file data is calculated by ROUGE-L algorithm and BLEU algorithm to obtain the final text sentiment clue, and the final annotation data result is obtained.

[0067] As shown in the specific embodiment Figure 3 After the text sentiment clues of the three data judged in step 2 are calculated by the ROUGE-L algorithm and the BLEU algorithm, they are integrated into a file data together with other judged sentiment labels to form a new piece of data and put into a pass file, so as to obtain the final complete annotation data result.

[0068] According to the second aspect of the present application, an electronic device is provided, which comprises a memory and a processor, wherein the processor is configured to implement the steps of a cross-validation method for sentiment data annotation when executing a computer program stored in the memory. Specifically, the steps include,

[0069] Step 1. Obtain initial data, and distribute the segmented initial data to multiple annotators for annotation; wherein the initial data includes pictures and text descriptions;

[0070] Step 2. Integrate the annotation results to obtain multiple different annotation results of a piece of data, and cross-validate the multiple different annotation results;

[0071] Step 3. Put the single annotation data consistent in the annotation results that pass the verification into a file, and name it as a pass file, and put the inconsistent data into another file, and name it as a return file;

[0072] Step 4. Re-segment the return file and distribute it to the annotators for re-annotation and judgment until multiple pass files are obtained, integrate all the pass files to obtain a final data file, and then select the text sentiment clue of the final file;

[0073] Step 5. Calculate the text sentiment clue of each piece of data in the final data file;

[0074] Step 6. According to the calculated score, select the text sentiment clue with the highest score, and splice the obtained text and picture data as the final complete annotation data result.

[0075] According to a third aspect of the present application, a computer readable storage medium is provided, having stored thereon a computer management program, which, when executed by a processor, implements the steps of a cross-validation method for sentiment data labeling.

[0076] Step 1. Obtain initial data, and distribute the same initial data to multiple annotators after being cut for labeling; wherein the initial data includes pictures and text descriptions;

[0077] Step 2. Collect and integrate the labeling results to obtain multiple different labeling results of a piece of data, and perform cross-validation on the multiple different labeling results;

[0078] Step 3. Put the single labeling data consistent in the labeling results that pass the verification into a file, named as a passed file, and put the inconsistent ones back into another file, named as a returned file;

[0079] Step 4. Re-cut the returned file and distribute it to the annotators for re-labeling and judgment until multiple passed files are obtained, integrate all the passed files to obtain a final data file, and then perform text sentiment clue judgment and selection on the final file;

[0080] Step 5. Calculate the text sentiment clue of each data in the final data file;

[0081] Step 6. According to the calculated score, select the text sentiment clue with the highest score, and splice the obtained text and picture data as the final complete labeling data result.

[0082] In summary, the present application adds the automatic abstract evaluation methods ROUGE-L algorithm and BLEU algorithm, uses the ROUGE-L algorithm and BLEU algorithm to calculate the text sentiment clue in the sentiment data, and obtains the final sentiment data labeling result. In this way, the precision of the text sentiment clue can be greatly improved, and the accuracy of the sentiment data can be improved. At the same time, the process of data labeling can be obviously accelerated, the efficiency of cross-validation of sentiment data labeling can be improved, and the cost of cross-validation of data labeling can be reduced.

[0083] Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the preferred embodiments and all the changes and modifications falling within the scope of the present application.

[0084] Finally, it should be noted that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art will appreciate that the technical solutions described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalent features, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A cross-validation method for affective data annotation, characterized in that, The method comprises the following steps: Step 1. Obtain initial data, and distribute the same initial data to multiple annotators after being cut for labeling; wherein the initial data comprises pictures and text descriptions; Step 2. Collect and integrate the labeling results to obtain multiple different labeling results of a piece of data, and cross-verify the multiple different labeling results; comprising: after collecting and integrating the small pieces of labeled data, the same multiple pieces of data with different labeling results are obtained again, and the multiple pieces of data with different labeling results are cross-verified; first, the validity of the data is judged, comprising: determine whether the image and text are valid, if not, the data is directly put into the invalid file, and the next data is judged; if the image and text are valid, the data is sequentially judged for image and text sentiment, picture and picture auxiliary sentiment; wherein, the judgment of the image and text sentiment of the data comprises: comparing the image and text sentiment of multiple pieces of data, and judging whether the data passes according to the proportion of similar comparison of the data; if the judgment of most data is consistent, it means that the labeling passes, otherwise, it means that the verification fails, and the data is put into the inconsistent file; the judgment of the picture in the data comprises: judging whether the picture needs text extraction, if most data indicate that text extraction is needed, and a small amount of data indicate that text extraction is not needed, the data passes, otherwise, the verification fails, and the data needs to be put back for re-labeling; Step 3. Put the single labeling data consistent in the labeling results that pass the verification into a file, named as passed file, and put the inconsistent data into another file, named as put back file; Step 4. Re-cut the put back file and distribute it to the annotators for re-labeling and judgment until all passed files are obtained, integrate all passed files to obtain the final data file, and then judge and select the text sentiment clues in the final data file; Step 5. Calculate the text sentiment clues of each data in the final data file, comprising: calculating the text sentiment clues of each data in the selected file by using ROUGE-L algorithm and BLEU algorithm; wherein the calculation formula of the ROUGE-L algorithm is: In the formula, LCS(X, Y) is the length of the longest common subsequence of X and Y, m and n represent the lengths of the artificial standard abstract and the machine automatic abstract respectively, , Recall and precision represent the recall rate and the accuracy rate respectively, and β represents a hyperparameter, That is, ROUGE-L represents the Chinese text abstract index. Step 6. According to the calculated score, select the text sentiment clue with the highest score, and splice the obtained text and picture data as the final complete labeling data result.

2. The cross-validation method for emotion-oriented data labeling according to claim 1, wherein, The distribution of the same initial data to multiple annotators after being cut for labeling comprises the following steps: obtain initial data, and copy the data into multiple pieces of the same data; cut each piece of data, and the cutting size and proportion of each piece of data need to be the same; distribute the cut multiple pieces of data to multiple annotators for labeling.

3. An electronic device, comprising: A memory, a processor, the processor is used to execute the computer program stored in the memory to realize the steps of the cross-verification method for emotion data labeling according to any one of claims 1 to 2.

4. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is executed by the processor to realize the steps of the cross-verification method for emotion data labeling according to any one of claims 1 to 2.

Images