Text data annotation consistency detection method and device, equipment and storage medium

Abstract

The invention provides a text data annotation consistency detection method and device, equipment and a storage medium, and the method comprises the steps: carrying out the feature extraction of annotated text data, enabling the text data to be converted into a feature sequence composed of keyword slot position tags, carrying out the classification of the text data based on the feature sequence, and obtaining the annotation consistency of the text data. Therefore, the accuracy of text classification can be remarkably improved, texts with the same sentence pattern are classified into one class, and the consistency condition of corpus annotation can be reflected more visually, so that effective assistance is provided for annotation consistency check, and higher-quality training corpora are provided for subsequent natural language processing tasks.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a text data annotation consistency detection method, device, equipment and storage medium. Background Art

[0002] For natural language processing technology, during the annotation consistency detection process of the training corpus, when data of the same form type (in the context of natural language processing, mainly refers to sentence type) are labeled with the same type label in the corpus, it is generally believed that the annotation consistency is high; on the contrary, if the labels are varied, the annotation consistency is low and there are problems with the data annotation quality.

[0003] At present, the consistency of annotation is mainly maintained through manual inspection by quality inspectors among data annotators and continuous rework and correction by annotators. Related auxiliary methods, such as using text clustering algorithms to cluster similar texts into several categories, and then manually checking each category to find the corpus with annotation problems. Or the simplest method is to use Excel software to sort the data in alphabetical order, and then manually check the quality.

[0004] The method that relies solely on manual work has obvious disadvantages, that is, it completely relies on manual work, the work efficiency is low, and the probability of error is still high. The text clustering-assisted method has a relatively coarse clustering scale. When the classification is too coarse, the assistance to humans is unsatisfactory. The method assisted by phonetic sorting is similar to the situation assisted by text clustering, or even worse, and has very limited help in improving the efficiency of consistency checking. Summary of the invention

[0005] The object of the present invention is to provide a method, device, equipment and storage medium for detecting consistency of text data annotation.

[0006] The present invention provides a text data annotation consistency detection method, which comprises the steps of:

[0007] Get the annotated text data;

[0008] Performing feature extraction on the text data, extracting words as corresponding feature slot labels, and obtaining a feature sequence of the text data, wherein the feature sequence is composed of a plurality of keyword slot labels;

[0009] Classifying the text data according to the feature sequence;

[0010] The text data that are grouped into the same category are displayed and the annotation consistency is checked.

[0011] As a further improvement of the present invention, the feature extraction of the text data to extract words as corresponding slot labels specifically includes:

[0012] The text data is subjected to feature extraction, and according to the word types in the text data, the text data is extracted into verb slots, target entity slots, auxiliary word slots, quantifier slots, directional word slots, regional word slots, verb slots and question word slots.

[0013] As a further improvement of the present invention, classifying the text data according to the slot labels of the feature sequence specifically includes:

[0014] The text data is classified according to the feature sequence, and the text data having the same slot labels and slot label arrangement order are grouped into one category.

[0015] As a further improvement of the present invention, it also includes:

[0016] When a word in the text data has no corresponding slot label, it is represented by an empty string.

[0017] As a further improvement of the present invention, the text data is corpus data obtained by transcription.

[0018] The present invention also provides a text data annotation consistency detection device, which comprises:

[0019] A feature extraction module is configured to perform feature extraction on the text data, extract words as corresponding feature slot labels, and obtain a feature sequence of the text data, wherein the feature sequence is composed of a plurality of keyword slot labels;

[0020] A classification module, configured to classify the text data according to the feature sequence;

[0021] The display module is configured to display text data classified into the same category.

[0022] As a further improvement of the present invention, the feature extraction module is configured to: perform feature extraction on the text data, and extract it into verb slots, target entity slots, auxiliary word slots, quantifier slots, directional word slots, regional word slots, verb slots and question word slots according to the word types in the text data.

[0023] As a further improvement of the present invention, the classification module is configured to: classify the text data according to the feature sequence, and group the text data with the same slot labels and slot label arrangement order into one category.

[0024] As a further improvement of the present invention, the feature extraction module is further configured to: when a word in the text data has no corresponding slot label, it is represented by an empty string.

[0025] The present invention also provides a refrigeration device, comprising:

[0026] A memory for storing executable instructions;

[0027] The processor is used to implement the above-mentioned text data annotation consistency detection method when running the executable instructions stored in the memory.

[0028] The present invention also provides a computer storage medium, in which a computer program is stored, and the computer program is used to execute the above-mentioned text data annotation consistency detection method.

[0029] The beneficial effects of the present invention are as follows: the present invention performs feature extraction on the annotated text data, converts the text data into a feature sequence composed of keyword slot labels, and classifies the text data sequence based on the feature sequence, thereby significantly improving the accuracy of text classification, classifying texts with the same sentence structure into one category, and more intuitively reflecting the consistency of corpus annotation, thereby providing effective assistance for checking the consistency of annotations and providing higher quality training corpus for subsequent natural language processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the steps of a text data annotation consistency detection method in one embodiment of the present invention.

[0031] Figure 2 It is a schematic diagram of a characteristic sequence in one embodiment of the present invention.

[0032] Figure 3 It is a schematic diagram of characteristic sequence classification in one embodiment of the present invention.

[0033] Figure 4 It is a structural diagram of a text data annotation consistency detection device in one embodiment of the invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in combination with the specific implementation methods of the present invention and the corresponding drawings. Obviously, the described implementation methods are only part of the implementation methods of the present invention, not all implementation methods. Based on the implementation methods of the present invention, all other implementation methods obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0035] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0036] This implementation can significantly improve the accuracy of text classification by extracting features from annotated text data, converting the text data into a feature sequence composed of keyword slot labels, and classifying the text data based on the feature sequence. Texts with the same sentence structure can be classified into one category, thereby more intuitively reflecting the consistency of corpus annotation and providing effective assistance for checking annotation consistency.

[0037] like Figure 1 As shown, this embodiment provides a text data annotation consistency detection method, which is characterized by comprising the steps of:

[0038] S1: Obtain the annotated text data.

[0039] S2: Perform feature extraction on the text data, extract words as corresponding feature slot tags, and obtain a feature sequence of the text data, wherein the feature sequence is composed of a plurality of keyword slot tags.

[0040] S3: Classifying the text data according to the feature sequence.

[0041] S4: Display the text data that are classified into the same category and check the consistency of annotations.

[0042] In step S1, the method provided by the present invention is used to classify text data, and is used to assist in detecting the consistency of text data annotation, so as to improve the quality of training corpus for voice interaction modules of intelligent electronic devices. In this embodiment, the method is described by taking a smart refrigerator as an example. The text data in this embodiment is user corpus data obtained by transcription. Based on the user's voice input, the voice is transcribed into text and then annotated, and the text content corresponding to the user's voice is classified. The existing method can be referred to for transcribing voice into text data, which will not be repeated here. The text data is annotated, that is, texts with the same form, such as sentence types, are annotated with the same type of type label.

[0043] In step S2, it specifically includes:

[0044] The text data is subjected to feature extraction, and according to the word types in the text data, the text data is extracted into verb slots, target entity slots, auxiliary word slots, quantifier slots, directional word slots, regional word slots, verb slots and question word slots.

[0045] In the application scenario of a smart refrigerator, by converting specific keywords into the above-mentioned feature slots, it is possible to better represent the semantics and structure of the text in a targeted manner. The verb slot consists of words such as "put", "add", etc. that represent specific actions or behaviors, and it is used to reflect the specific actions in the text. The target entity slot consists of words such as "Chinese cabbage", "beef", etc. that represent specific entities, and it is used to reflect the main subjects or objects involved in the text. The particle slot consists of words such as "le" that are used to modify verbs or adjectives. The quantifier slot consists of words such as "one", "two kilograms", etc. that represent quantities or counts, and it is used to reflect information about quantities or degrees involved in the text. The locative slot consists of words such as "inside", "on", etc. that are used to represent positions or directions. The region slot consists of words such as "refrigerator compartment", "freezer compartment", etc. that represent regional scopes. Taking them as the features of the region slot, together with the locative words, they are used to reflect the location area and spatial scope where the action occurs in the text, or the location area and spatial scope where the target entity is located. The interrogative slot consists of words such as "ma", "is it", etc. that represent interrogative moods, and it is used to reflect whether the text is an interrogative sentence or a question that requires an answer.

[0046] Exemplarily, the label of the verb slot is verb_put and verb_have, the label of the target entity slot is FD_slots, the label of the particle slot is particle_le1, the label of the quantifier slot is FDquantity, the label of the locative slot is localizer_in, the label of the region slot is ref_dev, and the label of the interrogative slot is yn_form.

[0047] As Figure 2 shown, it is an example of the feature sequence of the text data related to the smart refrigerator.

[0048] The feature sequence of "add cucumber" is ('verb_put', 'FD_slots').

[0049] The feature sequence of "added one tomato" is ('verb_put', 'particle_le1', 'FDquantity', 'FD_slots').

[0050] The feature sequence of "add three eggplants inside" is ('FDquantity', 'FD_slots','verb_put', 'localizer_in').

[0051] The feature sequence of "Is there meat in the freezer compartment?" is ('ref_dev', 'localizer_in','verb_have', 'FD_slots', 'yn_form', '?').

[0052] The feature sequence of "two more cabbages were added" is ('verb_put', 'particle_le1', 'FDquantity', 'FD_slots').

[0053] The feature sequence of "Is there any cabbage in the refrigerator?" is ('ref_dev', 'localizer_in', 'verb_have', 'FD_slots', 'yn_form', '?').

[0054] The feature sequence of "two bundles of rapeseed are put inside" is ('FDquantity', 'FD_slots', 'verb_put', 'localizer_in').

[0055] The feature sequence for "add tomatoes" is ('verb_put', 'FD_slots').

[0056] By extracting features for different word types, we can obtain rich and diverse slot information, thereby more comprehensively representing the semantics and structure of the text, which is conducive to improving the ability to understand text data and providing more accurate and rich information for subsequent semantic understanding and natural language processing tasks.

[0057] In other embodiments of the present invention, users may have different idioms and ways of communication for different electronic devices. The categories of slot labels and the keywords corresponding to the slot labels may be adjusted according to the specific application scenarios of the electronic devices, and the slot labels may be customized to meet the voice interaction scenarios and usage requirements of different electronic devices. For example, in a smart speaker, you can focus on voice commands related to music playback, weather query, and schedule. In this case, specific slot labels related to these scenarios, such as music slots, weather slots, and schedule slots, can be set to capture key information for specific scenarios.

[0058] Furthermore, when a word in the text data does not have a corresponding slot label, an empty string is used to represent it. By using an empty string to represent a missing slot, the format of the text data can be kept consistent, which facilitates the subsequent feature extraction and processing process and avoids processing confusion caused by inconsistent data formats. When performing feature extraction, data classification and other steps, data can be simply filtered or filled based on the empty string to ensure the integrity and accuracy of the data.

[0059] In step S3, it specifically includes:

[0060] The text data is classified according to the feature sequence, and the text data having the same slot labels and slot label arrangement order are grouped into one category.

[0061] Text data with the same slot labels and slot label arrangement order are classified into one category, that is, one category of text represents a data set with similar semantic and sentence characteristics. This classification method helps to put text data with similar semantic intentions and sentence types in the same category, so as to better capture the commonality and similarity of text data, build an orderly corpus, and facilitate subsequent annotation consistency detection.

[0062] like Figure 3 As shown, the feature sequence examples above can be classified into four categories.

[0063] "Add cucumber" and "Add tomato" are of the same category, and their feature sequences are both ('verb_put', 'FD_slots').

[0064] "A tomato was added" and "Two cabbages were added" are of the same category, and their feature sequences are both ('verb_put', 'particle_le1', 'FDquantity', 'FD_slots').

[0065] "Three eggplants and put it inside" and "two bundles of rapeseed put inside" are in the same category, and their feature sequences are ('FDquantity', 'FD_slots', 'verb_put', 'localizer_in').

[0066] "Is there any meat in the freezer?" and "Is there any cabbage in the refrigerator?" are in the same category, and their feature sequences are ('ref_dev', 'localizer_in', 'verb_have', 'FD_slots', 'yn_form', '?').

[0067] The feature sequence of "two more cabbages were added" is ('verb_put', 'particle_le1', 'FDquantity', 'FD_slots').

[0068] The classification method for feature sequences may adopt existing clustering methods such as K-Means clustering, hierarchical clustering, etc., and this embodiment does not impose any specific limitation thereto.

[0069] In step S4, the categorized text data may be manually checked for consistency in annotations. The text data may be presented to quality inspectors using an annotation platform or other visualization tools to present the text data of each category.

[0070] By extracting features from the annotated text data, converting the text data into a feature sequence composed of keyword slot labels, and classifying the text data sequence based on the feature sequence, the accuracy of text classification can be significantly improved. Texts with the same sentence structure can be grouped together, which can more intuitively reflect the consistency of corpus annotations, thereby providing effective assistance for checking annotation consistency.

[0071] like Figure 4 As shown, the present invention also provides a text data annotation consistency detection device, which includes:

[0072] Feature extraction module 1 is configured to perform feature extraction on the text data, extract words as corresponding feature slot tags, and obtain a feature sequence of the text data, wherein the feature sequence is composed of multiple keyword slot tags.

[0073] Classification module 2, which is configured to classify the text data according to the feature sequence

[0074] The display module 3 is configured to display text data classified into the same category.

[0075] Furthermore, the feature extraction module 1 is also configured to: perform feature extraction on the text data, and extract it into verb slots, target entity slots, auxiliary word slots, quantifier slots, directional word slots, regional word slots, verb slots and question word slots according to the word types in the text data.

[0076] Furthermore, the classification module 2 is configured to classify the text data according to the feature sequence, and group the text data with the same slot labels and slot label arrangement order into one category.

[0077] Furthermore, the feature extraction module 1 is also configured to: when a word in the text data has no corresponding slot label, it is represented by an empty string.

[0078] The present invention also provides a refrigeration device, comprising:

[0079] A memory is used to store executable instructions.

[0080] The processor is used to implement the above-mentioned text data annotation consistency detection method when running the executable instructions stored in the memory.

[0081] The present invention also provides a computer storage medium, in which a computer program is stored, and the computer program is used to execute the above-mentioned text data annotation consistency detection method.

[0082] To summarize, this implementation method can significantly improve the accuracy of text classification by extracting features from the annotated text data, converting the text data into a feature sequence composed of keyword slot labels, and classifying the text data sequence based on the feature sequence. Texts with the same sentence structure can be classified into one category, which can more intuitively reflect the consistency of corpus annotation, thereby providing effective assistance for checking the consistency of annotation and providing higher quality training corpus for subsequent natural language processing tasks.

[0083] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0084] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A text data annotation consistency detection method, characterized in that: Includes steps: Get the annotated text data; Performing feature extraction on the text data, extracting words as corresponding feature slot labels, and obtaining a feature sequence of the text data, wherein the feature sequence is composed of a plurality of keyword slot labels; Classifying the text data according to the feature sequence; The text data that are grouped into the same category are displayed and the annotation consistency is checked.

2. The text data annotation consistency detection method according to claim 1 is characterized in that: The feature extraction of the text data to extract words as corresponding slot labels specifically includes: The text data is subjected to feature extraction, and according to the word types in the text data, the text data is extracted into verb slots, target entity slots, auxiliary word slots, quantifier slots, directional word slots, regional word slots, verb slots and question word slots.

3. The text data annotation consistency detection method according to claim 2 is characterized in that: The classifying the text data according to the slot labels of the feature sequence specifically includes: The text data is classified according to the feature sequence, and the text data having the same slot labels and slot label arrangement order are grouped into one category.

4. The text data annotation consistency detection method according to claim 2 is characterized in that: Also includes: When a word in the text data has no corresponding slot label, it is represented by an empty string.

5. The text data annotation consistency detection method according to claim 1 is characterized in that: The text data is corpus data obtained by transcription.

6. A text data annotation consistency detection device, characterized in that: include: A feature extraction module is configured to perform feature extraction on the text data, extract words as corresponding feature slot labels, and obtain a feature sequence of the text data, wherein the feature sequence is composed of a plurality of keyword slot labels; A classification module, configured to classify the text data according to the feature sequence; The display module is configured to display text data classified into the same category.

7. The text data annotation consistency detection device according to claim 6, characterized in that: The feature extraction module is configured to: perform feature extraction on the text data, and extract the features into verb slots, target entity slots, auxiliary word slots, quantifier slots, directional word slots, regional word slots, verb slots and question word slots according to the word types in the text data.

8. The text data annotation consistency detection device according to claim 6, characterized in that: The classification module is configured to classify the text data according to the feature sequence, and group the text data with the same slot labels and slot label arrangement order into one category.

9. The text data annotation consistency detection device according to claim 6, characterized in that: The feature extraction module is also configured to: when a word in the text data has no corresponding slot label, represent it with an empty string.

10. A refrigeration device, characterized in that: include: A memory for storing executable instructions; A processor is used to implement the text data annotation consistency detection method described in any one of claims 1 to 5 when running the executable instructions stored in the memory.

11. A computer storage medium, characterized in that: A computer program is stored therein, which is used to execute the text data annotation consistency detection method described in any one of claims 1 to 5.

Images