Tibetan syllable proofreading annotation data set construction method

Abstract

The invention discloses a Tibetan syllable proofreading annotation data set construction method, which comprises the steps of determining error types of a text, constructing a confusion set and a noise adding algorithm, and is characterized in that the error types comprise spelling errors and true word errors; the step of constructing the confusion set comprises the substeps of constructing a spelling error syllable confusion set, constructing a similar syllable confusion set, constructing a similar syllable confusion set, constructing a verb tense confusion set, constructing an error-prone virtual word confusion set and constructing confusion sets of different syllables; the noise addition algorithm comprises a joint noise addition algorithm matched with the confusion set and is used for reserving language information association between the original syllable and the replacement syllable by utilizing the confusion set. The invention relates to the technical field of language learning. According to the Tibetan syllable proofreading annotation data set construction method, through establishment of an algorithm and the data set, a computer can complete data annotation work of thousands of people in several minutes through the algorithm, a large amount of manpower resources are saved, time is saved, and the situation that errors are likely to be generated due to manual annotation is avoided.

Description

Technical Field

[0001] The invention relates to the technical field of language learning, and in particular to a method for constructing a Tibetan syllable proofreading and annotation data set. Background Art

[0002] Labeling data in the field of natural language processing and Tibetan text proofreading is a time-consuming and laborious task. Whether it is statistical machine learning, deep learning, or pre-trained language model training and testing, high-quality labeled datasets are required.

[0003] At present, there is no public high-quality annotated dataset that conforms to the error distribution in the field of automatic proofreading of Tibetan texts. Manually collecting and organizing annotated datasets consumes a lot of manpower and material resources. How to automatically construct a Tibetan error dataset that conforms to the error distribution plays a vital role in the study of automatic proofreading of Tibetan texts.

[0004] To this end, the present invention provides a method for constructing a Tibetan syllable proofreading and annotation dataset to solve the above problems. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a method for constructing a Tibetan syllable proofreading and annotation dataset to solve the above problems.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for constructing a Tibetan syllable proofreading and annotation data set, including determining the error type of the text, constructing a confusion set and a noise adding algorithm, wherein the error type includes spelling errors and real word errors;

[0007] The construction of the confusion set includes: constructing a confusion set of misspelled syllables, constructing a confusion set of similar-looking syllables, constructing a confusion set of similar-sounding syllables, constructing a confusion set of verb tenses, constructing a confusion set of easy-to-make function words, and constructing confusion sets of different syllables;

[0008] The noise adding algorithm includes a joint noise adding algorithm matched with the confusion set, which is used to retain the language information association between the original syllable and the replacement syllable by using the confusion set, such as information such as similar pronunciation, similar form, tense, collocation of function words and spelling errors, and to add noise to the training data by randomly replacing the syllables in the correct sentence by the confusion set.

[0009] Preferably, the spelling error itself does not conform to the Tibetan syllable spelling standard, and no context information is needed to determine the spelling error;

[0010] The real word errors include homophone replacement errors, shape errors, verb tense errors and function word collocation errors. The real word errors conform to the Tibetan syllable spelling standards, cannot be combined with the previous and next characters to form words with semantic information or do not conform to the current context. Judging real word errors requires combining contextual information.

[0011] Preferably, constructing a mis-spelled syllable confusion set includes a non-component set and a syllable component recognition algorithm, specifically replacing a certain component of a correct syllable to automatically generate a dynamic set of incorrect syllables, and a Tibetan syllable has a maximum of 7 characters and a minimum of 1 character, and its component set and non-component set are as follows:

[0012] Let B represent the base word set

[0013] Let Pr represent the set of prefix characters

[0014] Let U represent the set of added characters, then:

[0015] Let D represent the set of added characters

[0016] Let S represent the set of suffixes

[0017] Let ES represent the set of characters added later

[0018] Let Tvowel represent the vowel set

[0019] Set the non-prefixed word set

[0020] Set the non-superimposed word set

[0021] Set non-added characters set

[0022] Set the non-post-added word set

[0023] Set non-post-add word set

[0024] Among them, the NPr, NU, ND, NS, NES set and the syllable component recognition algorithm can automatically generate misspelled syllables. The specific algorithm can be described as follows: first, the syllable is processed using the syllable component recognition algorithm to split it into a component set, such as Decompose into a component set of {"", "upper added character", "base character", "lower added character", "", "vowel", "suffixed character", ""}. Next, remove the components with empty characters, randomly select a non-vowel, non-base character component, and non-lower added character from the remaining components, and randomly select a letter from the subset corresponding to its component position in the non-component set to replace it, and generate a misspelled syllable. The incorrect syllable confusion set is the process of automatically generating incorrect syllables through the algorithm, as shown in Algorithm 1:

[0025]

[0026]

[0027] Preferably, the construction of the confusion set of similar syllables is a component recognition algorithm and a set of similar-shaped components, replacing a component in the original syllable to automatically generate a dynamic set of similar-shaped characters. The specific algorithm is: using the component recognition algorithm on the syllable, splitting it into a set of components, removing components with empty characters, and randomly selecting a non-vowel and non-subscript component from the remaining components. If the selected component belongs to a group in the above-mentioned set of similar-shaped components, a different component is selected from the group to replace it, as shown in Algorithm 2:

[0028]

[0029] Preferably, the construction of the similar-sounding syllable confusion set is to establish the confusion set in a form similar to the homophone library. The specific algorithm is: first, each basic word is classified according to the "similar sound", and 52 groups of "similar-sounding" Tibetan basic words are established. The number of basic words in each group is at least 1 and at most 12. The 52 groups of similar-sounding initials are used as the "skeleton" for generating similar-sounding words; secondly, the basic words, vowels and suffixes are set to three sets A, B, and C, and the three sets are respectively represented as: basic word set The A set is 52 groups of basic characters with similar pronunciations, the B set is a set of vowel characters, and the C set is a set of post-added characters and further post-added characters, also known as the suffix set. The main idea of ​​the algorithm is that each element of the same group in the set A is combined with an element of the set B and an element of the set C to form a group of similar pronunciation characters, such as A group of similar-sounding characters can be formed, as shown in Algorithm 3:

[0031]

[0032] Preferably, a verb tense confusion set is constructed, and Tibetan verbs are divided into three tenses and one form: past tense, present tense, future tense and imperative tense. The specific method is: first, the verbs with only one form are deleted from the collected 1654 monosyllabic verbs, and then the tenses of the remaining verbs are deduplicated, and the repeated tenses are deleted to obtain 836 monosyllabic verbs with different verb forms as the verb tense confusion set.

[0033] Preferably, a confusion set of easy-to-make mistakes function words is constructed. Tibetan function words are divided into free function words and unfree function words. Free function words are not restricted by the continuation rules. Unfree function words have multiple forms and have fixed rules when used. They are restricted by the continuation rules. The continuation rules refer to that the continuation of unfree function words is affected by the phonetic strength of the word added after the previous syllable. Unfree function words have different variant forms.

[0034] The specific method is: collect the variants and continuation rules of various types of non-free function words to construct confusion sets of easily-mistaken function words. There are nine types of non-free function words, and 11 different confusion sets are established.

[0035] Preferably, the noise adding algorithm is specifically shown in Algorithm 4:

[0036]

[0037] s←Use the component recognition algorithm to split the syllable t into a set of components, and randomly select a non-vowel component from the components with non-blank characters

[0038] if s∈some subset e in E

[0039] t←Select a component from e that is different from s and replace it to generate a word similar to t

[0040] / / Noise adding method three

[0041] s←Use the component recognition algorithm to split the syllable t into a set of components, and randomly select a non-vowel, non-base character component from the components with non-blank characters

[0042] t←According to the selected component, determine the corresponding subset w in the non-component set W, randomly select a component from w to replace s, and generate a syllable that does not conform to the spelling rules

[0043] end for

[0044] end for

[0045] return Noise-added dataset

[0046] Beneficial Effects

[0047] The present invention provides a method for constructing a Tibetan syllable proofreading and annotation dataset. Compared with the prior art, it has the following beneficial effects:

[0048] (1) In the construction of Tibetan syllable proofreading and annotation dataset, labeling data is labor-intensive.

[0049]

[0050] This is a huge project. Through the establishment of algorithms and data sets, computers can complete the data labeling work of thousands of people in just a few minutes, saving a lot of human resources.

[0051] (2) In the construction of the Tibetan syllable proofreading and annotation dataset, the traditional data annotation project requires thousands of people to annotate and takes many days to complete, while the computer can complete the annotation in a few minutes. The annotation quality is uniform and consistent, and no human errors will occur. This saves time and avoids the errors that are easy to occur in manual annotation.

[0052] (3) The method for constructing the Tibetan syllable proofreading and annotation dataset avoids the traditional practice of one person annotating data for one day, which would cost hundreds to thousands of dollars, and thousands of people annotating data for one day, which would cost millions of dollars. In the long run, this saves considerable economic consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a Tibetan syllable structure diagram in a Tibetan syllable proofreading and annotation data set construction method of the present invention.

[0054] In the figure, 1 is the front added character position; 2 is the upper added character position; 3 is the base character position; 4 is the lower added character position; 5 is the vowel position; 6 is the back added character position; 7 is the third back added character position. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] See also Figure 1 , a method for constructing a Tibetan syllable proofreading annotation dataset, including determining the error types of the text, constructing a confusion set and a noise adding algorithm, the error types include spelling errors and real word errors;

[0057] Constructing confusion sets includes: constructing a confusion set of misspelled syllables, constructing a confusion set of similar-looking syllables, constructing a confusion set of similar-sounding syllables, constructing a confusion set of verb tenses, constructing a confusion set of easy-to-make function words, and constructing confusion sets of different syllables;

[0058] The noise adding algorithm includes a joint noise adding algorithm matched with the confusion set, which is used to retain the language information association between the original syllable and the replacement syllable by using the confusion set, such as information such as similar pronunciation, similar form, tense, collocation of function words and spelling errors, and to add noise to the training data by randomly replacing the syllables in the correct sentence by the confusion set;

[0059] Spelling errors do not conform to the Tibetan syllable spelling standards. No context information is needed to determine spelling errors.

[0060] Real word errors include homophone replacement errors, shape errors, verb tense errors, and function word collocation errors. Real word errors conform to the Tibetan syllable spelling standards, cannot be combined with the preceding and following characters to form a word with semantic information, or do not conform to the current context. To determine real word errors, contextual information is needed;

[0061] Constructing a confusion set of misspelled syllables: The confusion set of misspelled syllables is a dynamic set of incorrect syllables automatically generated by replacing a component of a correct syllable through a syllable component recognition algorithm and a non-component set. According to the spelling rules of Tibetan, the number and nature of the letters that appear in each component position in a Tibetan syllable are subject to certain constraints, and there are restrictive relationships between them. In addition, among the 7 character positions of a syllable, except for the base character position, there can be no characters in the other positions. Therefore, a Tibetan syllable has a maximum of 7 characters and a minimum of 1 character. Its component set and non-component set are as follows:

[0062] Let B represent the base word set

[0063] Let Pr represent the set of prefix characters

[0064] Let U represent the set of added characters, then:

[0065] Let D represent the set of added characters

[0066] Let S represent the set of suffixes

[0067] Let ES represent the set of characters added later

[0068] Let Tvowel represent the vowel set

[0069] Set the non-prefixed word set

[0070] Set the non-superimposed word set

[0071] Set non-added characters set

[0072] Set the non-post-added word set

[0073] Set non-post-add word set

[0074] Among them, the NPr, NU, ND, NS, NES set and the syllable component recognition algorithm can automatically generate misspelled syllables. The specific algorithm can be described as follows: first, the syllable is processed using the syllable component recognition algorithm to split it into a component set, such as Decompose into a component set of {"", "superimposed character", "base character", "subimposed character", "", "vowel", "subimposed character", ""}. Next, remove the components with empty characters, randomly select a non-vowel, non-base character component, and non-subimposed character from the remaining components, and randomly select a letter from the subset corresponding to its component position in the non-component set to replace it, and generate a misspelled syllable. For example, if a superimposed character of a syllable is selected, a letter is randomly selected from the subset NPr to replace it. Therefore, the incorrect syllable confusion set here is not a fixed set, but a process of automatically generating incorrect syllables through an algorithm, as shown in Algorithm 1:

[0075]

[0076]

[0077] Constructing a confusion set of similar syllables is a component recognition algorithm and a set of components with similar shapes. Replace a component in the original syllable to automatically generate a dynamic set of similar characters. The similar shapes of Tibetan syllables are mainly due to the similar components. Therefore, components with similar shapes are collected to establish a set of similar components. 35 groups of components with similar shapes are collected, such as and and With a set of similar components, combined with a component recognition algorithm, similar characters can be automatically generated. The specific algorithm is: use the component recognition algorithm on the syllable to split it into a set of components, remove the components with empty characters, and randomly select a non-vowel and non-subscript component from the remaining components. If the selected component belongs to a group in the above-mentioned set of similar components, select a different component from the group to replace it, such as replacing the syllable After splitting into component sets, the components in them are selected From including A component that is different from the subset of components with similar shapes is randomly selected Replace it to generate ' Similar to Algorithm 1, the details are shown in Algorithm 2:

[0078]

[0079]

[0080] The construction of the confusion set of similar sounding syllables is to establish the confusion set in a form similar to the homophone library. Tibetan syllables are similar to Chinese pinyin and have a certain structure of initials and finals. The base character, the front added character, the upper added character and the lower added character constitute the initial consonant part of the syllable, which is called the basic character. The back added character, the second back added character and the vowel constitute the final part of the syllable. The pronunciation of the initial consonant in Tibetan determines the basic pronunciation of a syllable. The pronunciation of the basic characters similar to each other can form similar initial consonants. The specific algorithm is: first, classify each basic character according to the "similar sound" and establish 52 groups of "similar sound" Tibetan basic characters. The number of basic characters in each group is at least 1 and at most 12. The 52 groups of similar initial consonants are used as the "skeleton" for generating similar sounding characters; secondly, the basic characters, vowels and suffixes are set to three sets A, B, and C. The three sets are represented as: basic character set

[0081] The A set is 52 groups of basic characters with similar pronunciations, the B set is a set of vowel characters, and the C set is a set of post-added characters and further post-added characters, also known as the suffix set. The main idea of ​​the algorithm is that each element of the same group in the set A is combined with an element of the set B and an element of the set C to form a group of similar pronunciation characters, such as A group of similar-sounding characters can be formed, as shown in Algorithm 3:

[0082]

[0083]

[0084] Constructing a verb tense confusion set. Tibetan verb tenses are relatively rich. Generally, Tibetan verbs can be divided into three tenses and one form: past tense, present tense, future tense and imperative tense. To construct a verb tense confusion set, the 1654 collected monosyllabic verbs are first deleted from the verbs with only one form; then the tenses of the remaining verbs are deduplicated, and the repeated tenses are deleted. After the above processing, a total of 836 monosyllabic verbs with different verb forms are obtained as the verb tense confusion set;

[0085] Construct a confusion set of easy-to-make mistakes function words. Tibetan function words can be divided into free function words and unfree function words. Free function words are not restricted by the continuation rules; unfree function words are just the opposite. They have multiple forms and have fixed rules when used. They are restricted by the continuation rules. The continuation rules mean that the continuation of unfree function words is affected by the phonetic strength of the word added after the previous syllable. Unfree function words have different variant forms. If used improperly, they will violate the continuation rules and cause function word collocation errors. Collect the variants and continuation rules of various unfree function words to construct a confusion set of easy-to-make mistakes function words. There are nine types of unfree function words. 11 different confusion sets are established. For example, the collocation errors between the genitive and the ergative can be established. Such a confusing set;

[0086] Noising algorithm: In terms of the noising algorithm, a joint noising algorithm is proposed based on different confusion sets, and the correct text is randomly noised to automatically generate large-scale high-quality annotated data. The purpose of the noising algorithm is to use the confusion set to retain the language information association between the original syllable and the replaced syllable, such as similar sound, similar shape, tense, function word collocation and spelling errors. The syllables in the correct sentence are randomly replaced by the confusion set to add noise to the training data, making it more realistic and close to the real scene. With the above-mentioned confusion sets that automatically generate incorrect syllables, similar syllables, similar syllables, verb tenses and easily wrong function words, the correct text can be automatically noised in conjunction with the noising algorithm to generate Tibetan proofreading error annotated corpus, as shown in Algorithm 4:

[0087]

[0088]

[0089] The longer the sentence is, the greater the probability of an incorrect syllable in the sentence. Therefore, the use of the hyperparameter percentage can effectively control the number of incorrect syllables in each sentence. The value of percentage is a decimal between 0 and 1.

[0090] After adding noise, the style of the labeled data set is shown in the following table:

[0091]

[0092] The specific method is as follows: according to Algorithm 1, a confusion set of misspelled syllables is established. Generally, a syllable is randomly selected in the sentence, and the syllable component recognition algorithm is used to split the components of the syllable. Then, Algorithm 1 is used to replace one of the components to generate an incorrect syllable. According to Algorithm 2, a confusion set of similar syllables is established. The steps are similar to Algorithm 1. According to Algorithm 3, a homophone set is established, and then a noisy algorithm is used to generate a labeling set.

[0093] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0094] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a Tibetan syllable proofreading and annotation dataset, comprising determining the error type of the text, constructing a confusion set and a noise adding algorithm, characterized in that: The error types mentioned include spelling errors and real word errors; The construction of the confusion set includes: constructing a confusion set of misspelled syllables, constructing a confusion set of similar-looking syllables, constructing a confusion set of similar-sounding syllables, constructing a confusion set of verb tenses, constructing a confusion set of easy-to-make function words, and constructing confusion sets of different syllables; The noise adding algorithm includes a joint noise adding algorithm matched with the confusion set, which is used to retain the language information association between the original syllable and the replacement syllable by using the confusion set, such as information such as similar pronunciation, similar form, tense, collocation of function words and spelling errors, and to add noise to the training data by randomly replacing the syllables in the correct sentence by the confusion set.

2. The method for constructing a Tibetan syllable proofreading and annotation dataset according to claim 1, characterized in that: The spelling error itself does not conform to the Tibetan syllable spelling standard, and no context information is needed to determine the spelling error; The real word errors include homophone replacement errors, shape errors, verb tense errors and function word collocation errors. The real word errors conform to the Tibetan syllable spelling standards, cannot be combined with the previous and next characters to form words with semantic information or do not conform to the current context. Judging real word errors requires combining contextual information.

3. The method for constructing a Tibetan syllable proofreading and annotation dataset according to claim 1, characterized in that: The construction of the misspelled syllable confusion set includes a non-component set and a syllable component recognition algorithm. Specifically, a dynamic set of incorrect syllables is automatically generated by replacing a component of the correct syllable. The Tibetan syllable has a maximum of 7 characters and a minimum of 1 character. The component set and non-component set are as follows: Let B represent the base word set Let Pr represent the set of prefix characters Let U represent the set of added characters, then: Let D represent the set of added characters Let S represent the set of suffixes Let ES represent the set of characters added later Let Tvowel represent the vowel set Set the non-prefixed word set Set the non-superimposed word set Set non-added characters set Set the non-post-added word set Set non-post-add word set Among them, the NPr, NU, ND, NS, NES set and the syllable component recognition algorithm can automatically generate misspelled syllables. The specific algorithm can be described as follows: first, the syllable is processed using the syllable component recognition algorithm to split it into a component set, such as Decompose into a component set of {"","upper added character","base character","lower added character","","vowel","suffixed character",""}. Next, remove the components with empty characters, randomly select a non-vowel, non-base character component, and non-lower added character from the remaining components, and randomly select a letter from the subset corresponding to its component position in the non-component set to replace it, and generate a misspelled syllable. The incorrect syllable confusion set is the process of automatically generating incorrect syllables through the algorithm, as shown in Algorithm 1:

4. The method for constructing a Tibetan syllable proofreading and annotation dataset according to claim 1, characterized in that: Constructing a confusion set of similar syllables is a component recognition algorithm and a set of similar components. Replacing a component in the original syllable automatically generates a dynamic set of similar characters. The specific algorithm is: using the component recognition algorithm on the syllable, splitting it into a set of components, removing the components with empty characters, and randomly selecting a non-vowel and non-subscript component from the remaining components. If the selected component belongs to a group in the above-mentioned set of similar components, select a different component from the group to replace it, as shown in Algorithm 2:

5. The method for constructing a Tibetan syllable proofreading and annotation dataset according to claim 1, characterized in that: The construction of the confusion set of similar sound syllables is to establish the confusion set in a similar way to the homophone library. The specific algorithm is: first, classify each basic word according to the "similar sound", and establish 52 groups of "similar sound" Tibetan basic words. The number of basic words in each group is at least 1 and at most 12. The 52 groups of similar sound initials are used as the "skeleton" for generating similar sound words; secondly, the basic words, vowels and suffixes are set to three sets A, B, and C. The three sets are represented as: basic word set The A set is 52 groups of basic characters with similar pronunciations, the B set is a set of vowel characters, and the C set is a set of post-added characters and further post-added characters, also known as the suffix set. The main idea of ​​the algorithm is that each element of the same group in the set A is combined with an element of the set B and an element of the set C to form a group of similar pronunciation characters, such as A group of similar-sounding characters can be formed, as shown in Algorithm 3:

6. The method for constructing a Tibetan syllable proofreading and annotation dataset according to claim 1, characterized in that: Construct a verb tense confusion set. Tibetan verbs are divided into three tenses and one form: past tense, present tense, future tense and imperative tense. The specific method is as follows: first delete the verbs with only one form among the 1654 collected monosyllabic verbs, then deduplicate the tenses of the remaining verbs and delete the repeated tenses to obtain 836 monosyllabic verbs with different verb forms as the verb tense confusion set.

7. The method for constructing a Tibetan syllable proofreading and annotation dataset according to claim 1, characterized in that: Construct a confusion set of easy-to-make mistakes function words. Tibetan function words are divided into free function words and unfree function words. Free function words are not restricted by the continuation rules. Unfree function words have multiple forms and follow fixed rules when used. They are restricted by the continuation rules. The continuation rules mean that the continuation of unfree function words is affected by the pronunciation strength of the word added after the previous syllable. Unfree function words have different variant forms. The specific method is: collect the variants and continuation rules of various types of non-free function words to construct confusion sets of easily-mistaken function words. There are nine types of non-free function words, and 11 different confusion sets are established.

8. The method for constructing a Tibetan syllable proofreading and annotation dataset according to claim 1, characterized in that: The details of the noise adding algorithm are shown in Algorithm 4: The longer the sentence is, the greater the probability of an incorrect syllable in the sentence. Therefore, the use of the hyperparameter percentage can effectively control the number of incorrect syllables in each sentence. The value of percentage is a decimal between 0 and 1. After adding noise, the style of the labeled data set is shown in the following table: 。

Images