A multilingual text error correction method based on sequence labeling model

Abstract

A multilingual text error correction method based on a sequence annotation model comprises the following steps: step S1: corpus data collection; step S2: data generation; taking the multilingual Wikipedia corpus as a positive example, performing word segmentation on the sentence, and randomly selecting 15% of the words or characters, performing deletion, insertion or replacement operations to generate negative examples; step S3: data annotation; generating text error detection labels and correction behavior labels; and classifying and annotating the text error detection labels and correction behavior labels respectively; step S4: text error correction sequence annotation model training; step S5: text error correction alternative generation. The present invention overcomes the shortcomings of the prior art and solves the problem of high cost and difficult maintenance of establishing a text error correction model for each language.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a multilingual text error correction method based on a sequence annotation model. Background Art

[0002] As the number of non-native learners, writers and users continues to grow around the world, the demand for automated language assessments, such as text error correction, is also increasing. Among them, text error correction needs to automatically correct many errors in the original text, including grammar, spelling, content, etc., while maintaining the semantics of the original text.

[0003] Most existing text correction methods are oriented towards a single language. In a multilingual scenario, it is necessary to integrate text correction models for more than 100 languages. Therefore, this paper provides a general method for multilingual text correction to solve the problem of high cost and difficult maintenance of establishing a text correction model for each language. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a multilingual text correction method based on a sequence annotation model, which overcomes the shortcomings of the prior art and solves the problem of high cost and difficult maintenance of establishing a text correction model for each language.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0006] A multilingual text error correction method based on a sequence tagging model comprises the following steps:

[0007] Step S1: corpus data collection; select multiple languages ​​from the Wikidata multilingual knowledge base, obtain multilingual Wikipedia corpus, clean hypertext markup language, extract plain text, and segment and sentence the corpus;

[0008] Step S2: Data generation: Using multilingual Wikipedia corpus as positive examples, the sentences are segmented, and 15% of the words or characters are randomly selected for deletion, insertion or replacement to generate negative examples;

[0009] Step S3: Data labeling: Based on the positive example sentences, generate a text error detection label and a correction behavior label for each word or character in the negative example sentences; and classify and label the text error detection label and the correction behavior label respectively;

[0010] Step S4: training the multilingual text corpus through the sequence labeling model, and outputting the generated text error detection label type and the correction behavior label type with the highest probability;

[0011] Step S5: After predicting the correction behavior label, each word or character is added, deleted, or modified according to the corresponding label, and an alternative option is generated.

[0012] Preferably, the generation of negative examples in step S2 includes the following method:

[0013] Step S21: Randomly delete a word or character in the positive example;

[0014] Step S22: randomly insert a word or character of the same language into the positive example;

[0015] Step S23: Randomly replace a word or character of the same language in the positive example, and the edit distance between the replaced item and the replacement item does not exceed 2.

[0016] Preferably, the classification of the text error detection labels in step S3 includes a CORRECT label and an INCORRECT label, wherein the CORRECT label indicates that the word or character is correct, and the INCORRECT label indicates that the word or character is wrong;

[0017] The classification of the correction behavior label includes a KEEP label, an INSERT label, a DELETE label and a REPLACE label; wherein the KEEP label indicates that a word or character remains unchanged, the INSERT label indicates that a word or character is inserted before a word or character, the DELETE label indicates that the word or character is deleted, and the REPLACE label indicates that a word or character is replaced.

[0018] Preferably, the step S4 specifically includes the following steps:

[0019] Step S41: construct a dictionary of multilingual words or characters, and use the wordpiece algorithm to segment the corpus;

[0020] Step S42: embedding words or characters, mapping each word or character into a 768-dimensional word vector; mapping the position of the word or character to a 768-dimensional position vector;

[0021] Step S43: Add the 768-dimensional word vector and the position vector and input them into the multi-head attention layer to output a 768-dimensional hidden vector;

[0022] Step S44: Two fully connected layers are connected respectively, one fully connected layer is used to predict the error detection label, inputs a 768-dimensional hidden vector, and outputs a 2-dimensional vector, and the other fully connected layer is used to predict the correction behavior label, inputs a 768-layer hidden vector, and outputs a 4-dimensional vector;

[0023] Step S45: A softmax mapping layer is connected after the fully connected layer to perform normalization processing, and the probability of the wrong detection label and the probability of the corrected behavior label are calculated respectively;

[0024] Step S46: According to the probability distribution of the two labels, each word or character outputs the two label types with the highest probability.

[0025] Preferably, the step S4 further includes:

[0026] Step S47: loss function design:

[0027] The text error correction model includes two sequence labeling tasks, so the loss function of the model is divided into error detection loss function and correction behavior loss function:

[0028] loss=loss detection +loss correction

[0029]

[0030]

[0031] M d =2

[0032] M c =4

[0033]

[0034]

[0035] Among them, loss is the total loss function, loss detection is the error detection loss function, loss correction To correct the behavior type loss function, N is the number of words or characters in a single sentence, with a maximum of 60, and M d is the number of wrongly detected labels, M c To correct the number of behavior labels, is the weight of the j-th type of error detection label, is the CORRECT label weight, is the weight of the INCORRECT label, Correct the behavior label for the jth class, They are the weights of the KEEP tag, INSERT tag, REPLACE tag, and DELETE tag respectively.

[0036] Preferably, in step S5, after predicting the corrective action label, except for the KEEP label which remains unchanged, the remaining words or characters are added, deleted or modified according to the corresponding labels, and alternative options are generated; specifically, the following steps are performed:

[0037] (1) When the correction action label of a word or character is predicted to be a DELETE label, the word or character is directly deleted;

[0038] (2) When the correction action label of a word or character is predicted to be a REPLACE label, the word or character is masked and an alternative of the word or character is generated using the masking language model;

[0039] (3) When the correction action label of a word or character is predicted to be an INSERT label, a masking mark is added before the word or character, and an alternative is generated using the masking language model.

[0040] The present invention provides a multilingual text error correction method based on a sequence annotation model, which has the following beneficial effects: the general multilingual text error correction method provided by the present application solves the problem of high cost and difficult maintenance of establishing a text error correction model for each language. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the present invention or the technical solutions in the prior art, the drawings required for describing the prior art are briefly introduced below.

[0042] Figure 1 Flow chart of the present invention;

[0043] Figure 2 Schematic diagram of constructing positive examples, negative examples and labels in the present invention;

[0044] Figure 3 The text error correction sequence annotation model diagram of the present invention;

[0045] Figure 4 Operation diagram when the correction action label is predicted to be REPLACE in the present invention;

[0046] Figure 5 The operation diagram when the corrected action label is predicted to be INSERT in the present invention; DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention.

[0048] like Figure 1-5 As shown, a multilingual text error correction method based on a sequence tagging model includes the following steps:

[0049] Step S1: corpus data collection;

[0050] Select the top 100 languages ​​in the Wikidata multilingual knowledge base, download the Wikipedia corpus in these languages, clean the hypertext markup language, extract the plain text, and segment and sentence the corpus;

[0051] Step S2: data generation;

[0052] Using multilingual Wikipedia corpus as positive examples, we segment sentences and randomly select 15% of words or characters for deletion, insertion or replacement to generate negative examples.

[0053] Negative examples are generated in the following ways:

[0054] (21): Randomly delete a word or character in the positive example;

[0055] (22): Randomly insert a word or character of the same language into the positive example;

[0056] (23): Randomly replace a word or character in the same language in the positive example, and the edit distance between the replaced item and the replacement item does not exceed 2; for example, replace the English word apples with apple and the German word steigen with steigern, so that the generated negative example is closer to the actual error situation.

[0057] Step S3: Data labeling: Based on the positive example sentences, generate a text error detection label and a correction behavior label for each word or character in the negative example sentences; and classify and label the text error detection label and the correction behavior label respectively;

[0058] Among them, Figure 2 As shown, the classification of text error detection labels includes CORRECT labels and INCORRECT labels; the CORRECT label indicates that the word or character is correct, and the INCORRECT label indicates that the word or character is wrong;

[0059] The classification of correction behavior labels includes KEEP label, INSERT label, DELETE label and REPLACE label; wherein, KEEP label indicates that a word or character remains unchanged, INSERT label indicates that a word or character is inserted before a word or character, DELETE label indicates that the word or character is deleted, and REPLACE label indicates that a word or character is replaced.

[0060] Step S4: training the multilingual text corpus through the sequence labeling model, and outputting the generated text error detection label type and the correction behavior label type with the highest probability; specifically comprising the following steps:

[0061] Step S41: multilingual text segmentation;

[0062] Build a dictionary of multilingual words or characters and use the wordpiece algorithm to segment the corpus;

[0063] Step S42: word embedding layer;

[0064] Embed words or characters, map each word or character into a 768-dimensional word vector; map the position of the word or character (0-59) to a 768-dimensional position vector;

[0065] Step S43: multi-head attention layer;

[0066] The 768-dimensional word vector and position vector are added and input into the multi-head attention layer (12 heads and 6 layers), and a 768-dimensional hidden vector is output;

[0067] Step S44: fully connected layer;

[0068] They are connected to two fully connected layers, one fully connected layer is used to predict the error detection label, inputs a 768-dimensional hidden vector, and outputs a 2-dimensional vector. The other fully connected layer is used to predict the corrective behavior label, inputs a 768-layer hidden vector, and outputs a 4-dimensional vector.

[0069] Step S45: normalization processing;

[0070] After the fully connected layer, a softmax mapping layer is connected to perform normalization processing to calculate the probability of incorrect detection labels and the probability of corrected behavior labels respectively;

[0071] Step S46: output label;

[0072] According to the probability distribution of the two labels, each word or character outputs the two label types with the highest probability, such as Figure 3 shown.

[0073] Step S47: loss function design:

[0074] The text error correction model includes two sequence labeling tasks, so the loss function of the model is divided into error detection loss function and correction behavior loss function:

[0075] loss=loss detection +loss correction

[0076]

[0077]

[0078] Md =2

[0079] M c =4

[0080]

[0081]

[0082] Among them, loss is the total loss function, loss detection is the error detection loss function, loss correction To correct the behavior type loss function, N is the number of words or characters in a single sentence, with a maximum of 60, and M d is the number of wrongly detected labels, M c To correct the number of behavior labels, is the weight of the j-th type of error detection label, is the CORRECT label weight, is the weight of the INCORRECT label, Correct the behavior label for the jth class, They are the weights of the KEEP tag, INSERT tag, REPLACE tag, and DELETE tag respectively.

[0083] When data is generated, 15% of the words will be processed, and there is an imbalance between error detection labels and correction behavior labels. Most words or characters are CORRECT or KEEP labels, which seriously affects the parameter estimation of the model. Therefore, the loss function weights of the INCORRECT label and the INSERT label, REPLACE label, and DELETE label are increased to 6 and 24 respectively to reduce the impact of sample imbalance.

[0084] Among them, the ratio of training set and test set in the process of training the above sequence labeling model on multilingual text corpus is 97:3, the training set size is 97000000, the test set size is 3000000, the epoch is 10, the learning rate is 5e-5, and the evaluation indicators on the test set are as follows:

[0085] Table 1 Evaluation indicators

[0086] F1 Precision Recall Error Detection Tags 0.753 0.717 0.791 Corrective Behavior Tags 0.757 0.718 0.802

[0087] Step S5: After predicting the corrective action label, except for the KEEP label which remains unchanged, the remaining words or characters are added, deleted or modified according to the corresponding labels, and alternative options are generated; specifically, the following steps are performed:

[0088] (1) When the correction action label of a word or character is predicted to be a DELETE label, the word or character is directly deleted;

[0089] (2) When the correction action label of a word or character is predicted to be the REPLACE label, the word or character is masked and the masked language model is used to generate alternatives for the word or character (e.g. Figure 4 shown);

[0090] (3) When the correction action label of a word or character is predicted to be an INSERT label, a masking mark is added before the word or character, and the masking language model is used to generate alternatives (such as Figure 5 shown).

[0091] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multilingual text error correction method based on a sequence labeling model, characterized by: The following steps are involved: Step S1: corpus data collection; Select multiple languages ​​from the Wikidata multilingual knowledge base, obtain multilingual Wikipedia corpus, clean hypertext markup language, extract plain text, and segment and sentence the corpus; Step S2: Data generation: Using multilingual Wikipedia corpus as positive examples, the sentences are segmented, and 15% of the words or characters are randomly selected for deletion, insertion or replacement to generate negative examples; Step S3: data labeling; Based on the positive example sentences, a text error detection label and a correction behavior label are generated for each word or character in the negative example sentences; and the text error detection labels and the correction behavior labels are classified and labeled respectively; Step S4: training a text error correction sequence annotation model; training a multilingual text corpus through a sequence annotation model, and outputting the generated text error detection label type and the correction behavior label type with the highest probability; Step S5: Generating text error correction options; after predicting the correction behavior label, each word or character is added, deleted, or modified according to the corresponding label, and an option is generated; Wherein, the step S4 specifically includes the following steps: Step S41: construct a dictionary of multilingual words or characters, and use the wordpiece algorithm to segment the corpus; Step S42: embedding words or characters, mapping each word or character into a 768-dimensional word vector; mapping the position of the word or character to a 768-dimensional position vector; Step S43: Add the 768-dimensional word vector and the position vector and input them into the multi-head attention layer to output a 768-dimensional hidden vector; Step S44: Two fully connected layers are connected respectively, one fully connected layer is used to predict the error detection label, inputs a 768-dimensional hidden vector, and outputs a 2-dimensional vector, and the other fully connected layer is used to predict the correction behavior label, inputs a 768-layer hidden vector, and outputs a 4-dimensional vector; Step S45: A softmax mapping layer is connected after the fully connected layer to perform normalization processing, and the probability of the wrong detection label and the probability of the corrected behavior label are calculated respectively; Step S46: According to the probability distribution of the two labels, each word or character outputs the two label types with the highest probability.

2. The multilingual text error correction method based on a sequence labeling model according to claim 1, characterized in that: The generation of negative examples in step S2 includes the following methods: Step S21: Randomly delete a word or character in the positive example; Step S22: randomly insert a word or character of the same language into the positive example; Step S23: Randomly replace a word or character of the same language in the positive example, and the edit distance between the replaced item and the replacement item does not exceed 2.

3. The multilingual text error correction method based on a sequence labeling model according to claim 1, characterized in that: The classification of the text error detection label in step S3 includes a CORRECT label and an INCORRECT label, wherein the CORRECT label indicates that the word or character is correct, and the INCORRECT label indicates that the word or character is wrong; The classification of the correction behavior label includes a KEEP label, an INSERT label, a DELETE label and a REPLACE label; wherein the KEEP label indicates that a word or character remains unchanged, the INSERT label indicates that a word or character is inserted before a word or character, the DELETE label indicates that the word or character is deleted, and the REPLACE label indicates that a word or character is replaced.

4. The multilingual text error correction method based on a sequence labeling model according to claim 1, characterized in that: The step S4 further comprises: Step S47: loss function design: The text error correction model includes two sequence labeling tasks, so the loss function of the model is divided into error detection loss function and correction behavior loss function: loss=loss detection +loss correction M d ＝2 M c ＝4 Among them, loss is the total loss function, loss detection is the error detection loss function, loss correction To correct the behavior type loss function, N is the number of words or characters in a single sentence, with a maximum of 60, and M d is the number of wrongly detected labels, M c To correct the number of behavior labels, is the weight of the j-th type of error detection label, is the CORRECT label weight, is the weight of the INCORRECT label, Correct the behavior label for the jth class, They are the weights of the KEEP tag, INSERT tag, REPLACE tag, and DELETE tag respectively.

5. The multilingual text error correction method based on a sequence labeling model according to claim 4, characterized in that: In the step S5, after predicting the corrective action label, except for the KEEP label which remains unchanged, the remaining words or characters are added, deleted or modified according to the corresponding labels, and alternative options are generated; specifically, the following steps are performed: (1) When the correction action label of a word or character is predicted to be a DELETE label, the word or character is directly deleted; (2) When the correction action label of a word or character is predicted to be a REPLACE label, the word or character is masked and an alternative of the word or character is generated using the masking language model; (3) When the correction action label of a word or character is predicted to be an INSERT label, a masking mark is added before the word or character, and an alternative is generated using the masking language model.

Images