An intelligent error correction system for spelling errors in Chinese corpus

Abstract

The present invention discloses an intelligent error correction system for Chinese corpus spelling errors. The intelligent detector is composed of three modules: a text feature extraction module, a text error recognition module, and a text error correction module. The present invention does not require manual annotation of erroneous texts, and has the characteristics of simple implementation method, high recognition accuracy, good robustness, etc.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language text error correction, and in particular to an intelligent error correction system for spelling errors in Chinese corpus. Background Art

[0002] Text error correction is a technology that uses natural language processing technology to detect whether there are typos in a text and correct them. It is generally used in the text preprocessing stage. In the early days, this field mainly used expert rules (patterns) for error identification and calibration, but this method is time-consuming and labor-intensive, and the coverage and accuracy are very limited. In recent years, with the implementation of speech recognition technologies such as intelligent customer service and chatbots, the comprehensive performance of text error correction systems has been further improved based on data-driven methods, which has broad market prospects and application value.

[0003] As an important and challenging task, a well-working spelling error correction system essentially requires human-level language understanding capabilities. The current mainstream approach to solving this task is to use Bidirectional Encoder Representations from Transformers (BERT) to select a character from a list of candidate characters and correct it at each position in the sentence (including not correcting it). However, since BERT uses mask pre-training for language modeling, it does not have enough ability to explicitly detect whether there is an error at each position, and the accuracy of such methods may be suboptimal. Therefore, there is an urgent need for a text error correction system with high flexibility, high agility and high accuracy to solve the above problems. Summary of the invention

[0004] In order to overcome the above shortcomings of the current text classifier, the purpose of the present invention is to provide a two-stage intelligent error correction system for Chinese corpus spelling errors, which uses a text error recognition module and a text error correction module to realize the recognition and correction of erroneous characters, and the former is connected to the latter through soft masking technology. And the intelligent error correction system has high recognition accuracy and good robustness.

[0005] The technical solution adopted by the present invention to solve its technical problem is: an intelligent error correction system for Chinese corpus spelling errors, which consists of three modules: a text feature extraction module, a text error recognition module, and a text error correction module.

[0006] 1) Text feature extraction module: obtains dynamic encoding of corpus based on pre-trained language model;

[0007] 2) Text error recognition module: Based on the sequence language model, predict the probability of an error in each character in the text;

[0008] 3) Text error correction module: Mask the error probability based on the output of the text error recognition module and use the language model to predict the correct characters.

[0009] Furthermore, the text feature extraction module converts the text data into corpus dynamic encoding through the following steps:

[0010] 1) Pad the end with zeros or truncate to fix the length of each corpus. The processed corpus is recorded as x and the corpus length is recorded as seq.

[0011] 2) Construct a character-level dictionary D, record the size of the dictionary as n, perform one-hot encoding on each character in the text data according to its order in D, and record the encoding result as o∈R seq,n ;

[0012] 3) Send the one-hot encoding result to the pre-trained language model to obtain a high-level character representation containing semantic information, denoted as e∈R seq,d ;

[0013] Furthermore, the text error recognition module outputs the error probability of each character based on the text features through the following steps:

[0014] 1) Construct a network static computation graph;

[0015] 2) Model training: The extracted text features are input into the network static calculation graph, and the loss function and its derivatives to the learnable parameters of each layer are calculated according to the network output results and label information (whether each character has a spelling error), and the parameters of each layer are updated according to the calculated derivatives;

[0016] 3) Real-time evaluation and online deployment: After each parameter update, the performance measurement parameters of the model are solved on the validation set to obtain the final classifier and deploy it online.

[0017] According to the text error recognition module, for the network static calculation graph, the present invention further provides an implementation method based on a bidirectional long short-term memory module (Bi-LSTM), wherein the variables with superscript right arrows represent the network traversing text features from first to last, and the variables with superscript left arrows represent the network traversing text features from last to first. At each time step t in the sequence model, the input information e is first combined with the input information e. <t>< / t> and the hidden layer state h at the previous time step <t-1>< / t-1> , use the tanh function to obtain the candidate value of the current step memory cell state update value;

[0018]

[0019] Among them, h <t-1>< / t-1> is the hidden layer state at the previous time step, e <t>< / t>is the input information of the current step, W C is the linear transformation coefficient, b C is the bias term, and the superscript arrow represents the direction of network traversal. Next, define the gate unit: update gate Γ u , forget gate Γ f , output gate Γ o These gate units have similar definitions and all use the sigmoid function for output mapping.

[0020]

[0021]

[0022]

[0023] Where W u , W f , W o They are the linear transformation coefficients of the update gate, forget gate, and output gate, respectively, and b u , b f , b o are the bias terms of the update gate, forget gate, and output gate, respectively. The superscript arrows represent the direction of network traversal. σ(·) represents the sigmoid function, which controls the output value of the gate unit from 0 to 1, and is always very close to 0 or 1 in most cases. For the memory cell state c at the previous time step <t-1>< / t-1> , forget gate Γ f Decide whether to forget it; update the value of the memory cell state for the current step Update gate Γ u Decide whether to actually update it; then, update the cell state c <t>< / t> Through the output gate Γ o , we can get the current hidden layer state a <t>< / t> .

[0024]

[0025]

[0026] With the powerful modeling ability of the bidirectional sequence model for temporal dependence, the semantic features in the text data can be effectively extracted; the bidirectional features of each character are concatenated to obtain the final feature representation h <t>< / t> :

[0027]

[0028] Finally, the probability of error for a given character is estimated through a densely connected layer:

[0029] p <t>< / t> =P(error <t>< / t> =1|X)=σ(W e h <t>< / t> +b e )

[0030] where p <t>< / t> is the probability that the tth character contains an error, W e , b e is the linear discriminant coefficient, and σ is the sigmoid function.

[0031] According to the text error recognition module, for the model training described therein, the present invention provides a training method using a binary cross entropy (BCE) loss:

[0032]

[0033] in is the probability of error for a given character predicted by the model, is the actual label of a given character, m is the number of input corpus, and seq is the length of the input corpus.

[0034] Furthermore, the text error correction module outputs a correction representation of each character based on text features and error probability through the following steps:

[0035] 1) Construct a soft mask input sequence;

[0036] 2) Construct a network static computation graph;

[0037] 3) Model training: inputting the soft mask input sequence into the network static computation graph, calculating the derivative of the loss function with respect to the learnable parameters of each layer according to the network output result and the label information (embedded representation of the correct character), and updating the parameters of each layer according to the calculated derivative;

[0038] 4) Real-time evaluation and online deployment: After each parameter update, the performance measurement parameters of the model are calculated on the validation set to obtain the final classifier and deploy it online.

[0039] According to the text error correction module, for the soft mask input sequence, an arbitrary mask sequence is pre-given as Character error probability estimation sequence Construct the soft mask input sequence as:

[0040] e mask,i =p i *e′ i +(1-p i )*e i

[0041] Among them, emask,i is the soft mask of the generated i-th character, e i is the feature embedding of the i-th character output by the text feature extraction module. Masking is performed based on the error probability output by the text error recognition module. If the model infers that the probability of error for this character is high, the resulting soft mask input e mask,i Embedding e′ close to the mask i , otherwise it is close to the input embedding e i .

[0042] According to the text error correction module, the present invention further provides a Bert-based implementation method for the network static calculation graph. The BERT consists of 12 identical blocks. mask The sequence is taken as input. Each block contains a multi-head self-attention operation:

[0043] MultiHead(X)=Concat(head1;…, head h )W c

[0044] Among them, W c is the parameter to be learned, and each head is implemented by the following self-attention method:

[0045]

[0046] in, is the parameter to be learned for the jth attention head. This is followed by a feedforward network that transforms the aggregated information of each head again. Further, the hidden state sequence of the last layer of BERT is represented as t=1,2,...,seq, then for the tth character, the probability of error correction is defined as:

[0047]

[0048] in is the conditional probability that the kth character in the candidate list is modified as the tth character in the input sequence.

[0049] According to the text error recognition module, for the model training described herein, the training data consists of the original error-containing text sequence and the corresponding error-free text sequence. The method of creating such data is to randomly perturb the error-free text to generate different error-containing sequences, which are then input into the model for supervised calibration. The learning objectives are:

[0050]

[0051] Among them, m is the number of input corpus, seq is the length of input corpus, K is the size of candidate dictionary, is the probability that the tth character in the i-th input text can be corrected by the k-th item in the candidate dictionary, is the model prediction probability.

[0052] Compared with the prior art, the technical concept and benefits of the present invention are:

[0053] 1) Aiming at the characteristics of text error correction tasks, the present invention first uses a sequence language model to predict the error probability of each character, and then uses BERT to predict the probability of correcting the wrong characters in each candidate set. This method can make full use of the powerful learning ability of deep learning technology and the semantic extraction ability of the sequence model to quickly and intelligently extract the structural representation of the wrong characters. The whole process can be completed without human participation.

[0054] 2) Based on the deep learning network model, the present invention further makes full use of the industry's advanced pre-trained language model for fine-tuning, which can obtain better model performance and more reasonable training costs;

[0055] 3) The present invention can realize model training without labeled data, which can effectively reduce the labor cost of data annotation and has no high requirements for specific application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the implementation process of the intelligent error correction for spelling errors in Chinese corpus in this embodiment. DETAILED DESCRIPTION

[0057] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0058] like Figure 1 As shown, an intelligent error correction system for spelling errors in Chinese corpus is composed of three modules: a text feature extraction module 1, a text error recognition module 2, and a text error correction module 3.

[0059] 1) Text feature extraction module 1: Based on the pre-trained language model, obtain the dynamic encoding of the corpus;

[0060] 2) Text Error Recognition Module 2: Based on the sequence language model, predict the probability of an error for each character in the text;

[0061] 3) Text error correction module 3: Mask the error probability based on the output of the text error recognition module and use the language model to predict the correct characters.

[0062] Furthermore, the text feature extraction module 1 converts the text data into corpus dynamic coding through the following steps:

[0063] 1) Pad the end with zeros or truncate to fix the length of each corpus. The processed corpus is recorded as x and the corpus length is recorded as seq.

[0064] 2) Construct a character-level dictionary D, record the size of the dictionary as n, perform one-hot encoding on each character in the text data according to its order in D, and record the encoding result as o∈R seq,n ;

[0065] 3) Send the one-hot encoding result to the pre-trained language model to obtain a high-level character representation containing semantic information, denoted as e∈R seq,d ;

[0066] Furthermore, the text error recognition module 2 outputs the error probability of each character based on the text features through the following steps:

[0067] 1) Construct a network static computation graph;

[0068] 2) Model training: The extracted text features are input into the network static calculation graph, and the loss function and its derivatives to the learnable parameters of each layer are calculated according to the network output results and label information (whether each character has a spelling error), and the parameters of each layer are updated according to the calculated derivatives;

[0069] 3) Real-time evaluation and online deployment: After each parameter update, the performance measurement parameters of the model are solved on the validation set to obtain the final classifier and deploy it online.

[0070] According to the text error recognition module 2, for the network static calculation graph, the present invention further provides an implementation method based on a bidirectional long short-term memory module (Bi-LSTM), wherein the variables with superscript right arrows represent the network traversing text features from first to last, and the variables with superscript left arrows represent the network traversing text features from last to first. At each time step t in the sequence model, the input information e is first combined with the input information e. <t>< / t> and the hidden layer state h at the previous time step <t-1>< / t-1> , use the tanh function to obtain the candidate value of the current step memory cell state update value;

[0071]

[0072] Among them, h <t-1>< / t-1> is the hidden layer state at the previous time step, e <t>< / t> is the input information of the current step, W C is the linear transformation coefficient, b C is the bias term, and the superscript arrow represents the direction of network traversal. Next, define the gate unit: update gate Γu , forget gate Γ f , output gate Γ o These gate units have similar definitions and all use the sigmoid function for output mapping.

[0073]

[0074]

[0075]

[0076] Where W u , W f , W o They are the linear transformation coefficients of the update gate, forget gate, and output gate, respectively, and b u , b f , b o are the bias terms of the update gate, forget gate, and output gate, respectively. The superscript arrows represent the direction of network traversal. σ(·) represents the sigmoid function, which controls the output value of the gate unit from 0 to 1, and is always very close to 0 or 1 in most cases. For the memory cell state c at the previous time step <t-1>< / t-1> , forget gate Γ f Decide whether to forget it; update the value of the memory cell state for the current step Update gate Γ u Decide whether to actually update it; then, update the cell state c <t>< / t> Through the output gate Γ o , we can get the current hidden layer state a <t>< / t> .

[0077]

[0078]

[0079] With the powerful modeling ability of the bidirectional sequence model for temporal dependence, the semantic features in the text data can be effectively extracted; the bidirectional features of each character are concatenated to obtain the final feature representation h <t>< / t> :

[0080]

[0081] Finally, the probability of error for a given character is estimated through a densely connected layer:

[0082]

[0083] where p <t>< / t> is the probability that the tth character contains an error, W e , b eis the linear discriminant coefficient, and σ is the sigmoid function.

[0084] According to the text error recognition module 2, for the model training described therein, the present invention provides a training method based on binary cross entropy (BCE) loss:

[0085]

[0086] in is the probability of error for a given character predicted by the model, is the actual label of a given character, m is the number of input corpus, and seq is the length of the input corpus.

[0087] Furthermore, the text error correction module 3 outputs a correction representation of each character based on text features and error probability through the following steps:

[0088] 1) Construct a soft mask input sequence;

[0089] 2) Construct a network static computation graph;

[0090] 3) Model training: inputting the soft mask input sequence into the network static computation graph, calculating the derivative of the loss function with respect to the learnable parameters of each layer according to the network output result and the label information (embedded representation of the correct character), and updating the parameters of each layer according to the calculated derivative;

[0091] 4) Real-time evaluation and online deployment: After each parameter update, the performance measurement parameters of the model are calculated on the validation set to obtain the final classifier and deploy it online.

[0092] According to the text error correction module 3, for the soft mask input sequence, an arbitrary mask sequence is pre-given as Character error probability estimation sequence Construct the soft mask input sequence as:

[0093] e mask,i =p i *e′ i +(1-p i )*e i

[0094] Among them, e mask,i is the soft mask of the generated i-th character, e i is the feature embedding of the i-th character output by the text feature extraction module. Masking is performed based on the error probability output by the text error recognition module 2. If the model infers that the probability of error for this character is high, the soft mask input e mask,i Embedding e′ close to the mask i, otherwise it is close to the input embedding e i .

[0095] According to the text error correction module 3, the present invention further provides a Bert-based implementation method for the network static calculation graph. The BERT consists of 12 identical blocks. mask The sequence is taken as input. Each block contains a multi-head self-attention operation:

[0096] MultiHead(X)=Concat(head1;…, head h )W c

[0097] Among them, W c is the parameter to be learned, and each head is implemented by the following self-attention method:

[0098]

[0099] in, is the parameter to be learned for the jth attention head. This is followed by a feedforward network that transforms the aggregated information of each head again. Further, the hidden state sequence of the last layer of BERT is represented as Then for the tth character, the probability of error correction is defined as:

[0100]

[0101] in is the conditional probability that the kth character in the candidate list is modified as the tth character in the input sequence.

[0102] According to the text error correction module 3, for the model training, the training data consists of the original error-containing text sequence and the corresponding error-free text sequence. The method of creating such data is to randomly perturb the error-free text to generate different error-containing sequences, which are then input into the model for supervised calibration. The learning objectives are:

[0103]

[0104] Among them, m is the number of input corpus, seq is the length of input corpus, K is the size of candidate dictionary, is the probability that the tth character in the i-th input text can be corrected by the k-th item in the candidate dictionary, is the model prediction probability.

[0105] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. An intelligent error correction system for Chinese corpus spelling errors, consisting of three modules: a text feature extraction module, a text error recognition module, and a text error correction module; wherein: The text feature extraction module obtains the dynamic encoding of the corpus based on the pre-trained language model; the text error recognition module predicts the probability of an error in each character in the text based on the sequence language model; the text error correction module performs masking based on the error probability output by the text error recognition module, and uses the language model to predict the correct character; The text feature extraction module converts text data into corpus dynamic encoding through the following steps: 1) Pad the end with zeros or truncate to fix the length of each corpus. The processed corpus is recorded as x and the corpus length is recorded as seq. 2) Construct a character-level dictionary D, record the size of the dictionary as n, perform one-hot encoding on each character in the text data according to its order in D, and record the encoding result as o∈R seq,n ; 3) Send the one-hot encoding result to the pre-trained language model to obtain a high-level character representation containing semantic information, denoted as e∈R seq,d ; The text error recognition module outputs the error probability of each character based on the text features through the following steps: 1) Construct a network static computation graph; 2) Model training: The extracted text features are input into the network static calculation graph, the loss function and its derivatives to the learnable parameters of each layer are calculated according to the network output results and label information, and the parameters of each layer are updated according to the calculated derivatives; 3) Real-time evaluation and online deployment: After each parameter update, the performance measurement parameters of the model are calculated on the validation set to obtain the final classifier and deploy it online; For the constructed network static calculation graph, the variables with right arrows represent the network traversing the text features from the beginning to the end, and the variables with left arrows represent the network traversing the text features from the end to the beginning at each time step t in the sequence model, first combining the input information e <t>< / t> and the hidden layer state h at the previous time step <t-1>< / t-1> , use the tanh function to obtain the candidate value of the current step memory cell state update value; Among them, h <t-1>< / t-1> is the hidden layer state at the previous time step, e <t>< / t> is the input information of the current step, W C is the linear transformation coefficient, b C is the bias term, and the superscript arrow represents the direction of network traversal. Next, define the gate unit: update gate Γ u , forget gate Γ f , output gate Γ o These gate units have similar definitions and all use the sigmoid function for output mapping. Where W u , W f , W o They are the linear transformation coefficients of the update gate, forget gate, and output gate, respectively, and b u , b f , b o They are the bias terms of the update gate, forget gate, and output gate, respectively. The superscript arrow represents the direction of network traversal. σ(·) represents the sigmoid function, which controls the output value of the gate unit from 0 to 1. In most cases, it is always very close to 0 or 1. For the memory cell state c at the previous time step ＜t-1＞ , forget gate Γ f Decide whether to forget it; update the value of the memory cell state for the current step Update gate Γ u Decide whether to actually update it; then, update the cell state c ＜t＞ Through the output gate Γ o , we can get the current hidden layer state a ＜t＞ Concatenate the bidirectional features of each character to get the final feature representation h <t>< / t> : Finally, the probability of error for a given character is estimated through a densely connected layer: p <t>< / t> =σ(W e h <t>< / t> +b e ) Among them, p <t>< / t> is the probability that the tth character contains an error, W e , b e is the linear discriminant coefficient, σ is the sigmoid function; For the model training, a training method based on binary cross entropy loss is given: in, is the probability of error for a given character predicted by the model, is whether the given character actually contains an incorrect label, m is the number of input corpus, and seq is the length of the input corpus; The text error correction module outputs a correction representation of each character based on text features and error probability through the following steps: 1) Construct a soft mask input sequence; 2) Construct a network static computation graph; 3) Model training: inputting the soft mask input sequence into the network static computation graph, calculating the derivative of the loss function with respect to the learnable parameters of each layer according to the network output result and label information, and updating the parameters of each layer according to the calculated derivative; 4) Real-time evaluation and online deployment: After each parameter update, the performance measurement parameters of the model are calculated on the validation set to obtain the final classifier and deploy it online; For the soft mask input sequence described therein, the soft mask input sequence is constructed as: And mask,i =p i *And i ′+(1-p i )*And i in, is a pre-given arbitrary mask sequence, is the estimated probability sequence of characters containing errors, e mask,i is the soft mask of the generated i-th character, e i The feature embedding of the i-th character output by the text feature extraction module is masked based on the error probability output by the text error recognition module. If the model infers that the error probability of the character is high, the soft mask e mask,i Embedding close to the mask i ′, otherwise it is close to the input embedding e i ; For the network static calculation graph, the present invention provides an implementation method based on Bert. The BERT consists of 12 identical blocks. mask The sequence is taken as input and each block contains a multi-head self-attention operation: MultiHead(X)=Concat(head1;…, head h )W c Among them, W c is the parameter to be learned, and each head is implemented by the following self-attention method: in, is the parameter to be learned of the jth attention head, followed by a feedforward network that transforms the aggregated information of each head again; the hidden state sequence of the last layer of the BERT is represented as Then for the tth character, the probability of error correction is defined as: in The conditional probability that the kth character in the candidate list is used as the tth character in the input sequence for correction; For the model training described therein, the training data consists of the original error-containing text sequence and the corresponding error-free text sequence. The method of creating such data is to randomly perturb the error-free text to generate different error-containing sequences, which are then input into the model for supervised calibration. The learning objective is: Among them, m is the number of input corpus, seq is the length of input corpus, K is the size of candidate dictionary, is the probability that the tth character in the i-th input text can be corrected by the k-th item in the candidate dictionary, is the model prediction probability.

Images