A Chinese spelling error correction method based on maximum entropy principle

Abstract

The invention discloses an interpolation Chinese spelling error correction method based on the maximum entropy principle, and relates to the technical field of natural language processing; S1, given an input text sequence and a training label; S2, passing the input text sequence of S1 into a PLMs model or a CSC model to obtain a model output result and calculate a loss function; S3, selecting a penalty target through PPMEP according to the model output result of S2; S4, calculating an entropy regularization term for penalty by applying the Shannon formula according to the input text sequence, the model output result and the penalty target of S3; S5, fusing the entropy regularization term of S4 and the loss function of S2 and optimizing them according to the maximum entropy principle to obtain a new training target; S6, weighting the new training target in S5 according to a dynamic decreasing rule; the invention adopts the interpolation Chinese spelling error correction method, which can be seamlessly integrated into an existing model, and optimizes the model classification probability representation by using the maximum entropy principle, so that the correction selection of the model is more in line with the Chinese spelling error correction context.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to an interpolation-type Chinese spelling error correction method based on the maximum entropy principle. Background Art

[0002] CSC is a basic natural language processing task that has received extensive attention in the past few decades. Early CSC methods detected and corrected spelling errors in input based on rules such as chunks, semantics, and grammar. Such methods have high correction accuracy but poor generalization performance. Because rules require experts to spend a lot of time and formulate them based on knowledge and experience, they can only cover a limited number of cases. Later, some studies tried to use statistical language models to complete the CSC task. First, the characters in the input sentence are replaced according to the confusion set, and then the statistical language model is used to score these replaced sentences, and correction suggestions are given based on the scores. However, due to the poor expression and learning ability of statistical language models, such methods cannot perform CSC tasks well.

[0003] Since then, most methods have been developed in a way that integrates the external knowledge features of Chinese character pronunciation based on PLMs. For example, in 2020, a method designed a two-stage error detection-correction model, in which the detection network is based on Bi-GRU and the error correction network is based on the pre-trained language model BERT. In the first stage, the detection network of the model predicts the error probability of each character and passes the probability to the error correction network in the second stage through a soft masking mechanism to produce an error correction result. However, the strategy of masking erroneous characters may result in the loss of the phonetic and morphological information of the characters, so that the error correction is over-reliant on the semantic understanding ability of the error correction network. In 2021, a method proposed a new dynamic connection network DCN, but this method ignored a major essential feature of the CSC task and failed to make good use of the visual similarity features of characters. In 2023, a method introduced a pinyin-to-character prediction task, requiring the model to predict the correct character based only on speech information. In order to avoid overfitting of speech learning, a self-distillation module was designed, but the model performs two forward passes on each sample in the self-distillation module, which will incur more computational costs and increase the training burden. Summary of the invention

[0004] The purpose of the present invention is to provide a plug-in Chinese spelling correction method based on the maximum entropy principle, which can be seamlessly integrated into the existing model and optimize the model classification probability representation using the maximum entropy principle, so that the correction selection of the model is more in line with the Chinese spelling correction context.

[0005] To achieve the above object, the present invention provides an interpolation Chinese spelling error correction method based on the maximum entropy principle, comprising the following steps:

[0006] S1. The user enters the text that needs to be corrected and obtains an input text sequence of length n based on the open source dataset. and training labels of equal length ;

[0007] S2, input text sequence described in S1 Input the PLMs model or CSC model to obtain the model output results and calculate the loss function;

[0008] S3, select the penalty target through PPMEP according to the model output results of S2;

[0009] S4, according to the input text sequence, the model output results and the penalty target of S3, the entropy regularization term used for penalty is calculated by applying the Shannon formula;

[0010] S5, merge the entropy regularization term of S4 and the loss function of S2 and optimize according to the maximum entropy principle to obtain a new training target;

[0011] S6, iterate the new training target in S5 according to the dynamic decreasing rule, adjust the weight, and obtain accurate correction decision;

[0012] S7. Based on the correction decision obtained in S6, the text that needs to be corrected that is input by the user in S1 is corrected to obtain a final correction result.

[0013] Preferably, the specific steps of S2 are as follows:

[0014] S21, input text sequence in S1 Input PLMs model or CSC model to get output representation , the calculation process is as follows:

[0015] (1);

[0016] in, Is PLMs or existing CSC model, represents the feature representation of the last hidden layer of the model, i is 1,2,...n; , Represents the space where the features of the last hidden layer are located, yes The dimension of the last hidden layer;

[0017] S22, classify the output representation obtained in S21 through the classification layer to obtain the classification representation. The calculation process of the classification layer is as follows:

[0018] (2);

[0019] in, For classification, represents the classifier, Represented as the weight matrix used for linear transformation in the classifier, , Expressed as the dimension of the weight matrix, is represented as a bias vector, , is represented as the dimension of the bias vector, Expressed as the model vocabulary size;

[0020] S23, the classification obtained in S22 is expressed by The output of the function represents the probability of selecting the corresponding character in the vocabulary at each position. The calculation process is as follows:

[0021] (3);

[0022] in, Expressed as the predicted probability of the final character, Representative The probability of the character at position 1 taking the corresponding character in the vocabulary is, , Represented as the subscript of the corresponding character in the model vocabulary;

[0023] S24, select the character with the largest classification probability at each position as the correction result of the position to obtain the model output result, and the calculation process is as follows:

[0024] (4);

[0025] in, Represented as the model output result, It is expressed as a function to obtain the subscript of the maximum probability of the corresponding character. is the final predicted character;

[0026] S25, calculate the loss function according to the probability of the corresponding character in S23 , the calculation formula is as follows:

[0027] (5).

[0028] Preferably, the process of S3 selecting a penalty target through PPMEP is as follows:

[0029] S31. Compare the output results of the model during the training phase and training labels The elements at the corresponding positions get the model's response to the input sequence Correct the wrong prediction set F of the wrong elements;

[0030] S32. Compare model output sequences and training labels The elements at the corresponding positions get the model's response to the input sequence The set of correctly predicted elements ;

[0031] S33, from the collection Randomly select Proportional character composition set , as the set of correct character positions that need to be penalized;

[0032] S34, wrong prediction set F and randomly selected correct prediction set The corresponding character in the picture is the target that needs to be punished; The two sets are combined to obtain the final target set A that needs to be punished.

[0033] Preferably, the model in S31 is used for the input sequence The expression of the error prediction set F for correcting the erroneous elements is as follows:

[0034] (6);

[0035] in, Indicates the character to correct the error, k is 1, 2, ... k, is represented as a set of positive integers, represents the correct character in the training label, and k represents the number of incorrectly predicted characters in the sequence.

[0036] Preferably, the model in S32 is used for the input sequence The set of correctly predicted elements The expression is as follows:

[0037] (7);

[0038] in, Represented as all correctly predicted characters, is 1,2,... , Indicates the number of characters in the sequence that are correctly predicted;

[0039] Preferably, the expression of the correct character position set that needs to be penalized in S33 is as follows:

[0040] (8);

[0041] in, Represents the correctly predicted characters randomly selected, is 1,2,... , represents the number of correctly predicted characters randomly selected, , is a predefined hyperparameter.

[0042] Preferably, the expression of the final target set A that needs to be punished obtained in S34 is as follows:

[0043] (9);

[0044] in, is the character needed to construct the maximum entropy loss, is 1,2,... , , Indicates the total number of penalty characters.

[0045] Preferably, the expression for calculating the entropy regularization term for penalty in S4 is as follows:

[0046] (10);

[0047] in, Made of low entropy.

[0048] Preferably, in S5, the entropy regularization term of S4 and the loss function of S2 are integrated and optimized according to the maximum entropy principle to obtain a new training objective formula as follows:

[0049] (11);

[0050] in is a predefined anti-zero coefficient; in the fusion and When there are two loss values, according to the original loss function of the model used Choose different fusion methods according to the characteristics of For large models, direct subtraction is used as a new training target; Small model, yes Take the inverse and merge again, Represents the weight parameter of the current training step.

[0051] Preferably, the weight process in S6 follows the following dynamic decreasing rule, and the calculation expression is as follows:

[0052] (12);

[0053] Among them, step represents the number of times the model has updated the parameters. represents the initial weight.

[0054] Therefore, the present invention adopts the above-mentioned structure of an interpolation Chinese spelling error correction method based on the maximum entropy principle, which has the following beneficial effects compared with the prior art:

[0055] 1. The performance of Chinese spelling correction has been improved, which can be seamlessly integrated into the existing model. The maximum entropy principle is used to optimize the model classification probability representation, making the model's correction selection more suitable for the Chinese spelling correction context;

[0056] 2. It effectively narrows the gap between the pre-trained language model and the Chinese spelling correction task. This framework can effectively avoid overconfidence in misclassification and improve the robustness of the model without increasing the complexity of the model.

[0057] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of an embodiment of an interpolation-type Chinese spelling error correction method based on the maximum entropy principle of the present invention;

[0059] Figure 2 It is a framework diagram of an embodiment of an insertion-type Chinese spelling error correction method based on the maximum entropy principle of the present invention. DETAILED DESCRIPTION

[0060] Example

[0061] In describing the present invention, it should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not require further definition and explanation in the subsequent drawings.

[0062] like Figure 1-Figure 2 As shown, a plug-in Chinese spelling error correction method based on the maximum entropy principle of the present invention comprises the following steps:

[0063] S1. The user enters the text that needs to be corrected and obtains an input text sequence of length n based on the open source dataset. and training labels of equal length ;

[0064] S2, input text sequence described in S1 Input the PLMs model or CSC model to obtain the model output results and calculate the loss function;

[0065] The specific steps of S2 are as follows:

[0066] S21, input text sequence in S1 Input PLMs model or CSC model to get output representation , the calculation process is as follows:

[0067] (1);

[0068] in, Is PLMs or existing CSC model, represents the feature representation of the last hidden layer of the model, i is 1,2,...n; , Represents the space where the features of the last hidden layer are located, yes The dimension of the last hidden layer;

[0069] S22, classify the output representation obtained in S21 through the classification layer to obtain the classification representation. The calculation process of the classification layer is as follows:

[0070] (2);

[0071] in, For classification, represents the classifier, Represented as the weight matrix used for linear transformation in the classifier, , Expressed as the dimension of the weight matrix, is represented as a bias vector, , is represented as the dimension of the bias vector, Expressed as the model vocabulary size;

[0072] S23, the classification obtained in S22 is expressed by The output of the function represents the probability of selecting the corresponding character in the vocabulary at each position. The calculation process is as follows:

[0073] (3);

[0074] in, Expressed as the predicted probability of the final character, Representative The probability of the character at position 1 taking the corresponding character in the vocabulary is, , Represented as the subscript of the corresponding character in the model vocabulary;

[0075] S24, select the character with the largest classification probability at each position as the correction result of the position to obtain the model output result, and the calculation process is as follows:

[0076] (4);

[0077] in, Represented as the model output result, It is expressed as a function to obtain the subscript of the maximum probability of the corresponding character. is the final predicted character;

[0078] S25, calculate the loss function according to the probability of the corresponding character in S23 , the calculation formula is as follows:

[0079] (5);

[0080] S3, select the penalty target through PPMEP according to the model output results of S2;

[0081] The main target of PPMEP penalty is the characters that the model predicts incorrectly. Model prediction errors are usually due to the high probability of the wrong characters appearing, so it is used as a penalty target to reduce the probability of wrong predictions.

[0082] S31. Compare the output results of the model during the training phase and training labels The elements at the corresponding positions get the model's response to the input sequence The expression of the error prediction set F for correcting the erroneous elements is as follows:

[0083] (6);

[0084] in, Indicates the character to correct the error, k is 1, 2, ... k, is represented as a set of positive integers, represents the correct character in the training label, and k represents the number of incorrectly predicted characters in the sequence;

[0085] In addition to penalizing incorrectly predicted characters, PPMEP also needs to penalize a certain percentage of correctly predicted characters to improve the versatility and robustness of the model;

[0086] S32. Compare model output sequences and training labels The elements at the corresponding positions get the model's response to the input sequence The set of correctly predicted elements The expression is as follows:

[0087] (7);

[0088] in, Represented as all correctly predicted characters, is 1,2,... , Indicates the number of characters in the sequence that are correctly predicted;

[0089] S33, from the collection Randomly select Proportional character composition set , as the set of correct character positions that need to be penalized The expression is as follows:

[0090] (8);

[0091] in, Represents the correctly predicted characters randomly selected, is 1,2,... , represents the number of correctly predicted characters randomly selected, , is a predefined hyperparameter.

[0092] S34, wrong prediction set F and randomly selected correct prediction set The corresponding character in the picture is the target that needs to be punished; The expression of the final target set A that needs to be punished by merging the two sets is as follows:

[0093] (9);

[0094] in, is the character needed to construct the maximum entropy loss, is 1,2,... , , Indicates the total number of characters that need to be penalized;

[0095] S4. Based on the input text sequence, model output results and the penalty target of S3, the Shannon formula is used to calculate the entropy regularization term used for penalty. The expression is as follows:

[0096] (10);

[0097] in, It is composed of low entropy, that is, the probability of the selected penalty target is much higher than that of other characters. The more uneven the probability distribution is, the lower the entropy value calculated according to the Shannon formula is. The model makes wrong correction choices, often because the character with the highest classification probability is wrong. The wrong character has the highest probability and the probability gap with other characters is large, which will lead to overconfidence in the error. From the Shannon formula, we can see that entropy is a measure of system uncertainty. When a character has a high probability of appearing at a certain position in a sentence, its entropy value will be lower. According to the maximum entropy principle, when learning a probability model, the model with the largest entropy among all possible models should be selected as the optimal model, so that the model prediction risk is the smallest, and the maximum entropy model is No subjective assumptions are made about unknown situations during construction, so there will be no label bias, that is, the model performs well on training data and can be well generalized to unknown data; therefore, the penalty low entropy strategy in this article is mainly based on the maximum entropy principle, so that the model can be optimized in the direction of maximum entropy; during the training process, in order to avoid overconfidence in wrong predictions, the entropy of the probability distribution can be increased by penalizing wrong low entropy; in the process of increasing entropy, the probabilities of each classification are close to each other, at this time the high probability of wrong predictions is reduced, and the low probability of correct predictions is increased; the possibility of the model selecting the correct result is greatly improved, avoiding overconfidence in wrong predictions; in addition, by penalizing low entropy to reduce the probability of wrong predictions, the model fitting process can also be accelerated.

[0098] S5, the entropy regularization term of S4 and the loss function of S2 are combined and optimized according to the maximum entropy principle to obtain a new training target:

[0099] (11);

[0100] in is a predefined anti-zero factor to prevent When it is 0, the fusion loss function L fails; and When there are two loss values, according to the original loss function of the model used Choose different fusion methods according to the characteristics of For large models, direct subtraction is used as a new training target; Small models, avoid subtracting Now the loss function becomes negative, Take the inverse and merge again, Represents the weight parameter of the current training step; penalizing low entropy is to use it as a penalty source. The purpose of penalizing low entropy is to increase entropy, and the training process of the model is to continuously minimize the loss function process, so when the entropy regularization term With loss function When merging, you need to take its inverse or negative value to maximize entropy while minimizing loss;

[0101] S6, iterate the new training target in S5 according to the dynamic decreasing rule, adjust the weight, and obtain accurate correction decision. The weight process follows the calculation expression of the following dynamic decreasing rule as follows:

[0102] (12);

[0103] in, Represents the weight parameter of the current training step. The value of this parameter gradually decreases with the increase of the number of training rounds. It is used to control The proportion of total loss, step indicates the number of times the model has updated the parameters, Represents the initial weight, the early Larger This has a greater impact on the model, which can quickly reduce the probability of incorrectly predicting characters and speed up the model fitting process. The value of gradually decreases, The impact of will also be reduced, avoiding the problem of weakening the model's own error correction ability due to excessive equalization, and also preventing excessive smoothing of differences. Therefore, the penalty low entropy strategy runs through the entire model training process, and guides the model to make more accurate correction decisions by optimizing the classification probability representation of the model.

[0104] S7. Based on the correction decision obtained in S6, the text that needs to be corrected that is input by the user in S1 is corrected to obtain a final correction result.

[0105] In the specific implementation process, the plug-in Chinese spelling error correction framework (PPMEP) based on the maximum entropy principle first selects the errors generated by the model itself as the optimization source, and then reduces the probability of the model being overconfident in wrong predictions through the maximum entropy optimization principle, and relatively increases the probability of predicting the correct answer. Through this error-driven approach, the classification probability of the model can be optimized and the performance of the model on the CSC task can be improved. PPMEP is divided into two stages: penalty target selection and penalty low entropy. Among them, the penalty target selection is based on the correction output results of the model, and the predicted wrong characters and the randomly quantified predicted correct characters are selected as the penalty source; in the penalty low entropy stage, a maximum entropy loss is constructed according to the penalty target and combined with the original loss of the model, and then back-propagated to optimize the classification probability of the model and help the model make better correction choices.

[0106] The following are the detection and error correction levels of different models in three different test sets: SIGHAN13, SIGHAN14 and SIGHAN15. The “↑” in the table indicates that the performance of the corresponding baseline method has been further improved after PPMEP optimization:

[0107] Table 1 Detection and correction level indicators of three test sets in different models

[0108]

[0109] Therefore, the present invention adopts the above-mentioned structure and an insertion-type Chinese spelling correction method based on the maximum entropy principle, and proposes an insertion-type Chinese spelling correction method based on the maximum entropy principle. By seamlessly integrating it into a pre-trained language model or an existing Chinese spelling correction model, the maximum entropy optimization principle is used to reduce the probability of the model being overconfident in incorrect predictions, weaken the strong semantic choices provided by the model, and increase the probability of predicting the correct answer, so that the model can select the correct characters that fit the CSC context.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A plug-in Chinese spelling error correction method based on the maximum entropy principle, characterized in that: The following steps are involved: S1. The user enters the text that needs to be corrected and obtains an input text sequence of length n based on the open source dataset. and training labels of equal length ; S2, input text sequence described in S1 Input the PLMs model or CSC model to obtain the model output results and calculate the loss function; S3, select the penalty target through PPMEP according to the model output results of S2; S4, according to the input text sequence, the model output results and the penalty target of S3, the entropy regularization term used for penalty is calculated by applying the Shannon formula; S5, merge the entropy regularization term of S4 and the loss function of S2 and optimize according to the maximum entropy principle to obtain a new training target; S6, iterate the new training target in S5 according to the dynamic decreasing rule, adjust the weight, and obtain accurate correction decision; S7, based on the correction decision obtained in S6, correct the text that needs to be corrected that is input by the user in S1, and obtain a final correction result; The process of S3 selecting penalty targets through PPMEP is as follows: S31. Compare the output results of the model during the training phase and training labels The elements at the corresponding positions get the model's response to the input sequence Correct the wrong prediction set F of the wrong elements; S32. Compare model output sequences and training labels The elements at the corresponding positions get the model's response to the input sequence The set of correctly predicted elements ; S33, from the collection Randomly select Proportional character composition set , as the set of correct character positions that need to be penalized; S34, wrong prediction set F and randomly selected correct prediction set The corresponding character in the picture is the target that needs to be punished; The two sets are combined to obtain the final target set A that needs to be punished.

2. The plug-in Chinese spelling error correction method based on the maximum entropy principle according to claim 1, characterized in that: The specific steps of S2 are as follows: S21, input text sequence in S1 Input PLMs model or CSC model to get output representation , the calculation process is as follows: （1）； in, Is PLMs or existing CSC model, represents the feature representation of the last hidden layer of the model, is 1,2,...n; , Represents the space where the features of the last hidden layer are located, yes The dimension of the last hidden layer; S22, classify the output representation obtained in S21 through the classification layer to obtain the classification representation. The calculation process of the classification layer is as follows: （2）； in, For classification, represents the classifier, Represented as the weight matrix used for linear transformation in the classifier, , Expressed as the dimension of the weight matrix, is represented as a bias vector, , is represented as the dimension of the bias vector, Expressed as the model vocabulary size; S23, the classification obtained in S22 is expressed by The output of the function represents the probability of selecting the corresponding character in the vocabulary at each position. The calculation process is as follows: （3）； in, Expressed as the predicted probability of the final character, Representative The probability of the character at position 1 taking the corresponding character in the vocabulary is, , Represented as the subscript of the corresponding character in the model vocabulary; S24, select the character with the largest classification probability at each position as the correction result of the position to obtain the model output result, and the calculation process is as follows: （4）； in, Represented as the model output result, It is expressed as a function to obtain the subscript of the maximum probability of the corresponding character. is the final predicted character; S25, calculate the loss function according to the probability of the corresponding character in S23 , the calculation formula is as follows: （5）。 3. The plug-in Chinese spelling error correction method based on the maximum entropy principle according to claim 2, characterized in that: In S31, the model takes the input sequence The expression of the error prediction set F for correcting the erroneous elements is as follows: （6）； in, Indicates the character to correct the error, k is 1, 2, ... k, is represented as a set of positive integers, represents the correct character in the training label, and k represents the number of incorrectly predicted characters in the sequence.

4. The method for correcting Chinese spelling errors based on the maximum entropy principle according to claim 3, characterized in that: The model in S32 takes an input sequence The set of correctly predicted elements The expression is as follows: （7）； in, Represented as all correctly predicted characters, is 1,2,... , Indicates the number of characters in the sequence that were correctly predicted.

5. The plug-in Chinese spelling error correction method based on the maximum entropy principle according to claim 4, characterized in that: The expression of the correct character position set that needs to be penalized in S33 is as follows: （8）； in, Represents the correctly predicted characters randomly selected, is 1,2,... , represents the number of correctly predicted characters randomly selected, , is a predefined hyperparameter.

6. The method of inserting Chinese spelling error correction based on the maximum entropy principle according to claim 5, characterized in that: The expression of the final target set A that needs to be punished obtained in S34 is as follows: （9）； in, is the character needed to construct the maximum entropy loss, is 1,2,... , , Indicates the total number of penalty characters.

7. The plug-in Chinese spelling error correction method based on the maximum entropy principle according to claim 6, characterized in that: The expression for calculating the entropy regularization term used for penalty in S4 is as follows: （10）； in, Made of low entropy.

8. The plug-in Chinese spelling error correction method based on the maximum entropy principle according to claim 7, characterized in that: In S5, the entropy regularization term of S4 and the loss function of S2 are integrated and optimized according to the maximum entropy principle to obtain the new training target formula as follows: （11）； in is a predefined anti-zero coefficient; in the fusion and When there are two loss values, according to the original loss function of the model used Choose different fusion methods according to the characteristics of For large models, direct subtraction is used as a new training target; Small model, yes Take the inverse and merge again, Represents the weight parameter of the current training step.

9. The plug-in Chinese spelling error correction method based on the maximum entropy principle according to claim 8, characterized in that: The weight process in S6 follows the following dynamic decreasing rule, and the calculation expression is as follows: (12)； Among them, step represents the number of times the model has updated the parameters. represents the initial weight.

Images