Chinese spelling error correction method and device based on entity information enhancement, and computer readable storage medium

Abstract

The invention provides a Chinese spelling error correction method and device based on entity information enhancement and a computer readable storage medium. According to the method, a public entity dictionary is collected and further expanded; a Chinese spelling corpus focusing on entities is constructed by using the expanded entity dictionary and the collected unlabeled corpus, and entity information in the corpus is hidden by using three different mask strategies; the built corpus is used for conducting fine adjustment on a pre-training language model to enhance attention of the model to shallow entity information, entity representation in the model is further enhanced, and therefore the model is more suitable for a Chinese spelling error correction scene; according to the method, entity information in different models is fused through a model integration technology, so that the performance of the models in a Chinese spelling error correction task is further improved. According to the method, while the Chinese spelling error correction efficiency is kept, the attention on entity information is remarkably improved, and entity errors in wrong sentences are effectively corrected, so that the effectiveness of the method is verified.

Description

[0001] Technical Language

[0002] The present invention relates to a Chinese spelling error correction method, device and computer-readable storage medium based on entity information enhancement, and belongs to the technical field of natural language processing. Background Art

[0003] With the rapid development of Internet technology and the widespread popularity of mobile devices, the generation and processing of Chinese texts have become increasingly frequent. In this era of information explosion, people are increasingly relying on the Internet and mobile devices for communication, learning and work, which undoubtedly increases the demand for the generation and processing of Chinese texts. Due to the large number of Chinese characters and the flexibility of the pinyin input method, spelling errors are still a common problem in Chinese texts. Spelling errors not only affect the readability of the text, but may also lead to misunderstandings and wrong decisions. In academic research, news reporting, business communication and other occasions, accurate text expression is crucial. Spelling errors may change the meaning of the original text, cause readers to have a biased understanding of the information, and may even affect decision-making. Therefore, spelling error correction technology is of great significance in Chinese text processing.

[0004] Traditional Chinese spelling error correction methods are mainly based on rules and dictionary matching. Rule-based methods detect and correct errors by formulating a series of spelling rules, while dictionary matching methods correct errors by looking up the correct spelling in the dictionary. However, these methods often rely on manually formulated rules and the size of the dictionary, making it difficult to cover all spelling errors. To address these problems, researchers have begun to use deep learning technology and have achieved good results. In these deep learning-based methods, researchers usually convert Chinese text into a sequence of characters and then use a neural network model to model the character sequence. The model learns the dependencies between characters, so that it can identify spelling errors and generate correct spellings. In addition, some methods also combine language models to evaluate the correctness of character sequences by calculating the probability of the sequence, and select the sequence with the highest probability as the correct spelling.

[0005] In Chinese text, entity information plays a vital role, usually involving nouns or noun phrases with specific meanings, such as names of people, places, and organizations. In the task of Chinese spelling correction, ensuring the correctness of entities is crucial to the understanding and expression of the entire sentence. However, existing Chinese spelling correction methods have not fully integrated the entity information contained in the corpus, which leads to performance deficiencies in entity correction. Therefore, how to efficiently use entity information to improve the performance of Chinese spelling correction methods has become a hot topic in current research. Summary of the invention

[0006] The technical problem to be solved by the present invention is: the present invention provides a Chinese spelling correction method, device, and computer-readable storage medium based on entity information enhancement, which solves the problem that the existing spelling correction method has not fully integrated the entity information contained in the corpus, resulting in insufficient performance in entity correction. The present invention proposes a "entity information hiding-entity information recovery-entity information enhancement-entity information fusion" hierarchical structure, which can effectively make the pre-trained language model pay more attention to entity information, and efficiently use entity information to improve the performance of the Chinese spelling correction method.

[0007] The technical solution of the present invention is: In the first aspect, the present invention provides a Chinese spelling error correction method based on entity information enhancement, and the specific steps of the method are as follows:

[0008] Step 1: Collect publicly available entity dictionaries and expand their coverage. Then collect unlabeled corpus and perform data filtering. Finally, build a Chinese spelling corpus focused on entities through manual annotation and named entity recognition models.

[0009] Step 2: In the entity information hiding module, three different masking strategies are used to hide the entity information in the entity-focused Chinese spelling corpus;

[0010] Step 3: Pass the corpus after hiding the entity information to the entity information recovery module, and fine-tune the initial pre-trained language model to make it focus on the shallow entity information;

[0011] Step 4: Transfer the fine-tuned pre-trained language model to the entity information enhancement module, use the public Chinese grammar correction corpus to further fine-tune the entity information recovery model, and further enhance the entity information in the model;

[0012] Step 5: Pass the results of different pre-trained language models to the entity information fusion module. This method fuses the entity information by comparing the entity information between different models to further improve the performance of the model.

[0013] As a further solution of the present invention, step 1 includes the following specific steps:

[0014] Step 1.1, Entity dictionary construction: Collect publicly available entity dictionaries and structured entities on Wikipedia; then, use named entity recognition models to identify entities on news and government websites. Finally, aggregate entities from different sources and filter out duplicate entities to form the final entity dictionary;

[0015] Step 1.2, unlabeled corpus acquisition and filtering: Obtain unlabeled corpus from different news and government websites, remove meaningless symbols and limit the length of sentences. Finally, use upsampling technology to balance the distribution of spelling errors in the corpus;

[0016] Step 1.3, Constructing the entity-focused Chinese spelling corpus: Segment the cleaned corpus and use the named entity model to identify entities in the sentence, and use the confusion set to replace the characters in the entity to form a corpus containing entity errors. Finally, use the double annotation and expert review strategy to annotate the validation set and test set of the entity-focused Chinese spelling corpus, referred to as EFCEC;

[0017] As a further solution of the present invention, step 2 includes the following specific steps:

[0018] Step 2.1, entity location acquisition: Use the entity recognition model to identify the location of the entity in the Chinese spelling correction corpus focusing on the entity;

[0019] Step 2.2, entity information hiding: design three different masking strategies (static masking, full word masking and N-gram masking) to hide entity information in entities;

[0020] As a further solution of the present invention, step 3 includes the following specific steps:

[0021] Step 3.1, pre-trained language model acquisition: use Chinese BART and Chinese StructBERT as the initial pre-trained language model;

[0022] Step 3.2, entity information recovery: use the corpus after entity information hiding to fine-tune all parameters of the above two pre-trained language models to minimize the loss function of the language model;

[0023] As a further solution of the present invention, step 4 includes the following specific steps:

[0024] Step 4.1, entity model initialization: use the pre-trained language model after entity information recovery as all parameters to initialize the Seq2Seq and Seq2Edit models;

[0025] Step 4.2, entity information enhancement: Use publicly available Chinese spelling correction corpus to fine-tune the initialized Seq2Seq and Seq2Edit models, activate their spelling correction capabilities, and further improve the entity information representation capabilities in the model.

[0026] As a further solution of the present invention, step 5 includes the following specific steps:

[0027] Step 5.1, entity enhancement model selection: obtain the results of Seq2Seq and Seq2Edit models after entity information enhancement, and use the ChERRANT tool to extract the editing operations in different results;

[0028] Step 5.2, entity information fusion: Use the majority voting strategy to improve the accuracy of the entity-focused spelling error correction model, and iteratively apply editing operations to the incorrect sentences to form better correct sentences, and ensure that the editing operations that have been performed are not repeated during the application process.

[0029] In a second aspect, an embodiment of the present invention further provides a Chinese spelling correction device based on entity information enhancement, the device comprising a module for executing the method of the first aspect above.

[0030] In a third aspect, an embodiment of the present invention provides another terminal, comprising a processor, a communication interface, a display screen and a memory, wherein the processor, communication interface, display screen and memory are interconnected, wherein the memory is used to store a computer program that supports the terminal to execute the above method, and the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method of the above first aspect.

[0031] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method of the first aspect.

[0032] In a fifth aspect, an embodiment of the present invention provides a computer program product, which, when running on a computer, enables the computer to execute the method of the first aspect.

[0033] The beneficial effects of the present invention are:

[0034] 1. The present invention constructs a Chinese spelling error correction corpus focusing on entities, referred to as EFCEC. It not only focuses on spelling errors at the word level, but also pays more attention to error recognition and correction at the entity level. By constructing this corpus, we can more deeply study and understand the entity information representation in the pre-trained language model, thereby promoting research progress in related fields.

[0035] 2. The present invention proposes a hierarchical structure of "entity information hiding-entity information recovery-entity information enhancement-entity information fusion", which can effectively improve the model's attention to entity information, and enhance the generalization ability and robustness of the model through a phased training method.

[0036] 3. The present invention has conducted sufficient experiments on three standard public Chinese spelling correction datasets and a dataset constructed by the present invention to verify the effectiveness of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of the present invention; DETAILED DESCRIPTION

[0038] Example 1: Figure 1 As shown, a Chinese spelling correction method based on entity information enhancement, the specific steps of the method are as follows:

[0039] a1. Collect and expand entity dictionaries, obtain and filter unlabeled data, and build entity-focused Chinese spelling correction corpus (EFCEC), including:

[0040] 1) Collect publicly available entity dictionaries such as names of people, places, and institutions, and use regular expressions to extract structured entities from Wikipedia. Use named entity recognition models to identify entities on news and government websites. Finally, aggregate entities from different sources and filter out duplicate entities to form the final entity dictionary;

[0041] 2) Obtain unlabeled corpus from different news and government websites, remove meaningless symbols (English abbreviations, website links, email addresses, web page tags, etc.), and limit the length of sentences to a certain range. Finally, use upsampling technology to balance the distribution of spelling errors in the corpus;

[0042] 3) Segment the corpus after data filtering, use the named entity model to identify entities in the sentence, select characters with similar sound or shape from the confusion set to replace the characters in the identified entity by setting the threshold and the random selection ratio, so as to form a corpus containing entity errors. Finally, use the double annotation and expert review annotation strategy to annotate the validation set and test set of the Chinese spelling corpus focusing on entities, referred to as EFCEC. In this process, 8 undergraduates were asked to annotate the corpus, and each sentence was annotated by 2 undergraduates. Then 3 experts reviewed the annotated corpus to form the final validation set and test set;

[0043] a2. In the entity information hiding module, this method aims to improve the recognition and representation capabilities of the pre-trained language model for entities. Therefore, three different masking strategies are designed to hide entity information in the entity-focused Chinese spelling corpus, including:

[0044] 1) Use the expanded entity dictionary to fine-tune the entity recognition model and use it to identify the location of entities in the entity-focused Chinese spelling correction corpus;

[0045] 2) Design three different masking strategies (static masking, full word masking, and N-gram masking) to hide entity information in recognized entities. Static masking involves randomly masking a character from the entity range, full word masking involves randomly masking the entire entity, and N-gram masking involves randomly masking N consecutive characters in the entity. The three different masking strategies are used to improve the recognition and representation capabilities of the pre-trained language model for entities.

[0046] a3. In the entity information recovery module, the corpus behind the hidden entity information is used to fine-tune the pre-trained language model to focus on the shallow entity information, including:

[0047] 1) Initialize the parameters of the Chinese BART and Chinese StructBER pre-trained language models;

[0048] 2) Use the corpus after entity information hiding to fine-tune all parameters of the above two pre-trained language models to minimize the loss function of the language model, so that the pre-trained language model can learn richer entity information and focus on the shallow entity information in the model;

[0049] a4. In the entity information enhancement module, the public Chinese grammar correction corpus is used to further fine-tune the pre-trained language model to further improve the entity information representation capability in the model, including:

[0050] 1) Use the pre-trained language model after entity information recovery as the baseline model for Seq2Seq and Seq2Edit models. Different from the previous Seq2Seq and Seq2Edit models, the model after entity information recovery focuses more on shallow entity information;

[0051] 2) Use publicly available Chinese spelling correction corpus to fine-tune the initialized Seq2Seq and Seq2Edit models to activate the spelling correction capabilities in the Seq2Seq and Seq2Edit models and enrich the entity information representation capabilities in the models, thereby further improving their attention to entities.

[0052] a5. The results of different pre-trained language models are passed to the entity information fusion module. In the entity information fusion module, the entity information is fused by comparing the entity information between different models to further improve the performance of the model, including:

[0053] 1) First, obtain the results of the Seq2Seq and Seq2Edit models enhanced with entity information. Then, use the ChERRANT tool to extract the editing operations from these results and convert them into the form of (error type, start position, end position, error correction text).

[0054] 2) First, a threshold for accepting edit operations is set. Then, a majority voting strategy is used to improve the accuracy of the entity-focused spelling error model. Finally, by iteratively applying edit operations above the threshold to the incorrect sentence, a correct sentence with better error correction effect is generated, and it is ensured that the implemented edit operations are avoided during the application process.

[0055] The following is an embodiment of the device of the present invention, which is a module used to execute the above method of the present invention. Specifically, it may include:

[0056] The construction module collects publicly available entity dictionaries and expands their coverage. Then it collects unlabeled corpus and performs data filtering. Finally, it constructs entity-focused Chinese spelling corpus through manual annotation and named entity recognition model;

[0057] An entity information hiding module, which uses a masking strategy to hide entity information in entity-focused Chinese spelling corpus;

[0058] The entity information recovery module is used to fine-tune the initial pre-trained language model to focus on shallow entity information;

[0059] The entity information enhancement module is used to further fine-tune the entity information recovery model using the public Chinese grammar correction corpus and further enhance the entity information in the model;

[0060] The entity information fusion module is used to fuse entity information by comparing entity information between different models to further improve the performance of the model.

[0061] An embodiment of the present invention provides another terminal, including a processor, a communication interface, a display screen and a memory, wherein the processor, the communication interface, the display screen and the memory are connected to each other, wherein the memory is used to store a computer program that supports the terminal to execute the above method, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the above method. Each step of the method flow of the present invention can be implemented based on the structure of the terminal.

[0062] An embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, some or all of the steps of any one of the Chinese spelling correction methods recorded in the above method embodiments are executed.

[0063] In a fifth aspect, an embodiment of the present invention provides a computer program product, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program, the computer program being operable to cause a computer to execute some or all of the steps of any one of the Chinese spelling error correction methods described in the above method embodiments. The computer program product may be a software installation package.

[0064] The experiments of the present invention are conducted on a public dataset for Chinese text error correction tasks and a dataset (EFCEC) constructed by the present invention. The former currently includes SIGHAN Bake-off 2013, SIGHAN Bake-off 2014 and SIGHAN Bake-off 2015, a total of three versions.

[0065] In order to measure the effectiveness of the Chinese spelling correction method based on entity information enhancement in the Chinese spelling correction task, the present invention uses precision, recall and F1 value evaluation indicators to evaluate it at both the detection level and the correction level.

[0066] Precision: The percentage of sentences that are correctly identified as incorrect among all sentences that are incorrectly identified by the model. The calculation formula is as follows:

[0067]

[0068] Recall: The percentage of sentences correctly identified as correct among all sentences correctly identified by the model. The calculation formula is as follows:

[0069]

[0070] F1 value (F1): The harmonic mean of precision and recall, used to measure the performance of the model. Its calculation formula is as follows:

[0071]

[0072] In this section, the performance of the Chinese spelling error correction method based on entity information enhancement is verified on the SIGHAN13, SIGHAN14, SIGHAN15 and EFCEC datasets. The method is compared with BERT (a pre-trained language model that selects the most likely characters to be corrected for adjustment), SM-BERT (using the soft masking strategy in BERT to improve error detection performance), DCN (using a dynamic connection network to solve the mutual dependence between adjacent Chinese character pairs), PLOME (combining pinyin and stroke features in BERT pre-training), REALISE (integrating multimodal information including semantics, speech and graphics), LEAD (using contrastive learning techniques to integrate various information from dictionaries), DR-CSC (introducing an easy-to-integrate plug-and-play detection and reasoning module) and DORM (introducing a prediction target and self-distillation module based on pinyin-to-character mapping). Table 1 shows the comparison results on the SIGHAN13, SIGHAN14, SIGHAN15 and EFCEC datasets.

[0073] The following table shows the improvement scores of the EIALM model of the present invention on the detection and error correction tasks on the SIGHAN 13\14\15 and EFCSC test sets constructed by the present invention.

[0074] First, the model of the present invention shows significant improvements over BERT and SM-BERT. On the SIGHAN dataset, EIALM improves the detection task by an average of 7.1F1 and the error correction task by an average of 9.4F1. This may be because although the pre-trained language models have powerful representation capabilities, they pay equal attention to all words and may ignore error-prone entity words. Secondly, compared with REALISE and LEAD, EIALM still performs better even without incorporating additional multimodal information, indicating that activating the potential entity information of the pre-trained language model is crucial for spelling error correction. Third, the simple and efficient EIALM performs better than DR-CSC and DORM, which enhance the knowledge of pinyin, further verifying the effectiveness of the model proposed by the present invention. Fourth, considering that no additional multimodal information is used, the present invention is compared with only two baseline models on the EFCSC test set, where BERT represents the classic use of pre-trained language models, while REALISE represents the typical work of external information integration. On the one hand, we can see that REALISE is far superior to BERT, which shows that external knowledge is helpful to improve spelling error correction. On the other hand, we find that our EIALM performs best among all baselines, which indicates that injecting entity information not only improves the performance of regular spelling correction, but also benefits entity-focused spelling correction. Finally, our EIALM model significantly outperforms all baselines and achieves the best results on all datasets. This indicates that our EIALM injects entity information that helps improve the performance of regular spelling correction.

[0075] Table 1 Results of different methods on SIGHAN13, SIGHAN14, SIGHAN15 and EFCEC datasets

[0076]

[0077]

[0078] The above contents are combined with the accompanying drawings to elaborate the specific embodiments of the present invention. However, the present invention is not limited to the above embodiments. Various changes can be made within the scope of knowledge generally known by those skilled in the art without violating the purpose of the present invention.

Claims

1. A Chinese spelling error correction method based on entity information enhancement, characterized in that: The method comprises the following steps: Step 1: We collected publicly available entity dictionaries and expanded their coverage, then collected unlabeled corpora and performed data filtering. Finally, we constructed entity-focused Chinese spelling corpora through manual annotation and named entity recognition models. Step 2: In the entity information hiding module, a masking strategy is used to hide entity information in the Chinese spelling corpus focusing on the entity; Step 3: Pass the corpus after hiding the entity information to the entity information recovery module, and fine-tune the initial pre-trained language model to make it focus on the shallow entity information; Step 4: Transfer the fine-tuned pre-trained language model to the entity information enhancement module, use the public Chinese grammar correction corpus to further fine-tune the entity information recovery model, and further enhance the entity information in the model; Step 5: Pass the results of different pre-trained language models to the entity information fusion module, and fuse the entity information by comparing the entity information between different models to further improve the performance of the model.

2. The Chinese spelling error correction method based on entity information enhancement according to claim 1, characterized in that: The step 1 comprises the following specific steps: Step 1.1: Entity dictionary construction: Collect publicly available entity dictionaries and structured entities on Wikipedia; then, use named entity recognition models to identify entities on news and government websites; finally, aggregate entities from different sources and filter out duplicate entities to form the final entity dictionary; Step 1.2, unlabeled corpus acquisition and filtering: Obtain unlabeled corpus from different news and government websites, remove meaningless symbols and limit the length of sentences; finally, use upsampling technology to balance the distribution of spelling errors in the corpus; Step 1.3, constructing an entity-focused Chinese spelling corpus: segment the cleaned corpus and use the named entity model to identify entities in the sentence, and use the confusion set to replace the characters in the entity to form a corpus containing entity errors; finally, use the double annotation and expert review strategy to annotate the verification set and test set of the entity-focused Chinese spelling corpus, referred to as EFCEC.

3. The Chinese spelling error correction method based on entity information enhancement according to claim 1, characterized in that: The step 2 comprises the following specific steps: Step 2.1, entity location acquisition: Use the entity recognition model to identify the location of the entity in the Chinese spelling correction corpus focusing on the entity; Step 2.2, entity information hiding: Design three different masking strategies including static masking, full-word masking and N-gram masking to hide the entity information in the entity.

4. The Chinese spelling error correction method based on entity information enhancement according to claim 1, characterized in that: The step 3 comprises the following specific steps: Step 3.1, pre-trained language model acquisition: use Chinese BART and Chinese StructBERT as the initial pre-trained language model; Step 3.2, entity information recovery: Use the corpus after entity information hiding to fine-tune all parameters of the above two pre-trained language models to minimize the loss function of the language model.

5. The Chinese spelling error correction method based on entity information enhancement according to claim 1, characterized in that: The step 4 comprises the following specific steps: Step 4.1, entity model initialization: use the pre-trained language model after entity information recovery as all parameters to initialize the Seq2Seq and Seq2Edit models; Step 4.2, entity information enhancement: Use publicly available Chinese spelling correction corpus to fine-tune the initialized Seq2Seq and Seq2Edit models, activate their spelling correction capabilities, and further improve the entity information representation capabilities in the model.

6. The Chinese spelling error correction method based on entity information enhancement according to claim 1, characterized in that: The step 5 comprises the following specific steps: Step 5.1, entity enhancement model selection: obtain the results of Seq2Seq and Seq2Edit models after entity information enhancement, and use the ChERRANT tool to extract the editing operations in different results; Step 5.2, entity information fusion: Use the majority voting strategy to improve the accuracy of the entity-focused spelling error correction model, and iteratively apply editing operations to the incorrect sentences to form better correct sentences, and ensure that the editing operations that have been performed are not repeated during the application process.

7. A Chinese spelling correction device based on entity information enhancement, characterized in that: The method comprises a module for executing the method according to any one of claims 1 to 6.

8. A terminal, characterized in that: It includes a processor, a communication interface, a display screen and a memory, which are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 6.

10. A computer program product comprising instructions, which, when executed on a computer, enables the computer to perform the method according to any one of claims 1 to 6.

Images