A method for implementing Chinese text error correction based on similarity

Abstract

The present invention provides a method for implementing Chinese text error correction based on similarity, characterized in that it includes the following steps: compiling a word library; using the compiled word library, using word hash to mark the text to be detected, obtaining a sequence corresponding to the text to be detected, and recording id; using double-word hash to calculate the sequence, find all words in the text to be detected that are similar to the word library, and obtain a suspected similar word list after calculation; performing similarity calculation on each word in the suspected similar word list described in S3, retaining words with similarity greater than 70, and obtaining a suspected error list after calculation; error correction. The present invention can find possible errors in Chinese text in advance, greatly reducing labor costs, and can improve the detection rate and accuracy of existing text error correction, has a wide range of application prospects, and can also be applied to plagiarism judgment, article duplicate checking, etc.

Description

Technical Field

[0001] The present invention relates to the field of text error correction, and in particular to a method for implementing Chinese text error correction based on similarity. Background Art

[0002] At present, there are some relatively good detection solutions for Chinese texts, typos, and sensitive words. For typos, there are methods such as the Kenlm statistical language model tool, the transformer model, the conv_seq2seq model, the BERT model, and the improved model based on BERT for detection and recognition. For sensitive words, the sensitive word database is recorded and then detected and matched for detection and recognition. Although the accuracy and recall rate of typos and sensitive word detection and recognition need to be improved, the detection results can provide people with a certain reference and greatly reduce the manual workload.

[0003] Among them, in terms of typos recognition, PyCorrector is a Chinese text correction tool. PyCorrector detects the position of typos based on the language model, and corrects typos through pinyin sound similarity features, stroke Wubi edit distance features, and language model perplexity features. It integrates the various models mentioned above and provides quick usage methods for multiple models, such as: PyCorrector integrates Bert detection to identify typos. However, the existing error correction methods in Chinese texts cannot find all errors. The method in this article cannot solve all problems, but can only improve the detection rate and accuracy. Summary of the invention

[0004] The purpose of the present invention is to provide a method for implementing Chinese text error correction based on similarity, so as to improve the detection rate and accuracy of the existing Chinese text error correction method, better discover and correct expression errors, reduce the proofreading workload of text workers, and improve work efficiency. The core of the present invention is a fast similarity calculation method, and text error correction is only a specific application. In addition, it can also be applied to plagiarism judgment, article duplication check, etc., to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for implementing Chinese text error correction based on similarity, characterized by comprising the following steps:

[0007] S1: compile the word library, collect fixed expressions in the word library, and it is wrong to have too many words, too few words, or replace them with other words, so that the length of the word is greater than 4 words. After completion, a data structure that can be quickly retrieved is obtained;

[0008] S2: Use the word library compiled in S1 to mark the text to be detected using word hashing to obtain a sequence corresponding to the text to be detected, which records the IDs of all words corresponding to each character in the text to be detected. These words are all from the compiled word library;

[0009] S3: Use double-word hashing to calculate the sequence described in S2, and find all words in the text to be detected that are similar to the word library. The main basis is that the discreteness of the words in the word library in the text to be detected is less than 3 and the number of occurrences is greater than 50% of the word length. After calculation, a list of suspected similar words is obtained;

[0010] Among them, the discreteness is the number of characters that appear consecutively but are not in the correct word;

[0011] S4: Calculate the similarity of each word in the suspected similar word list described in S3, retain the words with similarity greater than 70, and obtain a suspected error list after calculation;

[0012] S5: Error correction, filter out the words in the suspected error list that meet one of the following conditions, and the remaining ones are the final error correction results:

[0013] 1) Similarity is 100;

[0014] 2) The similarity is less than 100, but it is a true substring of the word with similarity of 100.

[0015] In S1, the process of compiling the word library is as follows:

[0016] S1.1: Preparation: word set, all words that have been compiled; word hash: key is a character, value is the set of ids of all words containing this character; double word hash: key is two characters, value is the set of ids of words containing both characters in the key, where the order of characters is not required;

[0017] S1.2: If the word to be compiled already exists in the word set, the compilation is completed; otherwise, the word to be compiled is added to the word set, and at the same time, it has an id and enters S1.3;

[0018] S1.3: Perform the following processing on each character in the compiled word:

[0019] If it is the key of a word hash, add the id of the word to be compiled to the corresponding word id set; otherwise, add the key of this character to the word hash, and the corresponding value is a set with only one element of the id of the word to be compiled. At the same time, add this character and the key combination of all word hashes as the key in the double word hash, and the value is the value of the empty set;

[0020] S1.4: All values ​​of the double-word hash whose two characters are in the word to be compiled are added to the word id to be compiled.

[0021] In S5, the similarity calculation process is as follows:

[0022] S5.1: Preparation: Difference, used to record the difference between two words, initially 0; correct word, which is the word we use for compilation; similar word, which is the word in the text to be tested;

[0023] S5.2: Difference + = number of characters (belonging to the correct word and not belonging to the similar word) * length of the similar word;

[0024] S5.3: Difference + = number of characters (belonging to similar words and not to the correct word) * length of the correct word;

[0025] S5.4: For each character that belongs to both the correct word and a similar word, calculate the difference as follows;

[0026] S5.5: Difference + = |Occurrence position in the correct word – Occurrence position in the similar word|;

[0027] S5.6: Similarity = 100-difference*100 / (correct word length*similar word length).

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention provides a method for implementing Chinese text error correction. It is time-consuming and laborious to manually check whether there are errors in Chinese text. After using the error correction method provided in this article, possible errors in Chinese text can be found in advance, which greatly reduces the labor cost and can improve the detection rate and accuracy of existing text error correction. The present invention can be used in various Chinese texts and has a wide range of application prospects. The core of the present invention is a fast similarity calculation method. Text error correction is only a specific application. In addition, it can also be applied to plagiarism judgment, article duplication detection, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the overall process of the present invention based on similarity text error correction;

[0031] Figure 2 This is a schematic diagram of the vocabulary compilation process of the present invention;

[0032] Figure 3 It is a schematic diagram of the similarity calculation process of the present invention;

[0033] Figure 4 This is a topology diagram after the software deployment of this embodiment is completed. DETAILED DESCRIPTION

[0034] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0035] This is a pure software system that can be deployed as a standalone system or integrated with other systems. A deployment implementation method is as follows:

[0036] Example 1

[0037] 1. Hardware environment configuration requirements:

[0038] Configuration items Recommended parameters CPU 2 cores, main frequency no less than 2.0GHz Memory No less than 8GB harddisk Rotation speed 7200, no less than 100GB Network Card Gigabit LAN

[0039] 2. It is recommended that the system be designed using a microservice architecture and deployed using containers. The benefits of doing so are:

[0040] Improve system reliability and availability; make capacity expansion and reduction very easy; reduce the difficulty of integration with other systems;

[0041] 3. Java is recommended as the development language, as it has high development efficiency and the system can be deployed across platforms;

[0042] 4. Related software: The recommended operating system is CentOS7.0 or above, jdk1.8;

[0043] The topology diagram after deployment is as follows: Figure 4 shown.

[0044] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only preferred examples of the invention and are not intended to limit the present invention. Without departing from the novel spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for implementing Chinese text error correction based on similarity, characterized in that: The steps include: S1: compile the word library, collect fixed expressions in the word library, and it is wrong to have too many words, too few words, or replace them with other words, so that the length of the word is greater than 4 words. After completion, a data structure that can be quickly retrieved is obtained; S2: Use the word library compiled in S1 to mark the text to be detected using word hashing to obtain a sequence corresponding to the text to be detected, which records the IDs of all words corresponding to each character in the text to be detected. These words are all from the compiled word library; S3: Use double-word hashing to calculate the sequence described in S2, and find all words in the text to be tested that are similar to the word library. The main basis is that the discreteness of the words in the word library in the text to be tested is less than 3 and the number of occurrences is greater than 50% of the word length. After calculation, a list of suspected similar words is obtained; Among them, the discreteness is the number of characters that do not appear continuously in the correct word; S4: Calculate the similarity of each word in the suspected similar word list described in S3, retain the words with similarity greater than 70, and obtain a suspected error list after calculation; S5: Error correction, filter out the words in the suspected error list that meet one of the following conditions, and the remaining ones are the final error correction results: 1) Similarity is 100; 2) The similarity is less than 100 and is a true substring of the word with a similarity of 100; In S5, the similarity calculation process is as follows: S5.1: Preparation: Difference, used to record the difference between two words, initially 0; Correct word, used for compiled words; Similar word, is the word in the text to be tested; S5.2: Difference += the number of correct words that are not similar words * the length of similar words; S5.3: Difference += the number of similar words that are not correct words * the length of the correct word; S5.4: For each character that belongs to both the correct word and a similar word, calculate the difference as follows; S5.5: Difference += |occurrence position in correct word – occurrence position in similar word|; S5.6: Similarity = 100 - Difference * 100 / (Correct word length * Similar word length).

2. The method for implementing Chinese text error correction based on similarity according to claim 1 is characterized in that: In S1, the process of compiling the word library is as follows: S1.1: Preparation: word set, all words that have been compiled; word hash: key is a character, value is the set of ids of all words containing this character; double word hash: key is two characters, value is the set of ids of words containing both characters in the key, where the order of characters is not required; S1.2: If the word to be compiled already exists in the word set, the compilation is completed; Otherwise, add the word to be compiled to the word set, and it will have an id at the same time, and enter S1.3; S1.3: Perform the following processing on each character in the compiled word: If it is the key of a word hash, add the id of the word to be compiled to the corresponding word id set; otherwise, add the key of this character to the word hash, and the corresponding value is a set with only one element of the id of the word to be compiled. At the same time, add this character and the key combination of all word hashes as the key in the double word hash, and the value is the value of the empty set; S1.4: All values ​​of the double-word hash whose two characters are in the word to be compiled are added to the word id to be compiled.

Images