Method and device for identifying optical character recognition errors in corpus

Abstract

The embodiment of the present invention provides a method and device for identifying optical character recognition errors in a corpus. The method comprises: performing word segmentation on each sentence in the corpus to be proofread to obtain each word; for each word in the sentence, obtaining a score for the word based on a pre-acquired language model and sentence; if it is determined that the score of the word is lower than a preset normal score, the sentence is determined to be a sentence that may have an optical character recognition error. The method and device for identifying optical character recognition errors in a corpus provided by the embodiment of the present invention, by performing word segmentation on each sentence in the corpus to be proofread, obtaining a score for each word based on a pre-acquired language model and sentence, and determining the sentence as a sentence that may have an optical character recognition error when the score of the word is lower than a preset normal score, can identify most sentences with optical character recognition errors and can identify more optical character recognition errors.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and more specifically, to a method and apparatus for identifying optical character recognition errors in a corpus. Background Art

[0002] In order to conveniently obtain various materials, optical character recognition (OCR) is often performed on documents to obtain recognized corpus.

[0003] Due to the limitations of OCR technology, there are often some recognition errors in the corpus, such as identifying "l" as "1", "O" or "o" as "0", "m" as "nn", "h" as "n", etc. If these errors remain in the corpus, they will seriously affect the translation model trained using this corpus. Therefore, it is very important to identify and remove OCR errors in the corpus.

[0004] Existing methods for identifying OCR errors mainly perform error detection based on preset rules (such as repeated characters, illegal characters that are not allowed characters), etc. Therefore, the existing technology can only identify a small number of errors. Summary of the Invention

[0005] Embodiments of the present invention provide a method and apparatus for identifying optical character recognition errors in a corpus, so as to solve or at least partially solve the defect of the prior art that only a small number of errors can be identified.

[0006] In a first aspect, an embodiment of the present invention provides a method for identifying optical character recognition errors in a corpus, comprising:

[0007] Perform word segmentation on each sentence in the proofreading corpus to obtain each word;

[0008] For each word in the sentence, obtaining a score for the word based on a pre-acquired language model and the sentence;

[0009] If it is determined that the score of the word is lower than a preset normal score, the sentence is determined to be a sentence that may have an optical character recognition error.

[0010] Preferably, for each word in the sentence, the specific step of obtaining a score for the word based on the pre-obtained language model and the sentence includes:

[0011] According to the pre-acquired language model and the sentence, the perplexity of the word is obtained as the score of the word.

[0012] Preferably, if it is determined that the score of the word is lower than a preset normal score, after determining that the sentence is a sentence that may have an optical character recognition error, the method further includes:

[0013] The sentence is checked according to the context of the sentence to determine whether there is an optical character recognition error in the sentence.

[0014] Preferably, the language model is a five-gram language model.

[0015] In a second aspect, an embodiment of the present invention provides a device for identifying optical character recognition errors in a corpus, comprising:

[0016] The segmentation module is used to segment each sentence in the proofreading corpus into words and obtain each word;

[0017] A scoring module, configured to obtain a score for each word in the sentence based on a pre-acquired language model and the sentence;

[0018] The recognition module is configured to determine that the sentence is a sentence that may have an optical character recognition error if it is determined that the score of the word is lower than a preset normal score.

[0019] Preferably, the scoring module is specifically configured to obtain the perplexity of the word as the score of the word based on the pre-acquired language model and the sentence.

[0020] Preferably, the apparatus for identifying optical character recognition errors in a corpus further comprises:

[0021] The checking module is used to check the sentence according to the context of the sentence to determine whether there is an optical character recognition error in the sentence.

[0022] Preferably, the language model is a five-gram language model.

[0023] In a third aspect, an embodiment of the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the program is executed, the steps of a method for identifying optical character recognition errors in a corpus as provided in any one of the various possible implementations of the first aspect are implemented.

[0024] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a method for identifying optical character recognition errors in a corpus as provided in any one of the various possible implementations of the first aspect.

[0025] The method and apparatus for identifying optical character recognition errors in a corpus provided by an embodiment of the present invention segment each sentence in the corpus to be proofread into words, obtain a score for each word based on a pre-acquired language model and sentence, and determine that a sentence may contain an optical character recognition error when the score of the word is lower than a preset normal score. This method and apparatus can identify most sentences with optical character recognition errors and can identify more optical character recognition errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 A schematic diagram of a flow chart of a method for identifying optical character recognition errors in a corpus according to an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of the structure of an apparatus for identifying optical character recognition errors in a corpus according to an embodiment of the present invention;

[0029] Figure 3 The figure is a schematic diagram of the physical structure of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] Figure 1 Schematic diagram of a method for identifying optical character recognition errors in a corpus according to an embodiment of the present invention. Figure 1 As shown, the method includes: step S101, performing word segmentation on each sentence in the corpus to be proofread to obtain each word.

[0032] It can be understood that the corpus to be proofread is corpus obtained through optical character recognition (OCR).

[0033] Any word segmentation method can be used to segment each sentence in the proofreading corpus into several words.

[0034] Step S102: For each word in the sentence, obtain a score for the word based on the pre-acquired language model and sentence.

[0035] Specifically, a language model is an abstract mathematical model of language based on objective language facts, and is a correspondence relationship.

[0036] A language model is a simple, unified, and abstract formal system. The objective facts of language are described by the language model and are more suitable for automatic processing by electronic computers. Therefore, the language model is of great significance for the information processing of natural language.

[0037] Language models are typically applied to entire sentences to check their coherence. They are a common method for filtering corpora at the sentence level. Language models can also be used for grammar checking.

[0038] A common language model is established for sentences based on words, while the language model in the embodiment of the present invention is similar in principle to a common language model, but is established for words based on letters, and can therefore be used to detect whether a word is normal.

[0039] In the embodiment of the present invention, the language model based on letters is established as follows:

[0040] After word segmentation (word segmentation) of the entire monolingual text, a language model is established for each word obtained by segmentation.

[0041] It is understandable that when establishing a language model in the embodiment of the present invention, each letter in a word is regarded as each word in a common language model.

[0042] For each sentence, based on the language model, each word obtained after word segmentation of the sentence is scored in turn to obtain the score of the word.

[0043] The score is used to indicate the degree to which the word is a normal word.

[0044] It should be noted that before step S102 , training is performed based on the sample corpus to obtain a language model and a normal score T of each word appearing in the sample corpus.

[0045] The sample corpus is monolingual.

[0046] For each word appearing in the sample corpus, the minimum value of the word's score can be used as the normal score T of the word. When the sub-frequency of the word is greater than a preset number of times, the minimum value of the word's score can be used as the normal score T of the word.

[0047] Step S103: If it is determined that the score of the learned word is lower than the preset normal score, the sentence is determined to be a sentence that may have an optical character recognition error.

[0048] Specifically, the score of the word is compared with the normal score T of the word.

[0049] If the score of the word is greater than or equal to the normal score T of the word, the score of the next word is obtained and compared with the normal score of the next word.

[0050] If the score of the word is less than or equal to the normal score T of the word, the sentence is determined to be a sentence that may have an optical character recognition error, and the word may be an OCR error word.

[0051] The embodiment of the present invention performs word segmentation on each sentence in the corpus to be proofread, obtains a score for each word based on a pre-acquired language model and sentence, and determines that a sentence may have an optical character recognition error when the score of the word is lower than a preset normal score. This embodiment of the present invention can identify most sentences with optical character recognition errors and can identify more optical character recognition errors.

[0052] Based on the contents of the above embodiments, for each word in a sentence, the specific steps of obtaining a word score according to a pre-acquired language model and sentence include: obtaining the perplexity of the word as the word score according to the pre-acquired language model and sentence.

[0053] Specifically, for each word obtained after word segmentation of the sentence, the perplexity of the word can be obtained according to the pre-obtained language model and sentence, as the score of the word.

[0054] Perplexity (ppl) means the geometric mean of the inverse of the probability of sentence occurrence.

[0055] Perplexity (ppl) can also be thought of as the average branch factor, which is how many choices there are when predicting the next word.

[0056] In the embodiment of the present invention, the calculation formula of the perplexity is:

[0057]

[0058]

[0059] Where l represents the letters in the word; P represents the probability; and N represents the total number of letters in the word.

[0060] The embodiment of the present invention obtains the perplexity of a word based on a pre-acquired language model and sentence as a word score, thereby being able to identify sentences that may have optical character recognition errors based on the word score and to identify more optical character recognition errors.

[0061] Based on the contents of the above embodiments, if it is determined that the score of the learned word is lower than the preset normal score, then after determining the sentence as a sentence that may have an optical character recognition error, the method further includes: checking the sentence according to the context of the sentence to determine whether there is an optical character recognition error in the sentence.

[0062] Specifically, after identifying a sentence that may contain an optical character recognition error, the sentence may be checked based on the context of the sentence.

[0063] Checking the sentence may include grammar checking and / or semantic checking.

[0064] By checking, it can be determined whether there are any optical character recognition errors in the sentence.

[0065] If it is determined that there is an optical character recognition error in the sentence, manual or automatic correction can be performed, or the sentence can be discarded and the optical character recognition can be performed again.

[0066] The embodiment of the present invention checks a sentence that may have an optical character recognition error based on the context of the sentence to determine whether the sentence contains an optical character recognition error, thereby improving the accuracy of identifying optical character recognition errors.

[0067] Based on the contents of the above embodiments, the language model is a five-gram language model.

[0068] Specifically, the language model may adopt an n-gram language model. When n=5, the language model is a five-gram language model, adopting a five-gram grammar.

[0069] The N-gram model assigns probabilities to sentences and word sequences. A typical N-gram is a sequence of N words: a 2-gram (or bigram) is a sequence of two words, such as "please turn," "turn your," or "yourhomework"; a 3-gram (or trigram) is a sequence of three words, such as "please turn your," or "turn your homework"; and a 5-gram is a sequence of five words.

[0070] In this embodiment, the N-gram model probability distribution is based on the letters in each word. For example, a 2-gram is a sequence of two letters, such as lo and ma; a 3-gram is a sequence of three letters, such as lov and mat; and a 5-gram is a sequence of five letters, such as match and faith.

[0071] The embodiment of the present invention adopts a five-gram language model for the first time, and can score words based on the language model. When the score of a word is lower than a preset normal score, the sentence is determined to be a sentence that may have optical character recognition errors. Most sentences with optical character recognition errors can be identified, and more optical character recognition errors can be identified.

[0072] Figure 2 Schematic diagram of the structure of the apparatus for identifying optical character recognition errors in corpus according to an embodiment of the present invention. Figure 2 As shown, the apparatus includes a segmentation module 201, a scoring module 202, and an identification module 203, wherein:

[0073] The segmentation module 201 is used to segment each sentence in the proofreading corpus into words to obtain each word;

[0074] Scoring module 202, for obtaining a score for each word in a sentence based on a pre-acquired language model and the sentence;

[0075] The recognition module 203 is configured to determine that the sentence may contain an optical character recognition error if the score of the learned word is determined to be lower than a preset normal score.

[0076] Specifically, the segmentation module 201 , the scoring module 202 and the identification module 203 are electrically connected in sequence.

[0077] The segmentation module 201 can perform word segmentation on each sentence in the proofreading corpus by any word segmentation method, and divide the sentence into a plurality of words.

[0078] For each sentence, the scoring module 202 scores each word obtained after word segmentation of the sentence based on the language model to obtain a score for the word.

[0079] The recognition module 203 compares the score of the word with the normal score T of the word.

[0080] If the score of the word is greater than or equal to the normal score T of the word, the scoring module 202 obtains the score of the next word, and the recognition module 203 compares the score of the next word with the normal score of the next word.

[0081] If the score of the word is less than or equal to the normal score T of the word, the recognition module 203 determines that the sentence is a sentence that may have an optical character recognition error, and the word may be an OCR error word.

[0082] The device for identifying optical character recognition errors in a corpus provided in an embodiment of the present invention is used to execute the method for identifying optical character recognition errors in a corpus provided in the above-mentioned embodiments of the present invention. The specific methods and processes for implementing corresponding functions of each module included in the device for identifying optical character recognition errors in a corpus are detailed in the embodiment of the method for identifying optical character recognition errors in a corpus, and will not be repeated here.

[0083] The apparatus for identifying optical character recognition errors in a corpus is used in the methods for identifying optical character recognition errors in a corpus in the aforementioned embodiments. Therefore, the descriptions and definitions of the methods for identifying optical character recognition errors in a corpus in the aforementioned embodiments can be used to understand the various execution modules in the embodiments of the present invention.

[0084] The embodiment of the present invention performs word segmentation on each sentence in the corpus to be proofread, obtains a score for each word based on a pre-acquired language model and sentence, and determines that a sentence may have an optical character recognition error when the score of the word is lower than a preset normal score. This embodiment of the present invention can identify most sentences with optical character recognition errors and can identify more optical character recognition errors.

[0085] Based on the contents of the above embodiments, the scoring module is specifically used to obtain the perplexity of a word according to a pre-acquired language model and sentence as a score of the word.

[0086] Specifically, for each word obtained after word segmentation of the sentence, the scoring module can obtain the perplexity of the word according to the pre-acquired language model and sentence as the score of the word.

[0087] The embodiment of the present invention obtains the perplexity of a word based on a pre-acquired language model and sentence as a word score, thereby being able to identify sentences that may have optical character recognition errors based on the word score and to identify more optical character recognition errors.

[0088] Based on the contents of the above embodiments, the apparatus for identifying optical character recognition errors in a corpus further includes: a checking module for checking a sentence according to the context of the sentence to determine whether there is an optical character recognition error in the sentence.

[0089] Specifically, the inspection module is electrically connected to the identification module.

[0090] The checking module can check the sentence according to the context of the sentence.

[0091] By checking, it can be determined whether there are any optical character recognition errors in the sentence.

[0092] The embodiment of the present invention checks a sentence that may have an optical character recognition error based on the context of the sentence to determine whether the sentence contains an optical character recognition error, thereby improving the accuracy of identifying optical character recognition errors.

[0093] Based on the contents of the above embodiments, the language model is a five-gram language model.

[0094] Specifically, the language model may adopt an n-gram language model. When n=5, the language model is a five-gram language model, adopting a five-gram grammar.

[0095] The embodiment of the present invention adopts a five-gram language model for the first time, and can score words based on the language model. When the score of a word is lower than a preset normal score, the sentence is determined to be a sentence that may have optical character recognition errors. Most sentences with optical character recognition errors can be identified, and more optical character recognition errors can be identified.

[0096] Figure 3 FIG. 1 is a schematic diagram of the physical structure of an electronic device according to an embodiment of the present invention. Figure 3 As shown, the electronic device may include: a processor 301, a memory 302 and a bus 303; wherein the processor 301 and the memory 302 communicate with each other via the bus 303; the processor 301 is used to call computer program instructions stored in the memory 302 and executable on the processor 301 to execute the method for identifying optical character recognition errors in a corpus provided by the above-mentioned method embodiments, for example, including: performing word segmentation on each sentence in the corpus to be proofread to obtain each word; for each word in the sentence, obtaining a score for the word based on a pre-acquired language model and sentence; if it is determined that the score of the word is lower than a preset normal score, determining the sentence as a sentence that may have an optical character recognition error.

[0097] Another embodiment of the present invention discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the method for identifying optical character recognition errors in a corpus provided by the above-mentioned method embodiments, for example, including: performing word segmentation on each sentence in the corpus to be proofread to obtain each word; for each word in the sentence, obtaining a score for the word based on a pre-obtained language model and sentence; if it is determined that the score of the word is lower than a preset normal score, determining the sentence as a sentence that may have an optical character recognition error.

[0098] In addition, the logic instructions in the above-mentioned memory 302 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0099] Another embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable a computer to execute the method for identifying optical character recognition errors in a corpus provided by the above-mentioned method embodiments, for example, including: performing word segmentation on each sentence in the corpus to be proofread to obtain each word; for each word in the sentence, obtaining a score for the word based on a pre-obtained language model and sentence; if it is determined that the score of the word is lower than a preset normal score, determining the sentence as a sentence that may have an optical character recognition error.

[0100] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. That is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0101] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by means of hardware. With this understanding, the essence of the above technical solution or the portion that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of each of the above embodiments or certain portions of the embodiments.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for identifying optical character recognition errors in a corpus, characterized in that: include: Perform word segmentation on each sentence in the proofreading corpus to obtain each word; For each word in the sentence, obtaining a score for the word based on a pre-acquired language model and the sentence; If it is determined that the score of the word is lower than a preset normal score, the sentence is determined to be a sentence that may have an optical character recognition error; The specific steps of obtaining a score for each word in the sentence based on the pre-acquired language model and the sentence include: Obtaining the perplexity of the word according to the pre-acquired language model and the sentence as a score of the word; The meaning of the perplexity is the geometric mean of the inverse of the sentence occurrence probability.

2. The method for identifying optical character recognition errors in a corpus according to claim 1, wherein: If it is determined that the score of the word is lower than a preset normal score, then after determining that the sentence is a sentence that may have an optical character recognition error, the method further includes: The sentence is checked according to the context of the sentence to determine whether there is an optical character recognition error in the sentence.

3. The method for identifying optical character recognition errors in a corpus according to any one of claims 1 to 2, characterized in that: The language model is a five-gram language model.

4. A device for identifying optical character recognition errors in a corpus, characterized in that: include: The segmentation module is used to segment each sentence in the proofreading corpus into words and obtain each word; A scoring module, configured to obtain a score for each word in the sentence based on a pre-acquired language model and the sentence; a recognition module configured to determine the sentence as a sentence that may contain an optical character recognition error if it is determined that the score of the word is lower than a preset normal score; The scoring module is specifically used to obtain the perplexity of the word as the score of the word based on the pre-acquired language model and the sentence; the meaning of the perplexity is the geometric mean of the inverse of the sentence occurrence probability.

5. The device for identifying optical character recognition errors in corpus according to claim 4, characterized in that: Also includes: The checking module is used to check the sentence according to the context of the sentence to determine whether there is an optical character recognition error in the sentence.

6. The device for identifying optical character recognition errors in a corpus according to any one of claims 4 to 5, characterized in that: The language model is a five-gram language model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for identifying optical character recognition errors in a corpus as claimed in any one of claims 1 to 3 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for identifying optical character recognition errors in a corpus as claimed in any one of claims 1 to 3 are implemented.

Images