OCR wrongly written character correction algorithm based on Bayesian theorem

Abstract

The invention discloses an OCR wrongly written character correction algorithm based on the Bayesian theorem, and belongs to the technical field of wrongly written character correction, and the OCR wrongly written character correction algorithm based on the Bayesian theorem comprises the following steps: S1, collecting a large amount of text data in professional fields; forming a word frequency distribution model, and converting and training the frequency of the model; s2, correcting wrongly written characters through a Bayesian formula and a correction algorithm; and S3, the correction algorithm is optimized, the performance is enhanced, compared with a mainstream method, the method has extremely low training and maintenance cost, specifically, training does not need data labeling, the algorithm is high in interpretability, non-deep learning is achieved, the computing power cost is extremely low, and the whole algorithm is dynamically planned, does not depend on other extra tools and is maintained by memory deployment.

Description

Technical Field

[0001] The present invention relates to the technical field of typo correction, and more specifically, to an OCR typo correction algorithm based on Bayes' theorem. Background Art

[0002] Currently, the mainstream typo correction algorithms mainly rely on deep learning and large models. With the excellent understanding ability of large models, they can achieve strong generalization.

[0003] However, the side effects brought by using large models, such as high hardware computing power costs and training and maintenance costs, make it difficult to popularize this technology in small enterprises or companies.

[0004] On the other hand, due to the high threshold of large models, it is difficult for them to play a role in specific fields (small samples). For example, in the medical field, "physical health" is abbreviated as "physical fitness". Since the frequency of this word is very low, large models often regard these rare words as typos. Such problems often require specific tuning by senior large model algorithm engineers, which also greatly increases the usage cost of this technology. Summary of the Invention

[0005] 1. Technical Problems to be Solved:

[0006] Aiming at the problems existing in the prior art, based on Bayes' theory, the present invention designs and proposes an OCR typo correction technology with low cost in professional fields.

[0007] 2. Technical Solutions:

[0008] To solve the above problems, the present invention adopts the following technical solutions.

[0009] An OCR typo correction algorithm based on Bayes' theorem, comprising the following steps:

[0010] S1. Collect a large amount of text data in the professional field; form a word frequency distribution model, and perform conversion and training on the frequency of this model;

[0011] S2. Correct typos through Bayes' formula and correction algorithm;

[0012] S3. Optimize the correction algorithm to enhance performance.

[0013] Further improvement lies in that the specific content of S1 includes:

[0014] S11. Prepare a large amount of text data in the professional field, which may include relevant text inputs in the medical scenario (admission and discharge records, outpatient records, test reports), and use the text data output by OCR as the basic data;

[0015] S12. Training only needs to record the frequency of each word appearance, as well as the frequency of other words when word a appears;

[0016] S13. The conditional probability for the frequency conversion of the word frequency distribution model is:

[0017]

[0018] Among them, P(t i+1 丨t i ) represents the probability of t given t i , that is, the conditional probability; i+1 t ∈ T represents any character, and T is the set of characters; t

[0019] and t i represent two adjacent characters; i+1

[0020] freq(t i+1 丨t i ) represents the frequency of t given t i , Z is a fixed value of the total word frequency, taking any value greater than max[freq(t i+1 丨t i+1 丨t i )丨t ∈ T], so, Z = max[freq(t)丨t ∈ T].

[0021] A further improvement lies in that the specific content of S2 includes:

[0022] The Bayes' formula is:

[0023] P(A, B) = P(A) * P(B|A)

[0024] P(A, B) is the joint distribution probability of A and B;

[0025] P(A) is the prior probability or marginal probability of A, without considering any factors of B;

[0026] P(B|A) is the conditional probability of B after A occurs, and the value obtained from A is called the posterior probability of B.

[0027] A further improvement lies in that the correction algorithm of S2 includes:

[0028] S21. Assume a word is a misspelled word, and set the joint distribution of the corresponding characters of the word to be lower than a probability threshold θ. At the same time, set the word as word = [t i , t i+1 ,....t i+n , if and only if:

[0029] θ ≥ P(t i , t​i+1 ,....t i+n ), the threshold θ is regarded as an adjustable parameter;

[0030] S22, calculating the joint distribution based on the assumptions in step S21;

[0031] S23. Find and correct typos.

[0032] A further improvement is that the calculation formula of the joint distribution in S22 includes:

[0033] The joint probability of the first part of multiple words constructed according to the Bayesian formula is:

[0034] P(t i , t i+1 , ..., t i+n )=P(t i )*P(t i+1 |t i )*...*P(t i+n |t i-1+n )

[0035] The joint probability of the latter part of multiple words constructed according to the Bayesian formula is:

[0036] P(t i , t i+1 , ..., t i+n )=P(t i , t i+1 ,...t i+n-1 )*P(t i+n |t i-1+n ).

[0037] A further improvement is that the method for correcting typos in S23 includes:

[0038] S231, considering the joint distribution as an evaluation function, setting the conditions for typos, and setting the score value of the evaluation function to ≤θ;

[0039] S232, optimizing the evaluation function by utilizing the output of the maximization evaluation function;

[0040] S233. Use dynamic programming to solve.

[0041] A further improvement is that the transfer equation of the dynamic programming of S233 includes:

[0042]

[0043] dp[i] represents the maximum joint probability of the first i characters;

[0044] content[i - 1] represents the previous character in the sequence;

[0045] t represents the candidate replacement character for the current character char[i];

[0046] P(t丨char[i - 1]) represents the probability that t appears under the condition of the previous character char[i - 1].

[0047] A further improvement lies in that the optimization of the correction algorithm in S3 includes:

[0048] S31. Through two-way scanning bi-phase detection, maximize the accuracy of identifying typos. The conditions to be met are: when z is a typo among a, b, z, d, e, it is required that P(a, b, z) ≤ θ and P(e, d, z) ≤ θ hold simultaneously;

[0049] S32. When pruning the search space; by deleting candidates with a conditional probability of 0 for sequence[i - 1].

[0050] 3. Beneficial effects:

[0051] Adopting the technical solution provided by the present invention, compared with the prior art, it has the following beneficial effects:

[0052] The present invention is reasonably designed. Compared with the mainstream methods, this method has extremely low training and maintenance costs, specifically manifested in:

[0053] (1) Training does not require data annotation, and the algorithm has strong interpretability

[0054] (2) It is not deep learning, with extremely low computing power costs

[0055] (3) The entire algorithm is just a dynamic programming, does not rely on other additional tools, and is memory-deployed and maintained.

[0056] It should be noted that the structures not introduced in the present invention, since they do not involve the design key points and improvement directions of the present invention, are the same as the prior art or can be implemented using the prior art, and will not be elaborated here. Specific implementation manners

[0057] To facilitate the understanding of the present invention, several embodiments of the present invention are given below. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0058] Embodiment:

[0059] To overcome the deficiencies of the prior art, this patent is based on the Bayesian theory and designs and proposes an OCR typo correction technology with low cost in the professional field.

[0060] The entire algorithm is mainly divided into three major parts:

[0061] 1. Train the word frequency distribution model;

[0062] 2. Use Bayesian correction to correct typos;

[0063] 3. Performance enhancement.

[0064] Part 1. Train the word frequency distribution model:

[0065] The main purpose of the word frequency distribution model is to produce a word list to record the conditional probability distribution of characters and characters, as well as the probability of a single character.

[0066] For example, in "physical examination", it is necessary to count the frequency of "examination" when "physical" is given in the existing situation. And what are the probabilities of "physical" and "examination" appearing respectively.

[0067] The specific approach is as follows:

[0068] The first step is to prepare data:

[0069] The present invention prepares a large amount of text data in the professional field. In the medical scenario, text inputs related to medical care such as admission and discharge records, outpatient records, test reports, etc. can be prepared.

[0070] Here, it is assumed that typos are low-frequency data, so the text data output by OCR can be used as the basic data.

[0071] The second step is to train the data:

[0072] Training only requires recording the frequency of each character and the frequency of other characters when word a appears.

[0073] The third step is to convert frequency to probability:

[0074] The conditional probability is and

[0075] Here: P(t i+1 丨t i ) represents the probability of t i when t i+1 is given, that is, the conditional probability;

[0076] t∈T represents any character, and T is the set of characters; t i and t i+1 represent two adjacent characters;

[0077] freq(t i+1 丨t i ) represents when t is giveni At time t i+1 Frequency, since Z is a fixed value of the total word frequency, so take any value greater than max[freq(t i+1 |t i )|t ∈ T]. Here, let Z = max[freq(t)|t ∈ T].

[0078] Second part: Using Bayes' formula to correct typos

[0079] In this embodiment, assume that a word is a typo, and set the joint distribution of the characters corresponding to this word to be lower than a probability threshold θ. At the same time, set this word as word = [t i , t i+1 ,....t i+n . If and only if:

[0080] θ ≥ P(t i , t i+1 ,....t i+n ). This threshold θ is regarded as an adjustable parameter.

[0081] Based on the above assumptions, the correction algorithm can be divided into two steps:

[0082] 1. Calculate the joint distribution;

[0083] 2. Find and correct typos.

[0084] First step: Calculate the joint distribution

[0085] The core of locating typos is to calculate the joint distribution of words. Therefore, the conditional probabilities between characters have been obtained in the first part. According to Bayes' formula, the joint probability of the first part of multiple characters is:

[0086] P(t i , t i+1 ,..., t i+n ) = P(t i ) * P(t i+1 |t i ) *... * P(t i+n |t i-1+n )

[0087] According to Bayes' formula, the joint probability of the second part of multiple characters is:

[0088] P(t i , t i+1 ,..., t i+n ) = P(t i , t i+1 ,...t i+n-1 ) * P(t i+n |t i-1+n)

[0089] Step 2: Find and correct typos.

[0090] Set the joint distribution as the evaluation function, and regard this problem as an optimization problem:

[0091] Among them, the condition for the typo is: the score of the evaluation function

[0092] The optimization goal is: to maximize the output of the evaluation function

[0093] For this, use dynamic programming to solve this problem:

[0094] Definition of dynamic programming:

[0095] dp[i] represents the maximum joint probability of the first i characters;

[0096] Transition equation:

[0097]

[0098] content[i - 1] represents the previous character in the sequence;

[0099] t represents the candidate replacement character for the current character char[i];

[0100] P(t丨char[i - 1]) represents the probability that t appears under the condition of the previous character char[i - 1].

[0101] The pseudocode is as follows:

[0102]

[0103]

[0104] Part 3: Performance Enhancement

[0105] After the first step and the second step, typos can basically be found and corrected, but there are still two optimization points in the whole algorithm:

[0106] 1. Bidirectional scanning

[0107] 2. Search space pruning

[0108] Bidirectional scanning:

[0109] The first step and the second step mainly introduced the calculation of single - item conditional probability. To maximize the accuracy of identifying typos, through bidirectional detection. When z is a typo among a, b, z, d, e, it is required that P(a, b, z) ≤ θ and P(e, d, z) ≤ θ hold simultaneously;.

[0110] When pruning the search space; when calculating When this is the case, the space of char_space can vary based on different values of sequence[i - 1]. At this time, candidates with a conditional probability of 0 for sequence[i - 1] are removed to reduce the search space.

[0111] Example 1: To facilitate the explanation of the algorithm process, a specific example is given below:

[0112] The first step + the second step: Prepare data + Record word frequencies and:

[0113] Randomly select n medical records and record the word frequencies, as well as 2-gram (the frequency of b when a exists);

[0114] Suppose the following two results are obtained:

[0115] Suppose there are 100 sentences of "high-density lipoprotein" data, and among them, 3 sentences have "lipoprotein" written as "lipoprotein".

[0116] Then the obtained word frequency table is:

[0117] Word frequency high 100 dense 100 Spend 100 fat 100 Egg 100 white 97 since 3 The table of 2-gram is:

[0118]

[0119] The third step: Convert probabilities. Let z be the highest frequency, which is 100 here. Then the probability chart is:

[0120] Word Probability high 1 dense 1 Spend 1 fat 1 Egg 1 white 0.97 since 0.03

[0121] The table of 2-gram:

[0122]

[0123]

[0124] The fourth step: Find typos

[0125] Suppose a sentence is "high-density lipoprotin"

[0126] The idea of finding typos is to find the minimum joint distribution

[0127] This solution will calculate separately

[0128] P(high, density) = 1 * 1 = 1

[0129] P(high, density, lipoprotein) = P(high, density) * P(lipoprotein | density) = 1 * 1 = 1

[0130] P(high, density, lipoprotein, fat) = P(high, density, lipoprotein) * P(fat | lipoprotein) = 1 * 1 = 1

[0131] P(high, density, lipoprotein) = P(high, density, lipid) * P(protein|lipid) = 1 * 1 = 1

[0132] P(high, density, lipoprotein, self) = P(high, density, lipoprotein) * P(self|protein) = 1 * 0.3 = 0.3. It can be seen that the joint distribution of "high, density, lipoprotein, self" is 0.3, and it is also found that the probability drops sharply when traversing to "self".

[0133] If the dropped value is less than the originally defined value, it is considered that there is a typo at this position;

[0134] Let θ = 0.5;

[0135] Then θ > 0.3, so it is determined that "self" is a typo.

[0136] Fifth step of correction:

[0137] The correction goal is to find a t that maximizes the probability of P(high, density, lipoprotein, t)

[0138] Based on Bayes' formula, it is known that:

[0139] P(high, density, lipoprotein, t) = P(high, density, lipoprotein) * P(t|protein)

[0140] When P(high, density, lipoprotein) remains unchanged, maximizing P(high, density, lipoprotein, t) is equivalent to maximizing P(t|protein)

[0141] P(t|protein) includes:

[0142]

[0143] It can be seen that 0.97 is the maximum probability, so "self" will be corrected to "white".

[0144] "High, density, lipoprotein, self" will be corrected to "high, density, lipoprotein".

[0145] The above-described embodiments merely represent certain implementation manners of the present invention. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention patent; it should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several deformations and improvements can still be made, and these all belong to the protection scope of the present invention; therefore, the protection scope of the present invention patent should be subject to the appended claims.

Claims

1. An OCR typo correction algorithm based on Bayesian theorem, characterized in that: It includes the following steps: S1. Collect a large amount of text data in the professional field; form a word frequency distribution model, and perform transformation and training on the frequencies of this model; S2. Correct typos through Bayes' formula and a correction algorithm; S3. Optimize the correction algorithm to enhance performance.

2. The OCR typo correction algorithm based on Bayesian theorem according to claim 1, characterized in that: The specific content of S1 includes: S11. Prepare a large amount of text data in the professional field, which may include relevant text inputs (admission and discharge records, outpatient records, test reports) in the medical scenario, and use the text data output by OCR as the basic data; S12. For training, it only needs to record the frequency of each word appearing, and the frequency of other words when word a appears; S13. The conditional probability for frequency transformation of the word frequency distribution model is: Among them, P(t i+1 丨t i ) represents the probability of t given t i , that is, the conditional probability; i+1 ​ t∈T represents any character, T is a set of characters; t i With t i+1 Represents two adjacent characters; freq(t i+1 丨t i ) represents the frequency at a given t i when t i+1 frequency, Z is a fixed value of the total word frequency, taking any value greater than max[freq(t i+1 丨t i )丨t ∈ T], so, Z = max[freq(t)丨t ∈ T].

3. The OCR typo correction algorithm based on Bayesian theorem according to claim 1, characterized in that: The specific content of S2 includes: Bayes' formula is: P(A, B) = P(A) * P(B|A). P(A, B) is the joint distribution probability of A and B; P(A) is the prior probability or marginal probability of A, without considering any factors of B; P(B|A) is the conditional probability of B after A occurs, and the value obtained from A is called the posterior probability of B.

4. The OCR typo correction algorithm based on Bayesian theorem according to claim 1, characterized in that: The correction algorithm of S2 includes: S21, assuming that a word is a wrong character, and setting the joint distribution of the characters corresponding to the word to be lower than a probability threshold θ, and setting the word to word = [t i ,t i+1 ,....t i+n ], if and only if: θ≥P(t i ,t i+1 ,....t i+n ), the threshold θ is regarded as an adjustable parameter; S22. Calculate the joint distribution based on the assumptions in step S21; S23. Find and correct typos.

5. The OCR typo correction algorithm based on Bayesian theorem according to claim 4, characterized in that: The calculation formula for the joint distribution in S22 includes: The joint probability of the first part of multiple characters constructed according to Bayes' formula is: P(t i ,t i+1 ,...,t i+n )=P(t i )*P(t i+1 |t i )*...*P(t i+n |t i-1+n ) The joint probability of the second part of multiple characters constructed according to Bayes' formula is: P(t i ,t i+1 ,...,t i+n )=P(t i ,t i+1 ,...t i+n-1 )*P(t i+n |t i-1+n )。 6. The OCR typo correction algorithm based on Bayesian theorem according to claim 4, characterized in that: The method for correcting typos in S23 includes: S231. Regard the joint distribution as an evaluation function, set the conditions for typos, and set the score value of the evaluation function to ≤ θ; S232. Optimize it using the output that maximizes the evaluation function; S233. Solve it using dynamic programming.

7. The OCR typo correction algorithm based on Bayesian theorem according to claim 6, characterized in that: The transition equation of the dynamic programming in S233 includes: dp[i] represents the maximum joint probability of the first i characters; content[i - 1] represents the previous character in the sequence; t represents the candidate replacement character for the current character char[i]; P(t丨char[i - 1]) represents the probability of t appearing under the condition of the previous character char[i - 1].

8. The OCR typo correction algorithm based on Bayesian theorem according to claim 6, characterized in that: The optimization of the correction algorithm in S3 includes: S31. Through two-way scanning and two-phase detection, maximize the accuracy of identifying typos. The condition is: when z is a typo among a, b, z, d, e, it is required that P(a, b, z) ≤ θ and P(e, d, z) ≤ θ hold simultaneously; S32. When pruning the search space; delete the candidates with a conditional probability of 0 for sequence[i - 1].

Images