An artificial intelligence-based misspelling character recognition system and method

Abstract

The application discloses a wrong character recognition system and method based on artificial intelligence, and belongs to the technical field of character recognition systems, and comprises a text input and preprocessing module, a local AI wrong character preliminary judgment module, a full-text character retrieval and statistics module, a wrong character comprehensive probability calculation module, a dynamic adaptive threshold calculation module and a decision output module; the text input and preprocessing module is used for acquiring a text to be detected, and completes text cleaning, format unification and character coding; the local AI wrong character preliminary judgment module is used for preliminarily identifying single characters or local character fragments in the text, and outputs local wrong character probabilities; and the full-text character retrieval and statistics module is used for searching and counting the total appearance times of characters same as or similar to suspected wrong characters in the full text when the local preliminary judgment determines that the suspected wrong characters are wrong characters. The application has a three-level recognition mechanism of local preliminary judgment, global verification and dynamic decision, and has the advantages of unrestrained improvement.

Description

Technical Field

[0001] This invention belongs to the field of character recognition system technology, and in particular relates to an artificial intelligence-based misspelled character recognition system and method. Background Technology

[0002] Typo recognition is a key technology in text proofreading, official document review, educational correction, and OCR post-processing. Its purpose is to automatically detect character shape errors, pronunciation errors, semantically contradictory characters, or identify noisy characters in text, and provide location and correction suggestions to improve text standardization and accuracy. With the widespread application of digital text, high-precision, low-false-positive, and robust typo recognition systems have become an important requirement for office automation and intelligent processing.

[0003] Current misspelling detection technologies mainly employ methods such as character comparison based on dictionary matching and rule bases, local rationality detection based on single-sentence semantic analysis, classification and recognition of individual characters or local segments based on traditional machine learning or deep learning models, and simple filtering based on OCR output confidence. Because most of these methods focus on single characters or local text features, they do not fully utilize the character distribution patterns across the entire text.

[0004] Because existing misspelling recognition technologies rely solely on local features for judgment, they are prone to misclassifying correct characters that appear abnormal in certain parts of the text but are consistent and standardized throughout. Furthermore, since the judgment thresholds are mostly fixed values, they cannot adaptively adjust based on text length, character frequency, or text complexity, resulting in high false positive rates for long texts and high false negative rates for short texts. Additionally, existing technologies lack a mechanism for verifying the consistency of similar characters across the entire text, and the various modules operate independently without collaborative linkage, making it difficult to meet the overall accuracy and stability requirements for high-quality text proofreading. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides an artificial intelligence-based misspelled character recognition system and method, which has a three-level recognition mechanism of local preliminary judgment, global verification, and dynamic judgment, improving the advantage of no omissions and solving the problems of the prior art.

[0006] This invention is implemented as follows: an AI-based misspelling character recognition system, comprising: a text input and preprocessing module, a local AI misspelling preliminary judgment module, a full-text character retrieval and statistics module, a misspelling comprehensive probability calculation module, a dynamic adaptive threshold calculation module, and a decision output module. The text input and preprocessing module acquires the text to be detected, performs text cleaning, format unification, and character encoding. The local AI misspelling preliminary judgment module performs preliminary recognition of individual characters or local character fragments in the text, outputting the local misspelling probability. The full-text character retrieval and statistics module, when a local misspelling is initially judged as a suspected misspelling, retrieves and counts the total number of occurrences of characters identical or similar to the suspected misspelling in the full text, as well as the total number of characters in the full text. The misspelling comprehensive probability calculation module fuses the local misspelling probability with the full-text character distribution characteristics to calculate the comprehensive misspelling probability. The dynamic adaptive threshold calculation module adaptively calculates a dynamic decision threshold based on the full text length and the average frequency of characters in the full text. The decision output module compares the comprehensive misspelling probability with the dynamic decision threshold and outputs the final misspelling judgment result.

[0007] The misspelling probability calculation module executes the following calculation formula: Where P is the overall probability of misspellings. `local` represents the local error probability output by the local AI error detection module, `Nsame` represents the total number of occurrences of characters that are the same as or similar to the current character in the entire text, and `Ntotal` represents the total number of characters in the entire text. To prevent the smoothing coefficient from being zero in the denominator, These are the weighting coefficients for local and global features. .

[0008] The dynamic adaptive threshold calculation module performs the following calculation formula: Where T is the final dynamic decision threshold, and T0 is the basic decision threshold. For balance coefficient, Here, Ltext is the normalization function, favg is the total text length, and favg is the average frequency of occurrence of characters in the total text.

[0009] As a preferred embodiment of the present invention, when the overall probability P is greater than the dynamic threshold T, the current character is determined to be a true typo; when the overall probability P is less than or equal to the dynamic threshold T, the current character is determined to be a normal character.

[0010] An AI-based method for identifying misspelled characters, applied to the system, includes the following steps: S1, acquiring the text to be detected and preprocessing it through a text input and preprocessing module; S2, performing preliminary judgment on each character of the text through a local AI misspelling module to obtain a local misspelling probability Plocal; S3, when Plocal reaches the suspected misspelling condition, triggering a full-text character retrieval and statistics module to count N_same and Ntotal; S4, obtaining a comprehensive misspelling probability P through a comprehensive misspelling probability calculation module; S5, obtaining a dynamic decision threshold T through a dynamic adaptive threshold calculation module; S6, comparing P and T to output the final judgment result on whether it is a misspelling.

[0011] As a preferred embodiment of the present invention, the statistics of identical or similar characters throughout the text are used to achieve global consistency verification and suppress misjudgments caused by local features.

[0012] As a preferred embodiment of the present invention, the dynamic decision threshold T adaptively increases with the increase of text length and adaptively decreases with the sparse distribution of characters.

[0013] As a preferred embodiment of the present invention, similar characters include characters with similar shapes, characters with similar sounds, and OCR characters that are easily confused.

[0014] As a preferred embodiment of the present invention, the local AI misspelling detection module is implemented based on character features, semantic features, or encoding features.

[0015] As a preferred embodiment of the present invention, after determining that a character is a true misspelling, the corresponding candidate character for correction is further output.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention achieves full-text character consistency verification through the collaborative work of a local preliminary judgment module and a full-text character statistics module. Because it can combine the occurrence patterns of characters throughout the text for judgment, it effectively reduces the false alarm rate caused by local feature anomalies. This invention uses a dynamic adaptive threshold instead of a fixed threshold. Since the threshold can automatically adjust with text length and character distribution, it improves the system's adaptability and accuracy in different scenarios. The various modules of this invention are interconnected and logically coherent. Due to the formation of a three-level recognition mechanism of local preliminary judgment, global verification, and dynamic decision-making, the overall technical effect is significantly better than existing technologies, possessing outstanding substantive features and significant progress. Attached Figure Description

[0017] Figure 1 This is a structural block diagram of the misspelled character recognition system based on artificial intelligence provided in an embodiment of the present invention. Detailed Implementation

[0018] To further understand the invention's content, features, and effects, the following embodiments are provided, and detailed descriptions are given in conjunction with the accompanying drawings.

[0019] The structure of the present invention will now be described in detail with reference to the accompanying drawings.

[0020] like Figure 1 As shown in the figure, an artificial intelligence-based misspelling character recognition system provided by this invention includes: a text input and preprocessing module, a local AI misspelling preliminary judgment module, a full-text character retrieval and statistics module, a misspelling comprehensive probability calculation module, a dynamic adaptive threshold calculation module, and a decision output module. The text input and preprocessing module is used to acquire the text to be detected and complete text cleaning, format unification, and character encoding. The local AI misspelling preliminary judgment module is used to perform preliminary recognition of a single character or a local character fragment in the text and output the local misspelling probability. The full-text character retrieval and statistics module is used to retrieve and count the total number of occurrences of characters that are the same as or similar to the suspected misspelling in the full text, as well as the total number of characters in the full text, when the local preliminary judgment determines that it is a suspected misspelling. The misspelling comprehensive probability calculation module is used to fuse the local misspelling probability and the full-text character distribution features to calculate the comprehensive misspelling probability. The dynamic adaptive threshold calculation module is used to adaptively calculate the dynamic decision threshold based on the full text length and the average frequency of characters in the full text. The decision output module is used to compare the comprehensive misspelling probability with the dynamic decision threshold and output the final misspelling judgment result.

[0021] The text input and preprocessing module is the entry point for the entire system. Since raw text may contain issues such as formatting problems, garbled symbols, and inconsistent encoding, this module performs standardization to provide a stable data foundation for subsequent modules. The local AI-based initial misspelling module performs preliminary screening. Because it doesn't require global statistics on all characters in the text, it can quickly locate suspected misspellings through local initial judgment, improving overall system efficiency. The full-text character retrieval and statistics module only activates after a suspected misspelling appears. Because this module can obtain the patterns of character occurrences throughout the text, it provides a global basis for determining whether a character is incorrectly written, effectively compensating for the limitations of local judgments. The misspelling comprehensive probability calculation module is the core unit for accurate judgment. Since this module integrates local initial judgment results with global distribution features, the resulting comprehensive probability is more reliable than a single local probability. The dynamic adaptive threshold calculation module adjusts the judgment criteria according to the actual text conditions. Since text length and character density directly affect the judgment difficulty, dynamic thresholds can adapt to different scenarios, avoiding misjudgments and missed detections caused by fixed thresholds. The decision output module performs the final judgment based on the calculation results of the preceding module. Since the comprehensive probability and dynamic threshold are both generated by data, the decision result has objectivity and stability.

[0022] Through the collaborative work of the six modules mentioned above, this invention can effectively solve the technical problems raised in the background section. Because existing technologies lack global verification, they are prone to misclassifying universal characters as typos. This invention, however, achieves global consistency verification through a full-text character retrieval and statistics module, thus significantly reducing the false alarm rate. Furthermore, existing technologies use fixed thresholds, which cannot adapt to different texts. This invention, through a dynamic adaptive threshold calculation module, generates adaptability thresholds, thereby improving the recognition accuracy of texts of different lengths. Finally, because existing technologies have independent modules without linkage, while the modules in this invention are interconnected and logically coherent, the overall recognition effect is superior to the simple summation of existing technologies.

[0023] The misspelling probability calculation module executes the following calculation formula: (1) Where P is the overall probability of misspellings, `local` represents the local error probability output by the local AI error detection module, `Nsame` represents the total number of occurrences of characters that are the same as or similar to the current character in the entire text, and `Ntotal` represents the total number of characters in the entire text. To prevent the smoothing coefficient from being zero in the denominator, These are the weighting coefficients for local and global features. .

[0024] This formula weights and fuses the probability of local misspellings with the character distribution characteristics of the entire text to obtain a more comprehensive judgment criterion that better reflects reality. Since the probability of local misspellings only reflects the abnormality of a single character and cannot reflect the usage habits of the entire text, it is necessary to incorporate the frequency of occurrence of characters throughout the text for correction. When the same or similar characters appear frequently throughout the text, it indicates that the character is a common character in the text, and therefore its probability of being a misspelling will decrease accordingly. When the same or similar characters appear very infrequently throughout the text, it indicates that the character does not conform to the overall usage habits of the text, and therefore its probability of being a misspelling will increase accordingly. The smoothing coefficient in the formula is used to avoid the denominator being zero, and the weighting coefficient is used to balance the contribution ratio of local features and global features, making the calculation results more reasonable.

[0025] Furthermore, the dynamic adaptive threshold calculation module executes the following calculation formula: (2) Where T is the final dynamic decision threshold, and T0 is the basic decision threshold. For balance coefficient, Here, Ltext is the normalization function, favg is the total text length, and favg is the average frequency of characters in the text. This formula generates an adaptive decision threshold based on the text length and average character frequency. Since longer texts have more stable character distributions and a lower probability of anomalous characters, the decision threshold needs to be increased to reduce false positives. Conversely, shorter texts have sparser character distributions and may have fewer occurrences of normal characters, thus the decision threshold needs to be decreased to avoid missed detections. The normalization function converts the text length and character frequency into calculable values, and the balancing coefficient adjusts the ratio of the base threshold to the adaptive component, ensuring the threshold remains stable while maintaining flexibility.

[0026] When the overall probability P is greater than the dynamic threshold T, the current character is determined to be a true misspelling; when the overall probability P is less than or equal to the dynamic threshold T, the current character is determined to be a normal character. This defined decision rule is the basis for the final output conclusion of this invention. This rule is implemented based on data comparison without human intervention. Since the overall probability represents the true likelihood of a character being a misspelling, and the dynamic threshold represents the judgment standard under the current text, when the overall probability is greater than the dynamic threshold, the character can be determined to be outside the normal usage range and is a true misspelling; when the overall probability is less than or equal to the dynamic threshold, the character can be determined to conform to the usage rules of the entire text and is a normal character. This decision rule has clear logic, strong reproducibility, and can ensure stable and consistent recognition results.

[0027] Furthermore, an AI-based misspelling character recognition method is applied to the AI-based misspelling character recognition system, comprising the following steps: S1, acquiring the text to be detected and completing text preprocessing through a text input and preprocessing module; S2, performing preliminary judgment on each character of the text through a local AI misspelling initial judgment module to obtain a local misspelling probability Plocal; S3, when Plocal reaches the suspected misspelling condition, triggering a full-text character retrieval and statistics module to count N_same and Ntotal; S4, obtaining a misspelling comprehensive probability P through a misspelling comprehensive probability calculation module; S5, obtaining a dynamic judgment threshold T through a dynamic adaptive threshold calculation module; S6, comparing P and T, and outputting the final judgment result of whether it is a misspelling.

[0028] Further, the statistics of the same or similar characters in the full text is used to implement global consistency check and suppress misjudgment caused by local features. This statistical process is not a simple counting, but based on the character index generated by the preprocessing module, it quickly retrieves all characters similar in glyph and pronunciation to the suspected wrong character in the full text. By counting their occurrence frequency, it reflects the usage rule of the character in the full text, and then corrects the one-sidedness of the initial local judgment -- it avoids misjudging the correct character uniformly and standardly used in the full text as a wrong character due to abnormal local character features (such as illegible font and blurred OCR recognition), which corresponds to the "false positive elimination" scenario mentioned above, and further improves the recognition accuracy.

[0029] The dynamic decision threshold T adaptively increases as the text length increases, and adaptively decreases as the character distribution becomes sparser. This feature is implemented by the dynamic adaptive threshold calculation module, and the core logic is: the longer the text length is, the more stable the character distribution is, and the lower the probability of abnormal characters occurring. Increasing the threshold can reduce unnecessary false positives (for example, the threshold for long text in the foregoing is increased to 0.80 to avoid misjudgment of standardized high-frequency characters); the shorter the text length is, the sparser the character distribution is, and normal characters may also occur fewer times. Reducing the threshold can avoid missed detection (for example, the threshold for short text in the foregoing is reduced to 0.48 to ensure that low-frequency wrong characters are accurately recognized), so as to achieve adaptive adaptation under different scenarios.

[0030] Similar characters include glyph-similar characters, pronunciation-similar characters and OCR easily confused characters. Among them, glyph-similar characters refer to characters with similar glyph structure (e.g., "己(jǐ), 已(yǐ), 巳(sì)", "辩(biàn, argue), 辨(biàn, distinguish), 辫(biàn, braid)"), pronunciation-similar characters refer to characters with the same or similar pronunciation (e.g., "的(de), 得(de), 地(de)", "辩(biàn), 辨(biàn)"), and OCR easily confused characters refer to characters that are easily misjudged in the optical recognition process (e.g., "0 and O", "1 and l", "6 and 9"). This definition clarifies the retrieval range of the full-text character retrieval and statistics module, ensures that the global check can cover the most common wrong character types in practical applications, improves the universality and practicability of the method, and echoes the function of the system modules described above.

[0031] The local AI initial wrong character judgment module is implemented based on character features, semantic features or coding features. Character features mainly refer to visual features such as strokes, radicals and structures of characters; semantic features mainly refer to the collocation rationality of characters in the context; coding features mainly refer to the Unicode encoding of characters and the coding association of glyph-similar / pronunciation-similar characters. The module can adaptively select a single feature or a multi-feature fusion method for initial judgment according to the type of text to be detected (such as printed text, handwritten text and OCR text), which not only ensures the efficiency of initial judgment, but also ensures the accuracy of initial judgment, and provides reliable basic data for subsequent global check and comprehensive probability calculation.

[0032] After a character is identified as a genuine misspelling, the system further outputs the corresponding candidate characters for correction. This function is implemented by the decision output module and the confusion set matching logic working together. After a character is identified as a genuine misspelling, the system will select 3-5 candidate characters for correction that best match the misspelling based on the confusion sets of similar shapes and sounds, combined with the semantic context of the entire text. The characters will be prioritized according to the sorting logic (Score(c_i)) in Formula 2 above, prioritizing the candidate characters with the highest semantic matching degree and the highest frequency of use that fits the entire text. This eliminates the need for manual retrieval and correction, improves the automation level of text proofreading, and expands the practical application scenarios of the method (such as official document review and educational correction).

[0033] Specifically: Examples of true typos are as follows: The total number of characters in the full text is Ntotal=1000, the probability of local misspelling is Plocal=0.70, the number of times the same / similar characters appear is N_same=2, β=0.6, and ε=1.

[0034] Calculate the overall probability: P = 0.6 × 0.70 + 0.4 × (1 - 2 / 1001) ≈ 0.8192 Dynamic threshold: T0=0.5, γ=0.7, σ=0.65, T=0.7×0.5+0.3×0.65=0.545 Since 0.8192 > 0.545, it is determined to be a genuine typo.

[0035] Examples of normal characters (excluding false positives) are as follows: Plocal=0.65, N_same=80, Ntotal=1000P≈0.853, dynamic threshold T=0.86. Since 0.853<0.86, it is determined to be a normal character.

[0036] Here is an example of preventing missed detections with short text: With Ntotal=50, Plocal=0.55, and N_same=1, the dynamic threshold adaptively decreases to T=0.48, P≈0.73>0.48, thus it is determined to be a true typo.

[0037] Here is an example of preventing false alarms for long texts: With Ntotal=3000, Plocal=0.62, and N_same=120, the threshold for long texts is adaptively increased to T=0.80. Since 0.76 < 0.80, it is determined to be a normal character.

[0038] Working principle: After input and preprocessing, the text enters the local AI initial misspelling detection stage. When the local AI initial misspelling detection module identifies a suspected misspelling, the system triggers the full-text character retrieval and statistics module. This module counts the frequency of identical and similar characters and the total number of characters in the text. Since this data reflects the usage patterns of characters throughout the text, it provides a basis for subsequent comprehensive judgment. The misspelling probability calculation module integrates the local misspelling probability with the full-text character distribution characteristics to calculate a comprehensive probability that accurately reflects the likelihood of a misspelling. The dynamic adaptive threshold calculation module generates a dynamic judgment threshold based on text length and character frequency. Because the dynamic threshold can adapt to different text situations, it avoids the judgment bias caused by a fixed threshold. The judgment output module compares the comprehensive probability with the dynamic threshold and outputs a final judgment conclusion on whether the text is a misspelling based on the comparison result.

[0039] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0040] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A misspelled character recognition system based on artificial intelligence, characterized in that, include: The system comprises a text input and preprocessing module, a local AI-based initial misspelling module, a full-text character retrieval and statistics module, a comprehensive misspelling probability calculation module, a dynamic adaptive threshold calculation module, and a decision output module. The text input and preprocessing module acquires the text to be detected and performs text cleaning, format standardization, and character encoding. The local AI-based initial misspelling module performs preliminary identification of individual characters or partial character fragments in the text and outputs the local misspelling probability. The full-text character retrieval and statistics module retrieves and counts the total number of occurrences of characters identical or similar to the suspected misspelling in the full text, as well as the total number of characters in the full text, when the local initial misspelling is determined to be a suspected misspelling. The comprehensive misspelling probability calculation module fuses the local misspelling probability with the full-text character distribution characteristics to calculate the comprehensive misspelling probability. The dynamic adaptive threshold calculation module adaptively calculates a dynamic decision threshold based on the full text length and the average frequency of characters in the full text. The decision output module compares the comprehensive misspelling probability with the dynamic decision threshold and outputs the final misspelling judgment result.

2. The misspelled character recognition system based on artificial intelligence as described in claim 1, characterized in that: The misspelling probability calculation module executes the following calculation formula: Where P is the overall probability of misspellings. `local` represents the local error probability output by the local AI error detection module, `Nsame` represents the total number of occurrences of characters that are the same as or similar to the current character in the entire text, and `Ntotal` represents the total number of characters in the entire text. To prevent the smoothing coefficient from being zero in the denominator, These are the weighting coefficients for local and global features. .

3. The misspelled character recognition system based on artificial intelligence as described in claim 1, characterized in that: The dynamic adaptive threshold calculation module performs the following calculation formula: Where T is the final dynamic decision threshold, and T0 is the basic decision threshold. For balance coefficient, Here, Ltext is the normalization function, favg is the total text length, and favg is the average frequency of occurrence of characters in the total text.

4. The system according to claim 3, characterized in that, The decision output module executes the following decision rules: when the overall probability P is greater than the dynamic threshold T, the current character is determined to be a true misspelling; when the overall probability P is less than or equal to the dynamic threshold T, the current character is determined to be a normal character.

5. A method for recognizing misspelled characters based on artificial intelligence, applied to the system described in any one of claims 14, characterized in that, The process includes the following steps: S1. Obtain the text to be detected and preprocess it using the text input and preprocessing module; S2. Perform initial character-by-character judgment on the text using the local AI misspelling module to obtain the local misspelling probability Plocal; S3. When Plocal reaches the suspected misspelling condition, trigger the full-text character retrieval and statistics module to calculate N_same and Ntotal; S4. Obtain the comprehensive misspelling probability P using the misspelling comprehensive probability calculation module; S5. Obtain the dynamic judgment threshold T using the dynamic adaptive threshold calculation module; S6. Compare P and T and output the final judgment result on whether it is a misspelling.

6. The method according to claim 5, characterized in that, The statistics of identical or similar characters throughout the text are used to achieve global consistency verification and suppress misjudgments caused by local features.

7. The method according to claim 5, characterized in that the dynamic decision threshold T adaptively increases with the increase of text length and adaptively decreases with the sparse distribution of characters.

8. The method according to claim 5, characterized in that, Similar characters include characters with similar shapes, characters with similar sounds, and characters that are easily confused in OCR.

9. The method according to claim 5, characterized in that, The local AI-based initial misspelling module is implemented based on character features, semantic features, or encoding features.

10. The method according to claim 5, characterized in that, After determining that it is a genuine typo, the corresponding correction candidate character is further output.

Images