Wrongly written character recognition method, device and equipment and computer readable storage medium

Abstract

The invention discloses a wrongly written character recognition method, device and equipment and a computer readable storage medium. The method comprises the steps of collecting a handwritten text image; the handwritten text image is detected and recognized through the text detection model and the text recognition model, the confidence degree corresponding to each character included in the handwritten text image is determined, and according to the confidence degree corresponding to each character, high-credibility characters are screened out from the characters included in the handwritten text image; classifying the single character images corresponding to the high-credibility characters through an image-text comparison learning model so as to screen out visual error characters from the high-credibility characters; executing a semantic correction path corresponding to the language type of the visual error character to determine an optimal correction word corresponding to the visual error character; and displaying the position of the visual error character and the optimal correction word corresponding to the visual error character on the interactive interface. Therefore, the problems of false alarm and missing detection are solved by screening high-credibility characters and then performing visual classification, and then the accuracy is improved through a language-based error correction path.

Description

Technical Field

[0001] This application relates to the technical field of image recognition and processing, and more specifically, to a method, apparatus, device, and computer-readable storage medium for identifying misspelled words. Background Technology

[0002] With the rapid development of digital education, smart office, and human-computer interaction technologies, the demand for automatic recognition and error correction of handwritten text is increasing. Among related technologies, the detection of typos in handwritten text mainly relies on OCR (Optical Character Recognition) technology, which first converts the handwritten image into text and then checks whether the text contains errors.

[0003] However, the system judges errors based solely on the identified text and cannot utilize the visual features of the original handwriting; when a character is illegible, existing methods often directly treat it as a misspelling; moreover, the use of a unified correction framework for misspellings in different languages ​​results in low accuracy in misspelling detection. Summary of the Invention

[0004] In view of the above problems, this application proposes a method, apparatus, device and computer-readable storage medium for identifying misspelled words, which can solve the above problems.

[0005] In a first aspect, embodiments of this application provide a method for identifying misspelled characters. This method includes: acquiring a handwritten text image; detecting and recognizing the handwritten text image using a text detection model and a text recognition model, determining the confidence level of each character in the handwritten text image, and filtering out highly reliable characters from the characters in the handwritten text image based on the confidence level of each character; classifying the single-character images corresponding to the highly reliable characters using a text-image comparison learning model to filter out visually misspelled characters from the highly reliable characters; executing a semantic correction path corresponding to the language type of the visually misspelled character to determine the optimal correction word corresponding to the visually misspelled character; and displaying the position of the visually misspelled character and its corresponding optimal correction word on an interactive interface.

[0006] Secondly, embodiments of this application also provide a misspelling recognition device, which includes: a collection module for collecting handwritten text images; a first execution module for detecting and recognizing the handwritten text images using a text detection model and a text recognition model, determining the confidence level corresponding to each character in the handwritten text image, and filtering out high-confidence characters from the characters in the handwritten text image based on the confidence level corresponding to each character; a second execution module for classifying the single-character images corresponding to the high-confidence characters using a text-image comparison learning model, so as to filter out visually misspelled characters from the high-confidence characters; a third execution module for executing the semantic correction path corresponding to the language type of the visually misspelled character, so as to determine the optimal correction word corresponding to the visually misspelled character; and a display module for displaying the position of the visually misspelled character and its corresponding optimal correction word on an interactive interface.

[0007] Thirdly, embodiments of this application also provide a typo recognition device, including a processor, a memory, and one or more application programs; the one or more application programs are stored in the memory and configured to be executed by the processor to implement the above-described typo recognition method.

[0008] Fourthly, embodiments of this application also provide a computer-readable storage medium storing program code, wherein the above-mentioned misspelling recognition method is executed when the program code is run by a processor.

[0009] The technical solution provided in this application includes the following method: acquiring handwritten text images; detecting and recognizing the handwritten text images using a text detection model and a text recognition model, determining the confidence level of each character in the handwritten text image, and filtering out high-confidence characters from the characters in the handwritten text image based on the confidence level of each character; classifying the single-character images corresponding to the high-confidence characters using a text-image comparison learning model to filter out visually incorrect characters from the high-confidence characters; executing the semantic correction path corresponding to the language type of the visually incorrect character to determine the optimal correction word corresponding to the visually incorrect character; and displaying the position of the visually incorrect character and its corresponding optimal correction word on an interactive interface. Therefore, by using text detection and recognition models to filter out high-confidence characters, the misclassification of low-confidence characters as misspellings is avoided, effectively reducing false positives caused by blurred handwriting. Secondly, image-text comparison learning is used to classify the single-character images corresponding to these high-confidence characters, overcoming the missed detection problem caused by traditional methods relying solely on text sequences and failing to utilize image features. Next, corresponding semantic correction paths are executed for different language characteristics, improving the accuracy of error correction. Finally, by directly displaying the location of visually misspelled characters and their optimal correction words on the interactive interface, precise error location and intuitive presentation are achieved, meeting users' needs for interactive correction. This approach aims to improve the accuracy of misspelling detection and enhance the user experience. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments and drawings obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0011] Figure 1 A flowchart illustrating a misspelling recognition method provided in an embodiment of this application is shown.

[0012] Figure 2 A schematic diagram of a misspelling recognition device provided in an embodiment of this application is shown.

[0013] Figure 3 This is a schematic diagram of the structure of a misspelling recognition device provided in an embodiment of this application.

[0014] Figure 4 This illustration shows a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0016] With the rapid development of digital education, smart office, and human-computer interaction technologies, the demand for automatic recognition and error correction of handwritten text is increasing. Among related technologies, the detection of typos in handwritten text mainly relies on OCR (Optical Character Recognition) technology, which first converts the handwritten image into text and then checks whether the text contains errors.

[0017] However, the system judges errors based solely on the identified text and cannot utilize the visual features of the original handwriting; when a character is illegible, existing methods often directly treat it as a misspelling; moreover, the use of a unified correction framework for misspellings in different languages ​​results in low accuracy in misspelling detection.

[0018] To address the aforementioned issues, this application provides a method, apparatus, device, and computer-readable storage medium for identifying misspelled characters. The method includes: acquiring a handwritten text image; detecting and recognizing the handwritten text image using a text detection model and a text recognition model, determining the confidence level of each character in the handwritten text image, and filtering out high-confidence characters from the characters in the handwritten text image based on the confidence level of each character; classifying the single-character images corresponding to the high-confidence characters using a text-image comparison learning model to filter out visually misspelled characters from the high-confidence characters; executing a semantic correction path corresponding to the language type of the visually misspelled character to determine the optimal correction word corresponding to the visually misspelled character; and displaying the position of the visually misspelled character and its corresponding optimal correction word on an interactive interface.

[0019] Therefore, by using text detection and recognition models to filter out high-confidence characters, the misclassification of low-confidence characters as misspellings is avoided, effectively reducing false positives caused by blurred handwriting. Secondly, image-text comparison learning is used to classify the single-character images corresponding to these high-confidence characters, overcoming the missed detection problem caused by traditional methods relying solely on text sequences and failing to utilize image features. Next, corresponding semantic correction paths are executed for different language characteristics, improving the accuracy of error correction. Finally, by directly displaying the location of visually misspelled characters and their optimal correction words on the interactive interface, precise error location and intuitive presentation are achieved, meeting users' needs for interactive correction. This approach aims to improve the accuracy of misspelling detection and enhance the user experience.

[0020] Please see Figure 1 , Figure 1 A flowchart illustrating a misspelling recognition method provided in an embodiment of this application is shown. Figure 1 As shown, the method may include steps 110 to 150.

[0021] In step 110, the handwritten text image is acquired.

[0022] In some implementations, the handwritten text is placed in a designated area, and an image acquisition device is used to capture an image of the handwritten text within the designated area, so as to obtain a clear, unobstructed image of the handwritten text. The image acquisition device can be a camera or a scanner.

[0023] In some implementations, users can also directly transmit handwritten text images to the system so that the system can complete the acquisition of handwritten text images.

[0024] In some implementations, the handwritten text can be student assignments, exam papers, class notes, handwritten letters, or reports, etc.

[0025] Based on the collected handwritten text images, the system detects and recognizes the handwritten text images to filter out typos. Specifically:

[0026] In step 120, the handwritten text image is detected and recognized by the text detection model and the text recognition model. The confidence level of each character in the handwritten text image is determined, and high-confidence characters are selected from the characters in the handwritten text image based on the confidence level of each character.

[0027] In some implementations, the text detection model employs DBNet (Differentiable Binarization Network). This text detection model belongs to the deep learning-based scene text detection technology. Its core task is to accurately locate all regions containing text in a complex handwritten text image and output the spatial coordinates of these regions as axis-aligned or rotated bounding boxes. In other words, the text detection model determines the coordinates corresponding to the regions containing text in a handwritten text image.

[0028] In some implementations, the text recognition model employs the STVRv2 model. This text recognition model belongs to the deep learning-based sequence-to-sequence text recognition technology. Its core task is to decode the single-line handwritten text sub-images cropped by the text detection model, converting their visual information into readable character sequences.

[0029] In some implementations, high-confidence characters are potential misspellings, which are the primary candidate set for misspelling analysis.

[0030] The handwritten text image is detected and recognized using a text detection model and a text recognition model to filter out characters that may be misspelled from the handwritten text image. Specifically, in some implementations, the step "detecting and recognizing the handwritten text image using a text detection model and a text recognition model to filter out highly reliable characters from the characters included in the handwritten text image" may include the following steps:

[0031] (1) Use a text detection model to detect text lines in a handwritten text image and determine the bounding box coordinates of each text line in the handwritten text image.

[0032] (2) Based on the bounding box coordinates of each text line, crop out the corresponding text line sub-image from the handwritten text image;

[0033] (3) Use a text recognition model to identify the characters in the text line subgraph and determine the confidence level of each character in the text line subgraph.

[0034] (4) Based on the confidence level of each character in the text line subgraph and the first preset threshold, select high-confidence characters from the characters included in the text line subgraph.

[0035] The text detection model feeds the handwritten text image into a backbone network, which extracts multi-level visual feature maps of the handwritten text image through multiple convolutional operations. These feature maps contain both local details (such as strokes) and global semantic information.

[0036] The text detection model generates a probability map (or threshold map) based on multi-level visual feature maps. Each pixel value on this probability map represents the probability that its corresponding location belongs to a text region. The text detection model innovatively introduces a differentiable approximation function, combining this probability map with an adaptive threshold map to smoothly generate the final probability map (binary map). This process is differentiable, allowing the entire network to backpropagate gradients directly from the binarized result, thus achieving more efficient and accurate end-to-end training.

[0037] The text detection model then uses post-processing algorithms (such as DB post-processing) to aggregate connected components based on the generated binarized graph, and outputs a rectangular bounding box that tightly surrounds each line of handwritten text. Its coordinate format is (xmin, ymin, xmax, ymax), where (xmin, ymin) are the coordinates of the top left corner and (xmax, ymax) are the coordinates of the bottom right corner.

[0038] The text detection model outputs one or more sets of bounding box coordinates, each set representing the position of a detected line of text in the image. Based on these coordinates, individual text line sub-images can be cropped from the original handwritten text image.

[0039] After cropping the corresponding text line sub-image from the handwritten text image based on the bounding box coordinates of each text line, the text line sub-image is input into a text recognition model. The text recognition model feeds the text line sub-image into a convolutional neural network (CNN) as an encoder. This encoder transforms the two-dimensional image data into a series of one-dimensional, high-dimensional feature vector sequences through multiple convolutional operations. These feature vectors are arranged from left to right, and each vector encodes the visual features of the corresponding image region (e.g., a character or part of a character).

[0040] Next, the text recognition model inputs the above-mentioned feature vector sequence into a sequence modeling module. This module is composed of a recurrent neural network (RNN), such as a bidirectional long short-term memory network (BiLSTM). Its role is to capture the context dependencies within the character sequence and understand the order and semantic associations between characters.

[0041] Finally, the text recognition model introduces a decoder with an attention mechanism. At each decoding time step, the decoder dynamically focuses on the most relevant part of the encoder feature sequence according to the previously recognized characters and the current context information through the attention mechanism. The decoder also predicts the next most likely character based on this focused information. The text recognition model finally outputs a probability distribution covering the entire predefined character set (such as Chinese characters, English letters, numbers, punctuation marks, etc.). In this probability distribution, each character corresponds to a probability, indicating the confidence level that the text recognition model believes the character should be output at the current time step.

[0042] The text recognition model selects the character with the highest probability according to the probability distribution output at each step of the decoder and finally concatenates them into a complete recognized string. At the same time, for each recognized character, the corresponding highest probability value is recorded as the confidence level of the character. For example, if the text recognition model recognizes the character "学" and its probability is 0.98, then the confidence level of this character is 0.98.

[0043] In some embodiments, the step of "screening out highly reliable characters from the characters included in the text line sub-graph according to the confidence level corresponding to each character in the text line sub-graph and a first preset threshold" may include: in the text line sub-graph, determining the characters corresponding to the confidence level greater than the first preset threshold as highly reliable characters.

[0044] In some embodiments, the step of "screening out highly reliable characters from the characters included in the text line sub-graph according to the confidence level corresponding to each character in the text line sub-graph and a first preset threshold" may include: in the text line sub-graph, determining the characters corresponding to the confidence level less than or equal to the first preset threshold as lowly reliable characters.

[0045] In a specific embodiment, the value of the first preset threshold can be 0.95. It can be understood that the present application does not limit the specific value of the first preset threshold.

[0046] Low-confidence characters are those that are illegible, have scribbled parts, have been erased, or are severely distorted, making it difficult for the model to identify them with high confidence. The module filters out these low-confidence characters and presents them for human judgment to determine if they are typos. For example, the model can directly mark characters with a confidence level of 0.95 or less as "to be determined" (i.e., low-confidence characters).

[0047] By excluding low-confidence characters from the screening process and having them judged manually, the accuracy of typo detection can be effectively improved. Specifically:

[0048] In step 130, the single-character images corresponding to high-confidence characters are classified using a text-image comparison learning model in order to filter out visually incorrect characters from the high-confidence characters.

[0049] In some implementations, the image-text contrastive learning model specifically refers to a multimodal deep neural network based on a contrastive learning framework. Its core function is to learn joint embedding representations of image and text data from two different modalities, and to quantify the semantic similarity between any given image and text description. In the implementation of this application, the image-text contrastive learning model uses a pre-trained CLIP (Contrastive Language-Image Pre-training) model as its infrastructure.

[0050] The image-text comparison learning model comprises two main encoders: an image encoder and a text encoder. The image encoder is a Visual Transformer (ViT) or ResNet, responsible for encoding the input single-word image into a high-dimensional vector (i.e., image embedding I). The text encoder is a Transformer, responsible for encoding the input text prompt into another high-dimensional vector (i.e., text embedding T). The core mechanism of the image-text comparison learning model is to calculate the cosine similarity between image embedding I and text embedding T; the higher the value, the better the match between the image and the text description.

[0051] The image-text comparison learning model used in this application is a supervised fine-tuning model that transforms the original multimodal retrieval task into a binary classification task, namely, determining whether a handwritten character image belongs to the first category or the second category.

[0052] The dataset used in the image-text contrast learning model of this application includes positive samples and negative samples. Among them, for positive samples, a large number of images of standard Chinese characters or English words with regular and clear writing are collected, and the text description "photo of a correct character" is paired with them; for negative samples, images of artificial or commonly seen misspelled characters similar in form (such as writing "未" as "末", "then" as "thne") are collected, and the text description "photo of a wrong character" is paired with them.

[0053] The single-character image to be judged is uniformly normalized to a fixed size (for example, 64x64 pixels), and then input into the fine-tuned image encoder. The image-text contrast learning model will calculate the similarity scores between the single-character image and positive samples and negative samples respectively. Through the Softmax function, the probability that the highly credible character belongs to the first category can be obtained, and visual misspelled characters (referring to those handwritten characters with common writing mistakes) can be screened out from the highly credible characters. Specifically, in some embodiments, the step of "classifying the single-character image corresponding to the highly credible character through the image-text contrast learning model to screen out visual misspelled characters from the highly credible characters" may include the following steps:

[0054] (1) Obtain the single-character image of the highly credible character in the handwritten text image;

[0055] (2) Input the single-character image into the image-text contrast learning model and calculate the visual probability that the single-character image belongs to the first category; the text encoder of the image-text contrast learning model is frozen, and the image encoder is fine-tuned to classify the single-character image into the first category or the second category

[0056] (3) Screen out visual misspelled characters from the highly credible characters according to the visual probability and the second preset threshold.

[0057] As can be seen from the above description, through the text detection model, the bounding box coordinates of the entire text line in the handwritten text image can be obtained, and then according to the bounding box coordinates, the entire text line can be cropped out in the handwritten text image to obtain the text line sub-image; during the process of the text recognition model processing the text line sub-image, its internal architecture (usually through the attention mechanism or feature map alignment) can know which region in the text line sub-image each recognized character corresponds to. Therefore, the text recognition model can output the local bounding box coordinates of each highly credible character. Then, the local bounding box coordinates are superimposed and calculated with the global bounding box coordinates of the corresponding text line in the handwritten text image to obtain the global character bounding box coordinates of the highly credible character in the handwritten text image.

[0058] Finally, based on the global character bounding box coordinates, the corresponding single-character image region is cropped from the handwritten text image and its size is normalized to obtain the standard single-character image for visual judgment, thus completing the acquisition of the single-character image of each highly reliable character in the handwritten text image.

[0059] The acquired single-character images are uniformly normalized to a fixed size (e.g., 64x64 pixels) and then input into the image-text contrast learning model. This application employs a fine-tuning strategy for the image-text contrast learning model: freezing the text encoder and only fine-tuning the image encoder. That is, during training, the weights of the text encoder remain unchanged, while only the parameters of the image encoder are updated. This significantly reduces the computational resources and GPU memory consumption required for training, while preserving the powerful general language understanding capabilities of the image-text contrast learning model. Through learning, the image encoder is able to better extract visual features related to the "first category / second category" discrimination.

[0060] In one specific implementation, the first category is erroneous characters (referring to handwritten characters with common writing errors), and the second category is correct characters (referring to handwritten characters whose writing conforms to standard specifications). A visual probability assessment model is used to determine the visual probability of a single character image being an erroneous character. Then, based on the specific value of the visual probability, handwritten characters with common writing errors are filtered out from the high-confidence characters.

[0061] In some implementations, the step "filtering out visually incorrect characters from highly reliable characters based on visual probability and a second preset threshold" may include: identifying highly reliable characters with visual probabilities greater than the second preset threshold as visually incorrect characters.

[0062] In some implementations, the step "filtering out visually incorrect characters from high-confidence characters based on visual probability and a second preset threshold" may include: determining high-confidence characters whose visual probability is less than or equal to the second preset threshold as correct characters.

[0063] In some implementations, the second preset threshold can be 0.5. It is understood that this application does not limit the specific value of the second preset threshold.

[0064] Therefore, by cooperating with the text detection model and the text recognition model, the system filters out the characters that it can identify from the handwritten text image. Then, the image-text comparison learning model filters out erroneous characters from the high-confidence characters, and finally corrects the filtered erroneous characters. Specifically:

[0065] In step 140, the semantic correction path corresponding to the language type of the visual error character is executed to determine the optimal correction word corresponding to the visual error character.

[0066] In some embodiments, the language types include a first language and a second language. Among them, the first language is Chinese. That is, when the visually misspelled character is Chinese, the visually misspelled character is corrected through the semantic correction path corresponding to Chinese set in the system to determine its corresponding optimal correction word. The second language is a non-Chinese language (e.g., English). That is, when the visually misspelled character is English, the visually misspelled character is corrected through the semantic correction path corresponding to English set in the system to determine its corresponding optimal correction word.

[0067] In some embodiments, the optimal correction word refers to the correct word that the system believes is most likely to be the user's intention, which is finally output by executing the semantic correction path corresponding to the language type of the target character determined as a "visually misspelled character". For example, the handwritten text image includes "Today the weather is so nice!". Through the above embodiment, "汽" is determined as a visually misspelled character, and then through the semantic correction path corresponding to Chinese set in the system, the optimal correction word corresponding to "汽" is determined to be "气". That is to say, the user's intention should be "Today the weather is so nice!". By setting different independent semantic correction paths according to the language type of the visually misspelled character, it is because there are essential differences in the writing system, common error types, and linguistic rules among different languages. Using a unified processing flow will lead to a significant decrease in the error correction accuracy. Through the customized dual-path design of this application, the error correction features of each language can be captured more accurately, thereby greatly improving the overall performance of the system. Specifically, in some embodiments, the step of "executing the semantic correction path corresponding to the language type of the visually misspelled character to determine the optimal correction word corresponding to the visually misspelled character" may include the following steps:

[0068] (1) If the language type is the first language, extract the context text composed of a preset number of characters before and after the visually misspelled character centered on the visually misspelled character;

[0069] (2) Based on the pinyin of the visually misspelled character, recall K first candidate correction words with the same pinyin or similar initials and finals from the preset million-level homophonic and near-homophonic word library;

[0070] (3) Input the context text and the K first candidate correction words into the MacBERT language model, and calculate the cosine similarity between the semantic representation vector of each first candidate correction word after being substituted into the context text and the overall semantic vector of the context; The embedding layer of the MacBERT model integrates pinyin embedding information;

[0071] (4) Select the first candidate correction word corresponding to the highest cosine similarity as the optimal correction word, and determine the semantic confidence of the optimal correction word.

[0072] In some embodiments, the value of the preset quantity can be 5. Chinese semantic expressions usually rely on words or phrases consisting of 2 to 4 characters. For example, in the sentence "The weather is great today", "today", "weather", and "is great" are all independent semantic units. When a misspelled character appears in a certain word (such as writing "weather" as "天汽"), the system needs sufficient context to identify the words associated with it before and after, so as to understand the overall sentence meaning. That is, taking a range of 5 characters is sufficient to cover the complete structure of the word where the misspelled character is located and include 1 to 2 adjacent words before and after it, while reducing the situation where misleading semantic information may be introduced due to longer context.

[0073] In some embodiments, the preset million-level homophonic and near-homophonic word library refers to a large data structure dedicated to the candidate word recall stage of Chinese misspelled characters, which contains a large number of Chinese characters and their homophonic and near-homophonic relationship mappings. The design goal of the preset million-level homophonic and near-homophonic word library is to ensure that for any handwritten Chinese character input, the system can efficiently recall all possible correct candidate words that may be miswritten due to "near-homophony", thus ensuring a high recall rate for the error correction process.

[0074] This application generates and maintains the preset million-level homophonic and near-homophonic word library through the strategy of "combining static and dynamic, hierarchical construction". The preset million-level homophonic and near-homophonic word library integrates multiple authoritative and wide-coverage public corpora to build a static base table. For example, Wikipedia Chinese corpus, SCOWL, and professional term libraries in various fields, etc. Then, data cleaning and standardization are performed on the static base table (such as preprocessing operations such as deduplication, traditional-simplified conversion, and normalization of variant characters for the data from the above sources). And using a mature Chinese pinyin conversion tool, the standard Mandarin pinyin (including initials, finals, and tones) is marked for each entry (single character or word) in the word library, forming a high-quality static word library containing about 600,000 entries as the static base table.

[0075] In order to cope with the evolution of language and the needs of users' personalized expressions, the system continuously collects the actual usage logs of users during online operation, especially those cases of misspelled characters that have been successfully corrected and the new words manually added by users. And on a monthly cycle, new words or new usages with high frequencies that appear in the logs and are not included in the current word library are statistically analyzed. Sampling inspections are carried out on the extracted high-frequency new words, and their rationality, standardization, and popularity are reviewed by language experts or the operation team to prevent the contamination of the word library by vulgar, incorrect, or temporary Internet terms. The new words that pass the review and their pinyin information are regularly appended to the main word library to achieve the adaptive growth of the word library.

[0076] In some embodiments, to support efficient recall queries, the system pre-constructs a reverse index based on a preset homophone and near-homophone dictionary of millions of words, which groups all words with exactly the same pinyin (including tones) into the same group. For example, "formula", "official business", and "working hours" all belong to the "gōng shì" group. It also defines the "near-homophone" relationship according to the rules of Chinese phonology (such as similar initials: z / zh, c / ch, s / sh; similar finals: an / ang, en / eng; similar tones: first tone and second tone, fourth tone and light tone, etc.), and establishes associations for words with similar pronunciations. For example, it establishes near-homophone links between zài (在) and zǎi (载), cái (才).

[0077] Exemplarily, when recalling candidates for a misspelled word (such as "末 mò"), the system can quickly retrieve, through this index, all homophones with the pinyin mò (such as "墨", "默", "陌"), as well as words with similar pronunciations (such as "没 méi / mò", "莫 mò"), to form K first candidate correction words.

[0078] By recalling K first candidate correction words with the same pinyin or similar initials and finals as the visually misspelled characters from a preset homophone and near-homophone dictionary of millions of words, a fast rough ranking method is implemented to provide first candidate correction words with relatively high candidate probabilities for fine ranking in subsequent steps.

[0079] In some embodiments, the MacBERT language model refers to a deep language model based on the BERT (Bidirectional Encoder Representations from Transformers) architecture and optimized and improved for Chinese text correction tasks. The MacBERT model adopted in this application does not directly use its original form, but has undergone crucial structural modifications to better adapt to the application scenario of detecting handwritten misspelled words.

[0080] Since a large number of Chinese misspelled words stem from "near-homophone" errors (such as "在" and "再", "的" and "得"). However, the MacBERT language model only learns from the glyphs and context and cannot explicitly perceive the pronunciation of Chinese characters. To solve this problem, the embedding layer of the MacBERT language model is composed of a glyph embedding and a pinyin embedding spliced together. The glyph embedding is the standard embedding traditionally obtained from a word lookup table, representing the glyph information of Chinese characters. The pinyin embedding is for each Chinese character in the input sequence, with its corresponding Chinese pinyin (such as "zhī", "dào") added as auxiliary input, and it is converted into a pinyin embedding vector through another learnable lookup table. The glyph embedding vector and the pinyin embedding vector are spliced or added together, and then fed into the Transformer encoding layer.

[0081] Exemplarily, when the system needs to determine whether a candidate correction word (e.g., "未" in "未来") is more reasonable than the original misspelled word ("末"), the MacBERT language model can not only analyze the semantic fluency of the word "未来", but also utilize the pinyin differences between "未(wèi)" and "末(mò)" to assist in the judgment. This enables the MacBERT language model to have stronger robustness and higher accuracy when dealing with misspelled words caused by "similar pronunciation".

[0082] Replace the position of the original misspelled word in the context text with the first candidate correction word to generate a new candidate sentence. Send this new candidate sentence as a complete input sequence into the MacBERT language model. The embedding layer of the MacBERT language model integrates pinyin embedding information. For each Chinese character in the input sequence, the model will not only read the standard glyph embedding vector representing its glyph, but also read the pinyin embedding vector mapped by its corresponding Chinese pinyin, and fuse the two (such as concatenation or addition) as the final input representation. The input sequence integrated with pinyin information is then sent into the Transformer encoding layer. The MacBERT language model generates vector representations containing deep context semantics for each character in the sequence through multiple layers of self-attention mechanisms.

[0083] Extract the semantic representation vector of the first candidate correction word substituted in the output of the final encoding layer. This vector represents the complete semantic role of the first candidate correction word in the current context. At the same time, calculate the overall semantic vector of the entire candidate sentence. This is usually achieved by performing mean pooling on the representation vectors of all characters. Finally, calculate the cosine similarity between the semantic representation vector and the overall semantic vector. The value of the cosine similarity measures the consistency and fit between the semantics of the first candidate word and the overall semantics of its sentence. The closer this value is to 1, the higher the consistency.

[0084] Finally, take the first candidate correction word corresponding to the highest cosine similarity as the optimal correction word, and determine the semantic confidence of the optimal correction word (i.e., its corresponding cosine similarity). The semantic confidence indicates how well the optimal candidate word fits the semantics of the entire sentence in the current context.

[0085] In some embodiments, the step of "executing the semantic correction path corresponding to the language type of the visual misspelled character to determine the optimal correction word corresponding to the visual misspelled character" may include the following steps:

[0086] (1) If the language type is a second language, calculate the edit distance between its spelling and each entry in the preset dynamic dictionary based on the spelling of the visual misspelled character, and determine all entries with an edit distance less than or equal to the preset value as the second candidate correction words;

[0087] (2) The rationality of each second candidate correction word in the context text centered on the visual error character is scored by the 5-gram language model, and the second candidate correction word with the lowest confusion is selected as the optimal correction word.

[0088] In some implementations, the preset dynamic dictionary is a 600,000-entry dynamic dictionary built by the system. It is a large-scale vocabulary knowledge base specifically designed for the English misspelling candidate word recall stage. The core objective of the preset dynamic dictionary is to ensure that for any handwritten English misspelling, the system can efficiently recall all possible real English words as candidates through edit distance calculation, thereby guaranteeing a high recall rate in the error correction process.

[0089] In some implementations, the preset dynamic dictionary is stored using efficient data structures such as hash tables or inverted indexes, supporting fast term retrieval with O(1) or O(log n) time complexity. During the candidate recall phase, the system iterates through each term in the dictionary, calculates its edit distance (LevenshteinDistance) from the original misspelled word, and identifies all terms with a distance less than or equal to a preset value as second candidate correction words.

[0090] In some implementations, ablation experiments were conducted on large-scale real handwritten English text datasets to compare the impact of different edit distance thresholds (e.g., 1, 2, 3) on the overall error correction accuracy (F1-Score) of the system. Experimental results show that the system's overall performance reaches its peak when the threshold is 2; therefore, the preset value is set to 2.

[0091] In some implementations, a 5-gram language model refers to a language model based on n-gram statistical methods used to calculate the probability of English text sequences. The core function of a 5-gram language model is to evaluate whether a given English sentence or phrase conforms to the expression habits of native English speakers, thereby providing a basis for decision-making regarding the correction of English spelling errors.

[0092] After the system retrieves a batch of second candidate correction words from the pre-set dynamic dictionary, it needs to determine which second candidate correction word, when substituted into the original sentence, results in the most "natural" sentence. The system will substitute each second candidate correction word sequentially into the context text centered on the original misspelled word to generate a new candidate sentence. The pre-trained 5-gram language model is then used to calculate the perplexity of this new sentence. Perplexity is a standard metric for evaluating the quality of a language model. Essentially, it is the reciprocal of the geometric mean of the model's probabilities for the test set sequences. The lower the perplexity, the more confident the language model is in predicting the sentence, meaning the sentence is more fluent and conforms more to language habits. Therefore, the 5-gram language model scores (calculates perplexity) all candidate sentences separately, and finally selects the candidate correction word corresponding to the candidate sentence with the lowest perplexity as the optimal correction word.

[0093] Thus, it can be seen that by determining the language type to which the visual error belongs, and adopting the corresponding semantic correction path, the optimal correction word corresponding to the visual error can be accurately determined, thereby completing the correction of the visual error.

[0094] In step 150, the visually misspelled character and its corresponding optimal correction word are displayed on the interactive interface.

[0095] To present the complex, multi-layered judgment results from the backend to users intuitively and efficiently, and to support convenient correction operations, the system has designed a complete visual feedback and interaction mechanism. This mechanism ensures the traceability, understandability, and operability of the detection results.

[0096] Once the system completes the full-process analysis of a visual misspelling (including language type determination and optimal correction word generation), it will package all its relevant information in a structured manner. For example, its coordinates, the original misspelled text, the optimal correction word, the language type, and the confidence level will be written into a standardized JSON data packet.

[0097] The interactive interface or a dedicated rendering engine receives and parses the JSON data packet. Based on the language type of the misspelled character, a differentiated visual style is overlaid on the original image. For example, for a visually misspelled character in Chinese, a red rectangular border is drawn over the corresponding misspelled area in the handwritten text image, and a suggested word pop-up window appears nearby, displaying the corresponding optimal correction word and / or semantic confidence. As another example, for a visually misspelled character in English, a blue rectangular border is drawn, and the corresponding suggested word pop-up window is displayed.

[0098] In some implementations, each floating window provides explicit interactive controls, such as "Accept" and "Ignore" buttons. When the user clicks "Accept," the system automatically replaces the original text with the optimal correction words and updates the document content. When the user clicks "Ignore," the prompt disappears, and the system records this user feedback for future model optimization.

[0099] Please see Figure 2 , Figure 2 This illustration shows a schematic diagram of a misspelling recognition device provided in an embodiment of this application. The misspelling recognition device 200 includes: a data acquisition module 210, a first execution module 220, a second execution module 230, a third execution module 240, and a display module 250. Specifically:

[0100] Acquisition module 210 is used to acquire images of handwritten text;

[0101] The first execution module 220 is used to detect and recognize handwritten text images through a text detection model and a text recognition model, determine the confidence level of each character included in the handwritten text image, and filter out high-confidence characters from the characters included in the handwritten text image based on the confidence level of each character.

[0102] The second execution module 230 is used to classify the single-character images corresponding to high-confidence characters through a text-image comparison learning model, so as to filter out visually incorrect characters from the high-confidence characters.

[0103] The third execution module 240 is used to execute the semantic correction path corresponding to the language type of the visual error character in order to determine the optimal correction word corresponding to the visual error character.

[0104] Display module 250 is used to display the position of a visually misspelled character and its corresponding optimal correction word on the interactive interface.

[0105] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0106] In the several embodiments provided in this application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interface, device or module, and may be electrical, mechanical or other forms.

[0107] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0108] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a misspelling recognition device provided in an embodiment of this application. The misspelling recognition device 300 in this application may include one or more of the following components: processor 310, memory 320, and one or more application programs. The one or more application programs may be stored in memory 320 and configured to be executed by one or more processors 310. The one or more programs are configured to perform the misspelling recognition method as described in the foregoing method embodiments.

[0109] The processor 310 may include one or more processing cores. The processor 310 connects to various parts within the misspelling recognition device 300 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 320, and by calling data stored in the memory 320. Optionally, the processor 310 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 310 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 310 and may be implemented separately using a communication chip.

[0110] The memory 320 may include random access memory (RAM) or read-only memory (ROM). The memory 320 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 320 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described below, etc. The data storage area may also store data created by the misspelling recognition device 300 during use.

[0111] Please see Figure 4 , Figure 4The diagram shows a computer-readable storage medium 400 provided in an embodiment of this application. The computer-readable storage medium 400 stores program code, which can be called by a processor to execute the misspelling recognition method described in the above method embodiment.

[0112] The computer-readable storage medium 400 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 400 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 600 has storage space for program code 410 that performs any of the method steps described above. This program code can be read from or written to one or more computer program devices. The program code 410 may be compressed, for example, in a suitable form.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for identifying misspelled words, characterized in that, The method includes: Capture images of handwritten text; The handwritten text image is detected and recognized by a text detection model and a text recognition model. The confidence level of each character in the handwritten text image is determined. Based on the confidence level of each character, high-confidence characters are selected from the characters in the handwritten text image. The image-text comparison learning model is used to classify the single-character images corresponding to the high-confidence characters in order to filter out visually incorrect characters from the high-confidence characters. Execute the semantic correction path corresponding to the language type of the visual error character to determine the optimal correction word corresponding to the visual error character; The interactive interface displays the visually misspelled character and its corresponding optimal correction word.

2. The misspelling recognition method according to claim 1, characterized in that, The process involves detecting and recognizing the handwritten text image using a text detection model and a text recognition model, determining the confidence level of each character in the handwritten text image, and filtering out highly reliable characters from the characters in the handwritten text image based on the confidence level of each character. This includes: The text detection model is used to detect text lines in the handwritten text image to determine the bounding box coordinates of each text line in the handwritten text image. Based on the bounding box coordinates of each text line, the corresponding text line sub-image is cropped from the handwritten text image; The text recognition model is used to identify the characters in the text subgraph and determine the confidence level of each character in the text subgraph. Based on the confidence level and the first preset threshold corresponding to each character in the text line subgraph, the highly reliable characters are selected from the characters included in the text line subgraph.

3. The misspelling recognition method according to claim 2, characterized in that, The step of filtering out the highly reliable characters from the characters included in the text line subgraph based on the confidence level corresponding to each character in the text line subgraph and the first preset threshold includes: In the text line subgraph, characters with a confidence level greater than the first preset threshold are identified as highly reliable characters; And / or, in the text line subgraph, the characters corresponding to those with a confidence level less than or equal to the first preset threshold are determined as low-confidence characters.

4. The misspelling recognition method according to claim 1, characterized in that, The step of classifying the single-character images corresponding to the high-confidence characters using a text-image comparison learning model to filter out visually incorrect characters from the high-confidence characters includes: Obtain the single-character image of the highly reliable character in the handwritten text image; The single-character image is input into the image-text comparison learning model to calculate the visual probability that the single-character image belongs to the first category; the text encoder of the image-text comparison learning model is frozen, and the image encoder is fine-tuned to classify the single-character image into the first category or the second category. Based on the visual probability and the second preset threshold, the visually incorrect characters are filtered out from the highly reliable characters.

5. The misspelling recognition method according to claim 4, characterized in that, The step of filtering out the visually incorrect characters from the high-confidence characters based on the visual probability and the second preset threshold includes: The high-confidence characters corresponding to those with a visual probability greater than the second preset threshold are identified as visual error characters; And / or, the high-confidence characters corresponding to visual probabilities less than or equal to the second preset threshold are determined as correct characters.

6. The misspelling recognition method according to claim 1, characterized in that, The step of executing the semantic correction path corresponding to the language type of the visual error character to determine the optimal correction word corresponding to the visual error character includes: If the language type is the first language, then extract the context text consisting of a preset number of characters forward and backward, centered on the visual error character; Based on the pinyin of the visually misspelled characters, K first candidate correction words with the same pinyin or similar initials and finals are recalled from a preset database of millions of homophones and near-homophones. The context text and the K first candidate correction words are input into the MacBERT language model. The cosine similarity between the semantic representation vector of each first candidate correction word after it is substituted into the context text and the overall semantic vector of the context is calculated. The embedding layer of the MacBERT model incorporates pinyin embedding information. The first candidate correction word corresponding to the highest cosine similarity is selected as the optimal correction word, and the semantic confidence of the optimal correction word is determined.

7. The misspelling recognition method according to claim 1, characterized in that, The step of executing the semantic correction path corresponding to the language type of the visual error character to determine the optimal correction word corresponding to the visual error character includes: If the language type is a second language, then based on the spelling of the visual misspelling, the edit distance between it and each entry in the preset dynamic dictionary is calculated, and all entries whose edit distance is less than or equal to the preset value are determined as the second candidate correction words; The rationality of each second candidate correction word in the context text centered on the visual error character is scored using a 5-gram language model, and the second candidate correction word with the lowest perplexity is selected as the optimal correction word.

8. A typo recognition device, characterized in that, The device includes: The acquisition module is used to acquire images of handwritten text. The first execution module is used to detect and recognize the handwritten text image through a text detection model and a text recognition model, determine the confidence level corresponding to each character included in the handwritten text image, and filter out high-confidence characters from the characters included in the handwritten text image based on the confidence level corresponding to each character. The second execution module is used to classify the single-character images corresponding to the high-confidence characters through a text-image comparison learning model, so as to filter out visually incorrect characters from the high-confidence characters. The third execution module is used to execute the semantic correction path corresponding to the language type of the visual error character in order to determine the optimal correction word corresponding to the visual error character. The display module is used to display the position of the visual error character and its corresponding optimal correction word on the interactive interface.

9. A typo recognition device, characterized in that, include: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the misspelling recognition method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code, which can be called by a processor to execute the misspelling recognition method as described in any one of claims 1-7.

Images