A Multimodal Handwritten Chinese Character Error Correction Network Architecture Based on Contrast Learning

Abstract

This invention provides a multimodal handwritten Chinese character error correction network architecture based on contrastive learning, comprising: a pre-training module based on contrastive learning, a semantic character description sequence generation module, and a Chinese character error correction module. This invention fully utilizes the image and semantic features inherent in Chinese characters, leveraging the complementary advantages of multimodal features to achieve accurate recognition of handwritten Chinese characters. It can recognize characters never seen before without prior knowledge of their possible category set. The proposed method is generative, possessing a fundamental understanding of the structure and radicals of Chinese characters, enabling the identification of the structure and radicals of even previously unseen or incorrectly written characters. It eliminates the need to collect and label error character datasets, reducing labor and costs, and is suitable for widespread application.

Description

Technical Field

[0001] This invention belongs to the field of image acquisition algorithm technology, and in particular to a multimodal handwritten Chinese character error correction network architecture based on contrastive learning. Background Technology

[0002] With the increasing comprehensive national strength of China, the globalization of Chinese language and characters has become increasingly rapid, playing an important role in promoting global cultural exchange and enhancing the national image. Applying artificial intelligence technology to the learning process of Chinese language and characters has great development prospects.

[0003] Due to the highly complex internal structure and vast categories of Chinese characters, they pose a significant challenge for foreigners learning Chinese, who are prone to making unpredictable errors during the writing process. As a result, the Handwritten Chinese Character Error Correction (HCCEC) task has received increasing attention in recent years. However, the development of the HCCEC task has been relatively slow. Due to the complexity of Chinese characters and the significant differences in recognition between them and alphabetic words, there are currently no open-source datasets of incorrect characters. Such datasets are characterized by the difficulty of collection and their scarcity.

[0004] An existing technology discloses an OCR error correction method based on Chinese character-level features and a language model (application number: CN202110571433). This invention discloses a Chinese OCR error correction method based on Chinese character-level features and a language model, which includes the following steps:

[0005] 1) Train an OCR model to recognize Chinese images, and use data augmentation techniques to obtain data with OCR-corrected style errors;

[0006] 2) Introduce Chinese character stroke structure information and use the CBOW method to train word vectors so that the joint stroke structure word vectors can grasp the Chinese structure and stroke information;

[0007] 3) Construct a language model based on the joint stroke structure word vectors to generate a candidate set of character error correction.

[0008] Although the above inventions, through data augmentation techniques, can identify more salient features of OCR errors, improve model performance, solve the problem of recognizing characters with the same strokes but different structures, and improve the quality of generated error correction candidate sets, they still suffer from the same shortcomings as existing mainstream technologies, such as:

[0009] ① Application of multimodal information: Handwritten Chinese character recognition methods mainly utilize the image features of Chinese characters for recognition, and rarely utilize the semantic features inherent in the Chinese characters themselves.

[0010] ② Regarding the ability to recognize handwritten Chinese characters outside the training set: It is powerless for categories that do not appear in the training set. Ultimately, this is because a closed-set method requires predefined categories to assist in the recognition of handwritten Chinese characters. When encountering a Chinese character that has not been seen during training, the existing technology will mistakenly identify it as a "seen Chinese character" that has a certain degree of similarity to it. This is like being able to only answer questions on an exam paper that has already appeared in a mock exam.

[0011] ③ Regarding the correction of incorrect Chinese characters: Existing methods for correcting Chinese characters are only capable of correcting “misused characters” (misuse of similar-looking or similar-sounding characters), but they lack the ability to identify and correct “incorrect characters” that are written incorrectly and that differ from all existing Chinese characters.

[0012] ④ Existing recognition technologies are severely lacking in open-source misspelling datasets. Summary of the Invention

[0013] To address the aforementioned technical problems, this invention provides a multimodal handwritten Chinese character error correction network architecture based on contrastive learning. This architecture fully utilizes multiple modal information, such as the structural and semantic information of Chinese characters, to solve the problem of open vocabulary without predefined categories for Chinese character writing errors. By combining a multimodal encoder-decoder network trained using contrastive learning, it achieves error recognition and correction of handwritten Chinese characters.

[0014] A multimodal handwritten Chinese character error correction network architecture based on contrastive learning includes:

[0015] The pre-training module is based on contrastive learning, the ideographic character description sequence generation module is based on ideographic learning, and the Chinese character error correction module is based on ideographic learning.

[0016] The pre-training module based on contrastive learning refers to the following: In the pre-training part, the handwritten Chinese character image is used as the input of the image encoder, and the ideographic character description sequence (IDS) is used as the input of the text encoder. The pre-training module based on contrastive learning is trained using the contrastive learning method to obtain an image encoder that can recognize the ideographic character description sequence of the handwritten Chinese character image; and a text encoder for extracting the features of the corresponding ideographic character description sequence.

[0017] As an example, the ideographic character description sequence features include: radical features and structural features.

[0018] As an example, the structure of the contrastive learning-based pre-trained model includes:

[0019] An image encoder that takes handwritten Chinese character images as input; a text encoder that takes ideographic character description sequences as input; and a contrastive learning model that aligns the features output by both;

[0020] The image encoder, frozen after training, is followed by a multimodal decoder that includes image and text embeddings. The two are then aligned again using a matching loss to reduce the illusions of the pre-trained module based on contrastive learning. Finally, a predicted ideographic text description sequence is output.

[0021] As an example, the handwritten Chinese character image can also be other target characters, such as English, Russian, Latin, Japanese, and Korean.

[0022] As an example, the image encoder includes: a 12-layer ViT for extracting handwritten Chinese character image features.

[0023] Where: H represents the height of the image, W represents the width of the image, and C represents the channel dimension.

[0024] Furthermore, in order to represent the handwritten Chinese character image features using a one-dimensional vector, an end identifier is added to the end of the image block sequence; since the end identifier can focus on the features of all preceding image blocks, it can represent the global features of the image.

[0025] This represents the projection matrix, where C′ is the alignment dimension;

[0026] As an example, the text encoder includes: a 12-layer Transformer encoder and an embedding layer, which encodes the ideographic text description sequence as:

[0027] The text encoder also projects the semantic text description sequence features onto... Represents the projection matrix;

[0028] Where: T is the feature output by the ideographic text description sequence after passing through the text encoder, and the feature after being projected onto the unified feature space; D is the "dimension" of the feature output by the ideographic text description sequence after passing through the text encoder.

[0029] Furthermore, using the contrastive loss function Align the handwritten Chinese character image features and the ideographic character description sequence features;

[0030] As an example, for a batch of N training samples, the contrastive loss function The design is as follows:

[0031]

[0032] Among them: I j Tj Let represent the handwritten Chinese character image features and the ideographic character description sequence features of the j-th sample in the image block, respectively. To address the issue of identical Chinese character samples in the same image block, subsequent labels are aligned with the label at the position where the Chinese character first appears during calculation.

[0033] The ideographic character description sequence generation module refers to the process of generating an ideographic character description sequence for a corresponding handwritten Chinese character image using the image encoder in the ideographic character description sequence (IDS) generation part.

[0034] The ideographic character description sequence generation module structure includes: an image encoder with frozen weights and a multimodal decoder. The multimodal decoder contains features of the handwritten Chinese character image and the ideographic character description sequence; finally, it outputs a generated ideographic character description sequence.

[0035] Furthermore, for each handwritten Chinese character image-ideographic description sequence pair, the ideographic description sequence generation module is trained using a loss training model of a multimodal language model. The formula for the loss training model is designed as follows:

[0036]

[0037] Where: I represents the input handwritten Chinese character image features, T i For the ideographic text describing the sequence identifier, T0 is the start identifier, T N+1 It is the end identifier; CE is the cross-entropy loss.

[0038] The Chinese character error correction module is used to: determine the error type of the character and find the correct Chinese character corresponding to the erroneous Chinese character, and perform error correction; after the ideographic character description sequence generation module, the Chinese character error correction module will match the generated IDS with the IDS dictionary;

[0039] As an example, the matching process includes the following three steps:

[0040] Step 1: Precise matching;

[0041] If the IDS dictionary contains a sequence that is completely identical to the IDS of the target character, the target character is classified as a "correct character"; if no completely identical sequence is found, proceed to the second step.

[0042] Step 2: Unordered matching;

[0043] If there exists a sequence in the IDS dictionary where every element is identical to the ideographic character description sequence of the target character, differing only in order, then the target character is classified as a "structurally disordered character," and the sequence is the correct character corresponding to the target character.

[0044] Third, if no result can be obtained in the unordered matching, the IDS of the target character is input into the text encoder, and the output IDS features are compared with the IDS features in the IDS dictionary by cosine similarity. The character with the highest cosine similarity is selected as the correct character of the target sequence and classified as "radical error character".

[0045] The beneficial effects of this invention are:

[0046] 1. Multimodal data: This invention uses multimodal Chinese character data, making full use of the image and semantic features of Chinese characters themselves, and leveraging the complementary advantages of multimodal features to achieve accurate recognition of handwritten Chinese characters.

[0047] 2. No need for predefined categories: This invention can identify Chinese characters that have never been seen before, without needing to know in advance the set of categories to which the Chinese character may belong.

[0048] 3. It can identify misspelled characters: The method proposed in this invention is based on generation and has a fundamental understanding of the structure and radicals of Chinese characters. It can identify the structure and radicals of a Chinese character that has never been seen before or is even misspelled.

[0049] 4. Reduced labor costs: No need to collect and label misspelled character datasets. An innovative approach uses a dataset of uncommon characters as an alternative to a dataset of misspelled characters, and fully demonstrates the feasibility of this method in handwritten Chinese character error correction tasks. Attached Figure Description

[0050] Figure 1 This is a schematic diagram illustrating the overall process of a multimodal handwritten Chinese character error correction network architecture based on contrastive learning according to the present invention.

[0051] Figure 2 This is a schematic diagram of the overall architecture of a multimodal handwritten Chinese character error correction network architecture based on contrastive learning according to the present invention.

[0052] Figure 3 This is a schematic diagram of the Chinese character error correction module structure of a multimodal handwritten Chinese character error correction network architecture based on contrastive learning according to the present invention.

[0053] Figure 4 This is a verification example of the possibility of replacing incorrect characters with uncommon characters in a multimodal handwritten Chinese character error correction network architecture based on contrastive learning according to the present invention. Detailed Implementation

[0054] Below, for reference Figures 1 to 3 As shown, a multimodal handwritten Chinese character error correction network architecture based on contrastive learning includes:

[0055] The pre-training module is based on contrastive learning, the ideographic character description sequence generation module is based on ideographic learning, and the Chinese character error correction module is based on ideographic learning.

[0056] The pre-training module based on contrastive learning refers to: in the pre-training part, handwritten Chinese character images are used as image inputs, and ideographic character description sequences (IDS) are used as text inputs. The pre-training module based on contrastive learning is trained using a contrastive learning method to obtain an image encoder that can recognize ideographic character description sequences (IDS) of handwritten Chinese character images; and a text encoder for extracting radical features and structural features of the corresponding IDS.

[0057] As an example, the structure of the contrastive learning-based pre-trained model includes:

[0058] An image encoder that takes handwritten Chinese character images as input; a text encoder that takes ideographic character description sequences as input; and a contrastive learning model that aligns the features output by both.

[0059] As an example, the handwritten Chinese character image can also be other target characters, such as English, Russian, Latin, Japanese, and Korean.

[0060] As an example, the image encoder includes: a 12-layer ViT for extracting handwritten Chinese character image features.

[0061] Where: H represents the height of the image, W represents the width of the image, and C represents the channel dimension.

[0062] Furthermore, in order to represent the handwritten Chinese character image features using a one-dimensional vector, an end identifier is added to the end of the image block sequence; since the end identifier can focus on the features of all preceding image blocks, it can represent the global features of the image.

[0063] This represents the projection matrix, where C′ is the alignment dimension;

[0064] As an example, the text encoder includes: a 12-layer Transformer encoder and an embedding layer, which encodes the ideographic text description sequence as:

[0065] The text encoder also projects IDS features to... Represents the projection matrix;

[0066] Where: T is the feature output by the ideographic text description sequence after passing through the text encoder, and the feature after being projected onto the unified feature space; D is the "dimension" of the feature output by the ideographic text description sequence after passing through the text encoder.

[0067] Furthermore, using the contrastive loss function Align the handwritten Chinese character image features and the ideographic character description sequence features;

[0068] As an example, for a batch of N training samples, the contrastive loss function The design is as follows:

[0069]

[0070] Among them: I j T j Let represent the handwritten Chinese character image features and IDS features of the j-th sample in the image patch, respectively. To address the issue of identical Chinese character samples in the same image patch, subsequent labels are aligned with the label at the position where the Chinese character first appears during calculation.

[0071] The ideographic character description sequence generation module refers to the process of generating an ideographic character description sequence for a corresponding handwritten Chinese character image using the image encoder in the ideographic character description sequence (IDS) generation part.

[0072] The ideographic text description sequence generation module structure includes: an image encoder with frozen weights and a multimodal decoder. The multimodal decoder contains features of the handwritten Chinese character image and the ideographic text description sequence; finally, it outputs a generated ideographic text description sequence.

[0073] Furthermore, for each handwritten Chinese character image-ideographic description sequence pair, the ideographic description sequence generation module is trained using the loss function of a multimodal language model. The formula for the loss function is designed as follows:

[0074]

[0075] Where: I represents the input handwritten Chinese character image features, T i For the ideographic text describing the sequence identifier, T0 is the start identifier, T N+1 It is the end identifier; CE is the cross-entropy loss.

[0076] The Chinese character error correction module is used to: determine the error type of the character and find the correct Chinese character corresponding to the erroneous Chinese character, and perform error correction; after the ideographic character description sequence generation module, the Chinese character error correction module will match the generated IDS with the IDS dictionary;

[0077] As an example, the matching process includes the following three steps:

[0078] Step 1: Precise matching;

[0079] If the IDS dictionary contains a sequence that is completely identical to the ideographic character description sequence of the target character, then the target character is classified as a "correct character"; if no completely identical sequence is found, proceed to the second step.

[0080] Step 2: Unordered matching;

[0081] If there exists a sequence in the IDS dictionary where every element is identical to the ideographic character description sequence of the target character, differing only in order, then the target character is classified as a "structurally disordered character," and the sequence is the correct character corresponding to the target character.

[0082] Third, if no result can be obtained in the unordered matching, the ideographic character description sequence of the target character is input into the text encoder. The output ideographic character description sequence features are compared with the ideographic character description sequence features in the IDS dictionary using cosine similarity. The character with the highest cosine similarity is selected as the correct character of the target sequence and classified as "radical error character".

[0083] Example of verifying the possibility of using uncommon characters to replace misspelled characters: See details Figure 4 As shown;

[0084] Given the extreme scarcity of open-source misspelling datasets, this invention analyzes the error patterns of incorrect Chinese characters and finds that most of these errors stem from improper use of radicals and structural confusion. This phenomenon is very similar to the way rare characters appear, which are rarely used in daily life, and many people don't even recognize them. Considering that the target users of this invention are beginners in Chinese characters, when a beginner writes a rare character that almost no one recognizes, it is very likely a writing error. Therefore, this invention uses a rare character handwriting dataset as a substitute for the misspelling handwriting dataset for functional testing of this invention.

[0085] To better understand the design principles of this invention, the various names and related knowledge specifically involved in this invention will be explained in detail below:

[0086] Ideographic Sequences (IDS): Ideographic Sequences (IDS) are a syntax for describing the structure of Chinese characters defined by the Unicode standard. A sequence of descriptions is composed of a descriptive character and two or more specific characters (mainly Chinese characters) to represent the abstract structure of Chinese characters.

[0087] Handwritten Chinese character dataset:

[0088] HWDB1.0 contains 3,866 Chinese characters and 171 alphanumeric characters and symbols. Of the 3,866 Chinese characters, 3,740 are from the GB2312-80 Level 1 Chinese character set (out of a total of 3,755 characters).

[0089] HWDB1.1 contains 3,755 GB2312-80 Level 1 Chinese characters and 171 alphanumeric characters and symbols.

[0090] HWDB1.2 contains 3,319 Chinese characters and 171 alphanumeric characters and symbols. The Chinese character set (3,319 classes) in HWDB1.2 is a disjoint set of HWDB1.0.

[0091] HWDB1.0 and HWDB1.2 contain a total of 7185 Chinese characters (7,185 = 3,866 + 3,319), which includes all 6763 Chinese characters in GB2312.

[0092]

[0093] Open vocabulary recognition technology: Open vocabulary recognition refers to the process by which the model processes words that are not limited to a predefined, fixed vocabulary in Natural Language Processing (NLP) and Computer Vision (CV) tasks. In other words, the model can handle new words, rare words, or unseen words that are outside the training vocabulary. This contrasts with closed vocabulary, where the model can only process words from a fixed vocabulary, and words outside that vocabulary will not be recognized. This approach often has limitations when dealing with dynamic or complex language, while open vocabulary methods are more flexible.

[0094] Handwritten Chinese character recognition technology based on radicals:

[0095] Radical-based handwritten Chinese character recognition technology refers to a technique that improves the accuracy of handwritten Chinese character recognition by identifying the radicals (components or parts) of the character. This method breaks down Chinese characters into smaller components and derives the entire character by recognizing these radical features.

[0096] Specifically, radicals, as an important component of Chinese characters, not only carry semantic information but also help distinguish characters with similar shapes. Radical-based recognition technology typically includes the following steps:

[0097] Decomposition of Chinese character structure: Dividing handwritten Chinese characters into several radicals or character components.

[0098] Radical feature extraction: Extract features for each radical and analyze their shape, position and combination.

[0099] Combination and Matching: Based on the combination and interrelationship of radicals, Chinese characters are reconstructed and matched to identify complete Chinese characters.

[0100] The advantage of this technology is that it can reduce the difficulty of overall recognition by decomposing complex Chinese characters. Especially when dealing with complex or similar characters, the radical-based recognition method helps to improve the robustness and accuracy of the system.

[0101] Contrastive learning is a self-supervised learning method that learns effective feature representations by comparing the similarities and differences between samples. In contrastive learning, the model is trained to bring similar samples closer together and push dissimilar samples further apart. It is commonly used for representation learning tasks, enabling the model to learn useful features even when a large amount of labeled data is lacking.

[0102] The comparative learning process generally includes the following steps:

[0103] Construction of positive and negative sample pairs: For a sample, a "positive sample" (usually a data augmented version) is generated and compared with other unrelated samples as "negative samples".

[0104] Similarity measure: By calculating the similarity between sample pairs (e.g., cosine similarity), the model is trained to maximize the similarity of positive sample pairs while minimizing the similarity with negative samples.

[0105] Loss function: Common contrastive learning loss functions are contrastive loss or information theory lower bound estimation (InfoNCE loss), which are used to measure the distance difference between positive and negative sample pairs.

[0106] Contrastive learning is widely used in feature learning tasks in fields such as images and text, and performs well in both unsupervised and semi-supervised learning. For example, SimCLR and MoCo are classic methods based on contrastive learning.

[0107] In this way, the model can learn features with the same semantics in similar contexts, thereby improving its performance in tasks such as classification and clustering.

[0108] The above description is only a preferred embodiment of the present invention. It should be understood that the above description of the embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention, and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, etc. made within the idea and principle of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal handwritten Chinese character error correction network architecture based on contrastive learning, characterized in that, include: The pre-training module is based on contrastive learning, the ideographic character description sequence generation module is based on ideographic learning, and the Chinese character error correction module is based on ideographic learning. The pre-training module based on contrastive learning refers to the following: In the pre-training part, the handwritten Chinese character image is used as the input of the image encoder, and the ideographic character description sequence is used as the input of the text encoder. The pre-training module based on contrastive learning is trained using the contrastive learning method to obtain an image encoder that can recognize the ideographic character description sequence of the handwritten Chinese character image; and a text encoder for extracting the features of the corresponding ideographic character description sequence. The ideographic character description sequence generation module refers to the process of using the image encoder to generate the ideographic character description sequence of the corresponding handwritten Chinese character image in the ideographic character description sequence generation part. The Chinese character error correction module is used to: determine the error type of the character and find the correct Chinese character corresponding to the erroneous Chinese character, and perform error correction; after the ideographic character description sequence generation module, the Chinese character error correction module will match the generated ideographic character description sequence with the ideographic character description sequence dictionary; The structure of the pre-training module based on contrastive learning includes: an image encoder that takes handwritten Chinese character images as input; a text encoder that takes ideographic character description sequences as input; and a contrastive learning model that aligns the features output by both. The ideographic character description sequence generation module includes: an image encoder with frozen weights and a multimodal decoder. The multimodal decoder contains features of the handwritten Chinese character image and the ideographic character description sequence; finally, it outputs a generated ideographic character description sequence.

2. The multimodal handwritten Chinese character error correction network architecture based on contrastive learning according to claim 1, characterized in that, The handwritten Chinese character images are other target characters, including: English, Russian, Latin, Japanese, and Korean.

3. The multimodal handwritten Chinese character error correction network architecture based on contrastive learning according to claim 1, characterized in that, The image encoder includes a 12-layer ViT layer for extracting handwritten Chinese character image features. Where: H represents the height of the image, W represents the width of the image, and C represents the channel dimension; To represent the handwritten Chinese character image features using a one-dimensional vector, an end identifier was added to the end of the image block sequence; since the end identifier can capture the features of all preceding image blocks, it can represent the global features of the image. Let C' represent the projection matrix, and C' be the alignment dimension.

4. The multimodal handwritten Chinese character error correction network architecture based on contrastive learning according to claim 3, characterized in that, The text encoder includes a 12-layer Transformer encoder and an embedding layer. The text encoder encodes the ideographic text description sequence as follows: The text encoder also projects the semantic text description sequence features onto... Represents the projection matrix; Where: T is the feature output by the ideographic text description sequence after passing through the text encoder, and the feature after being projected onto the unified feature space; D is the "dimension" of the feature output by the ideographic text description sequence after passing through the text encoder.

5. The multimodal handwritten Chinese character error correction network architecture based on contrastive learning according to claim 4, characterized in that, Using the contrastive loss function Align the handwritten Chinese character image features and the ideographic character description sequence features; For a batch of N training samples, the contrastive loss function The design is as follows: Among them: I j T j Let represent the handwritten Chinese character image features and the ideographic character description sequence features of the j-th sample in the image block, respectively. To address the issue of identical Chinese character samples within the same image block, subsequent labels are aligned with the label at the position where the Chinese character first appears during calculation.

6. The multimodal handwritten Chinese character error correction network architecture based on contrastive learning according to claim 1, characterized in that, For each handwritten Chinese character image-ideographic description sequence pair, the ideographic description sequence generation module is trained using the loss of a multimodal language model. The formula for the loss is designed as follows: Where: I represents the input handwritten Chinese character image features, T i For the ideographic text describing the sequence identifier, T0 is the start identifier, T N+1 It is the end identifier; CE is the cross-entropy loss.

7. The multimodal handwritten Chinese character error correction network architecture based on contrastive learning according to claim 1, characterized in that, The matching process includes the following three steps: Step 1: Precise matching; If the dictionary of ideographic character description sequences contains a sequence that is completely identical to the ideographic character description sequence of the target character, then the target character is classified as a "correct character"; if no completely identical sequence is found, proceed to the second step. Step 2: Unordered matching; If a sequence exists in the dictionary of ideographic character description sequences where every element of the sequence is the same as the ideographic character description sequence of the target character, only the order is different, then the target character is classified as a "structurally disordered character", and the sequence is the correct character corresponding to the target character. Third, if no result can be obtained in the unordered matching, the ideographic character description sequence of the target character is input into the text encoder. The output ideographic character description sequence features are compared with the ideographic character description sequence features in the ideographic character description sequence dictionary using cosine similarity. The character with the highest cosine similarity is selected as the correct character of the target sequence and classified as "radical error character".

Images