Multi-modal handwritten Chinese character error correction network architecture based on comparative learning

Abstract

The invention provides a multi-modal handwritten Chinese character error correction network architecture based on comparative learning. The multi-modal handwritten Chinese character error correction network architecture comprises a pre-training module based on comparative learning, an ideographic description sequence generation module and a Chinese character error correction module, according to the method, image features and semantic features of Chinese characters are fully utilized, the complementary advantage of multi-modal features is exerted, accurate recognition of handwritten Chinese characters is achieved, Chinese characters which are never seen can be recognized, and it is not needed to know a category set which the Chinese characters possibly belong to in advance; according to the method provided by the invention, based on generation, the structure and the radical of the Chinese character are understood fundamentally, and the structure and the radical of a never seen Chinese character or even a wrongly written Chinese character can be identified; the method does not need to collect and mark a wrong character data set, reduces labor and cost, and is suitable for popularization.

Description

Technical Field

[0001] The present invention belongs to the technical field of image acquisition algorithms, and in particular to a multimodal handwritten Chinese character error correction network architecture based on contrastive learning. Background Art

[0002] With the growing comprehensive national strength of China, the globalization of Chinese language and Chinese characters has become increasingly rapid, playing an important role in promoting global cultural exchanges and enhancing the national image. Applying artificial intelligence technology to the learning process of Chinese language and Chinese characters is very promising.

[0003] Since Chinese characters have a very complex internal structure and a large number of categories, it is very difficult for foreigners who are just starting to learn Chinese characters. It is easy to make unpredictable mistakes in the process of writing Chinese characters. Therefore, the handwritten Chinese character error correction task (Handwritten Chinese Character Error Correction, HCCEC) has received more and more attention in recent years, but the development of the handwritten Chinese character error correction task is relatively slow. Due to the complexity of Chinese characters, there is a big difference in recognition from alphabetic words. There is currently no open source dataset for wrong characters. The dataset is difficult to collect and is scarce.

[0004] In the prior art, there is an OCR error correction method based on Chinese character-level features and language models. Application number: CN202110571433. The invention discloses a Chinese OCR error correction method based on Chinese character-level features and language models. The method comprises the following steps:

[0005] 1) Train an OCR model to recognize Chinese images, and use data augmentation technology to obtain data with post-OCR error styles;

[0006] 2) Introducing the stroke structure information of Chinese characters to train word vectors using the CBOW method, so that the joint stroke structure word vector can grasp the Chinese structure and stroke information;

[0007] 3) Build a language model based on the joint stroke structure word vector to generate a candidate set of character corrections.

[0008] Although the above invention can find more significant features of post-OCR errors and improve the model effect through data enhancement technology, solve the recognition error problem of characters with the same strokes but different structures, and improve the quality of generating error correction candidate sets, it still has the following defects like the existing mainstream technology, such as:

[0009] ① Application of multimodal information: Handwritten Chinese character recognition methods mainly use the image features of Chinese characters for recognition, and rarely use the semantic features of Chinese characters themselves.

[0010] ② Ability to recognize handwritten Chinese characters from categories outside the set: It is unable to recognize categories that do not appear in the training set. This is ultimately because a closed set method must have predefined categories to assist in the recognition of handwritten Chinese characters. When encountering a Chinese character that has not been seen in the training process, the existing technology will mistake it for a "seen Chinese character" that has a certain degree of similarity with it. This is like only being able to take an exam paper where all the questions have already appeared in the mock exam paper.

[0011] ③ Correction of incorrect Chinese characters: Existing Chinese character error correction methods are only capable of correcting "wrong characters" (misuse of similar characters or similar pronunciations), but lack the ability to identify and correct "wrong characters" that occur during the writing process and are different from all existing Chinese characters.

[0012] ④ Existing recognition technologies are extremely lacking in open source datasets of typos. Summary of the invention

[0013] In order to solve the above technical problems, the present invention provides a multimodal handwritten Chinese character error correction network architecture based on contrastive learning, which makes full use of multiple modal information such as structural information and semantic information of Chinese characters, solves an open vocabulary problem such as Chinese character writing errors without predefined categories, and combines a multimodal codec network with contrastive learning pre-training method to realize error recognition and correction of handwritten Chinese characters.

[0014] A multimodal handwritten Chinese character error correction network architecture based on contrastive learning, including:

[0015] Pre-training module based on contrastive learning, ideographic description sequence generation module and Chinese character error correction module;

[0016] The pre-training module based on contrastive learning means: in the pre-training part, the handwritten Chinese character image is used as the input of the image encoder, and the ideographic description sequence (IDS) is used as the input of the text encoder, and the pre-training module based on contrastive learning is trained by the contrastive learning method to obtain an image encoder with the function of recognizing the ideographic description sequence of the handwritten Chinese character image; and a text encoder for extracting the corresponding ideographic description sequence features;

[0017] As an example, the ideographic character description sequence features include: radical features and structural features.

[0018] As an example, the structure of the pre-training model based on contrastive learning includes:

[0019] An image encoder that takes images of handwritten Chinese characters as input; a text encoder that takes sequences of ideographic descriptions as input; and a contrastive learning model that aligns the features of the two outputs;

[0020] The trained frozen image encoder is followed by a multimodal decoder that contains image and text embeddings, which are then re-aligned with a matching loss to reduce the hallucinations of the contrastive learning-based pre-trained module; the final output is a predicted ideographic description sequence.

[0021] As an example, the handwritten Chinese character image may also be other target characters, such as English, Russian, Latin, Japanese and Korean.

[0022] As an example, the image encoder includes: 12 layers of ViT, which are used to extract handwritten Chinese character image features:

[0023] Where: H represents the height of the image, W represents the width of the image, and C represents the channel dimension.

[0024] Furthermore, in order to represent the handwritten Chinese character image features with a one-dimensional vector, an end identifier is added to the end of the image block sequence; since the end identifier can focus on the features of all the previous image blocks, the end identifier can represent the global features of the image.

[0025] represents the projection matrix, C′ is the alignment dimension;

[0026] As an example, the text encoder includes: 12 layers of Transformer encoders and 1 embedding layer, through which the ideographic description sequence is encoded as:

[0027] The text encoder also projects the ideographic description sequence features into represents the projection matrix;

[0028] Where: T is the feature output of the ideographic description sequence after passing through the text encoder, and the feature after being projected into the unified feature space; D is the "dimension" of the feature output of the ideographic description sequence after passing through the text encoder;

[0029] Furthermore, using the contrast loss function Aligning the handwritten Chinese character image features and the ideographic character description sequence features;

[0030] As an example, for a batch of N training samples, the contrast loss function is The design is as follows:

[0031]

[0032] Where: Ij , T j They represent the handwritten Chinese character image features and ideographic character description sequence features of the jth sample in the image block. In order to adapt to the problem of the same Chinese character samples in the same image block, the subsequent labels are aligned with the labels at the position where the Chinese character first appears during calculation;

[0033] The ideographic character description sequence generation module refers to: in the ideographic character description sequence (IDS) generation part, using the image encoder to realize the generation of the ideographic character description sequence corresponding to the handwritten Chinese character image;

[0034] The ideographic character description sequence generation module structure includes: an image encoder with frozen weights, a multimodal decoder, and the multimodal decoder contains the features of the handwritten Chinese character image and the ideographic character description sequence; and finally outputs a generated ideographic character description sequence.

[0035] Furthermore, for each handwritten Chinese character image-ideographic description sequence pair, the ideographic description sequence generation module is trained using a loss training model of a multimodal language model, and the formula of the loss training model is designed as follows:

[0036]

[0037] Where: I represents the input handwritten Chinese character image features, T i It is an ideographic character description sequence identifier, T0 is the start identifier, T N+1 is the end identifier; CE is the cross entropy loss.

[0038] The Chinese character error correction module is used to: determine the error type of the character and find the correct Chinese character corresponding to the wrong Chinese character, and perform error correction; after the ideographic character description sequence generation module, the Chinese character error correction module will match the generated IDS with the IDS dictionary;

[0039] As an example, the matching includes the following three steps:

[0040] Step 1: Exact matching;

[0041] If there is a sequence in the IDS dictionary that is completely consistent with the IDS of the target character, the target character is classified as a "correct character"; if there is no completely consistent sequence, proceed to the second step;

[0042] Step 2: Unordered matching;

[0043] If there is a sequence in the IDS dictionary where every element is the same as the ideographic description sequence of the target character, only the order is different, then the target character is classified as a "disordered character" and the sequence is the correct character corresponding to the target character;

[0044] Step 3: If no result can be obtained in the unordered matching, the IDS of the target character is input into the text encoder, and the output IDS features are compared with the IDS features in the IDS dictionary by cosine similarity. The character with the largest cosine similarity is selected as the correct character of the target sequence and classified as a "radical error character".

[0045] Beneficial effects of the present invention:

[0046] 1. Multimodal data: The present invention uses multimodal Chinese character data, fully utilizes the image features and semantic features of Chinese characters themselves, gives play to the complementary advantages of multimodal features, and realizes accurate recognition of handwritten Chinese characters.

[0047] 2. No need to predefine categories: The present invention can recognize Chinese characters that have never been seen before without knowing in advance the category set to which the Chinese character may belong.

[0048] 3. Can identify wrong characters: The method proposed in the present invention is based on generation and has a fundamental understanding of the structure and radicals of Chinese characters. It can identify the structure and radicals of a Chinese character that has never been seen before or even written incorrectly.

[0049] 4. Reduce labor costs: No need to collect and annotate wrong character datasets. Innovatively use uncommon character datasets as a substitute for wrong character datasets, and fully prove the feasibility of this method in handwritten Chinese character error correction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The figure is a schematic diagram of the overall process of a multimodal handwritten Chinese character error correction network architecture based on contrastive learning according to the present invention.

[0051] Figure 2 This is a schematic diagram of the overall architecture of a multimodal handwritten Chinese character error correction network architecture based on contrastive learning in the present invention.

[0052] Figure 3 This is a schematic diagram of the structure of a Chinese character error correction module of a multimodal handwritten Chinese character error correction network architecture based on contrastive learning in the present invention.

[0053] Figure 4 This is an example diagram of the possibility of verifying that rare characters can replace wrong characters in a multimodal handwritten Chinese character error correction network architecture based on contrastive learning in the present invention. DETAILED DESCRIPTION

[0054] Below, reference Figures 1 to 3 As shown in the figure, a multimodal handwritten Chinese character error correction network architecture based on contrastive learning includes:

[0055] Pre-training module based on contrastive learning, ideographic description sequence generation module and Chinese character error correction module;

[0056] The pre-training module based on contrastive learning means: in the pre-training part, the handwritten Chinese character image is used as the image input, the ideographic description sequence (IDS) is used as the text input, and the pre-training module based on contrastive learning is trained by the contrastive learning method to obtain an image encoder with the function of recognizing the ideographic description sequence (IDS) of the handwritten Chinese character image; and a text encoder for extracting the radical features and structural features corresponding to the IDS;

[0057] As an example, the structure of the pre-training model based on contrastive learning includes:

[0058] An image encoder that takes images of handwritten Chinese characters as input; a text encoder that takes sequences of ideographic descriptions as input; and a contrastive learning model that aligns the features of their outputs.

[0059] As an example, the handwritten Chinese character image may also be other target characters, such as English, Russian, Latin, Japanese and Korean.

[0060] As an example, the image encoder includes: 12 layers of ViT, which are used to extract handwritten Chinese character image features:

[0061] Where: H represents the height of the image, W represents the width of the image, and C represents the channel dimension.

[0062] Furthermore, in order to represent the handwritten Chinese character image features with a one-dimensional vector, an end identifier is added to the end of the image block sequence; since the end identifier can focus on the features of all the previous image blocks, the end identifier can represent the global features of the image.

[0063] represents the projection matrix, C′ is the alignment dimension;

[0064] As an example, the text encoder includes: 12 layers of Transformer encoders and 1 embedding layer, through which the ideographic description sequence is encoded as:

[0065] The text encoder also projects the IDS features into represents the projection matrix;

[0066] Where: T is the feature output of the ideographic description sequence after passing through the text encoder, and the feature after being projected into the unified feature space; D is the "dimension" of the feature output of the ideographic description sequence after passing through the text encoder;

[0067] Furthermore, using the contrast loss function Aligning the handwritten Chinese character image features and the ideographic character description sequence features;

[0068] As an example, for a batch of N training samples, the contrast loss function is The design is as follows:

[0069]

[0070] Where: I j , T j They represent the handwritten Chinese character image features and IDS features of the jth sample in the image block. In order to adapt to the problem of the same Chinese character samples in the same image block, the subsequent labels are aligned with the labels at the position where the Chinese character first appears during calculation;

[0071] The ideographic character description sequence generation module refers to: in the ideographic character description sequence (IDS) generation part, using the image encoder to realize the generation of the ideographic character description sequence corresponding to the handwritten Chinese character image;

[0072] The structure of the ideographic description sequence generation module includes: an image encoder with frozen weights, a multimodal decoder, and the multimodal decoder contains the features of the handwritten Chinese character image and the ideographic description sequence; and finally outputs a generated ideographic description sequence.

[0073] Furthermore, for each handwritten Chinese character image-ideographic description sequence pair, the ideographic description sequence generation module is trained using the loss of the multimodal language model, and the loss formula is designed as:

[0074]

[0075] Where: I represents the input handwritten Chinese character image features, T i It is an ideographic character description sequence identifier, T0 is the start identifier, T N+1 is the end identifier; CE is the cross entropy loss.

[0076] The Chinese character error correction module is used to: determine the error type of the character and find the correct Chinese character corresponding to the wrong Chinese character, and perform error correction; after the ideographic character description sequence generation module, the Chinese character error correction module will match the generated IDS with the IDS dictionary;

[0077] As an example, the matching includes the following three steps:

[0078] Step 1: Exact matching;

[0079] If there is a sequence in the IDS dictionary that is completely consistent with the ideographic description sequence of the target character, the target character is classified as a "correct character"; if there is no completely consistent sequence, the second step is entered;

[0080] Step 2: Unordered matching;

[0081] If there is a sequence in the IDS dictionary where every element is the same as the ideographic description sequence of the target character, only the order is different, then the target character is classified as a "disordered character" and the sequence is the correct character corresponding to the target character;

[0082] Step 3: If no result can be obtained in the unordered matching, the ideographic description sequence of the target character is input into the text encoder, and the cosine similarity of the output ideographic description sequence features is compared with the ideographic description sequence features in the IDS dictionary, and the character with the largest cosine similarity is selected as the correct character of the target sequence and classified as a "radical error character".

[0083] Example of the possibility of rare characters replacing wrong characters: see Figure 4 As shown;

[0084] In view of the fact that open source datasets of wrong characters are extremely scarce, the present invention analyzes the error patterns of wrong Chinese characters and finds that most of these errors are caused by improper use of radicals and chaotic structures; this phenomenon is very similar to the appearance of uncommon characters, which are rarely used in daily life and many people do not even recognize them; considering that the target users of the present invention are beginners of Chinese characters, when beginners write an uncommon character that almost no one recognizes, it is likely to be a writing error; therefore, the present invention uses a handwritten dataset of uncommon characters as a substitute for a handwritten dataset of wrong characters for functional testing of the present invention.

[0085] In order to better understand the design principle of the present invention, various names and related knowledge specifically involved in the present invention are explained in detail as follows:

[0086] Ideographic Description Sequence (IDS): Ideographic Description Sequence (IDS) is a Chinese character structure description syntax defined by the Unicode standard. The description sequence is composed of a description character and two or more specific characters (mainly Chinese characters) to represent the abstract structure of a Chinese character.

[0087] Handwritten Chinese character dataset:

[0088] HWDB1.0 contains 3,866 Chinese characters and 171 alphanumeric characters and symbols. Among the 3,866 Chinese characters, 3,740 are in the GB2312-80 Level 1 Chinese character set (a total of 3,755 Chinese characters).

[0089] HWDB1.1 contains 3,755 GB2312-80 first-level Chinese characters and 171 alphanumeric characters and symbols.

[0090] HWDB1.2 contains 3,319 Chinese characters and 171 alphanumeric and symbol characters. The Chinese character set (3,319 classes) in HWDB1.2 is a disjoint set of HWDB1.0.

[0091] HWDB1.0 and HWDB1.2 contain a total of 7185 Chinese characters (7,185=3,866+3,319), including all 6763 Chinese characters in GB2312.

[0092]

[0093] Open Vocabulary Recognition Technology: Open Vocabulary recognition means that in natural language processing (NLP) and computer vision (CV) tasks, the vocabulary processed by the model is not limited to a predefined, fixed vocabulary. In other words, the model can handle new words, rare words, or unseen words that are beyond the scope of the training vocabulary. In contrast, Closed Vocabulary means that the model can only process words in a fixed vocabulary, and words outside the vocabulary will not be recognized. This method often has limitations when dealing with dynamic or complex languages, while the open vocabulary method is more flexible.

[0094] Handwritten Chinese character recognition technology based on radicals:

[0095] Radical-based handwritten Chinese character recognition technology refers to a technology that improves recognition accuracy by identifying the radicals (components or components of Chinese characters) of Chinese characters during the handwritten Chinese character recognition process. This method breaks down Chinese characters into smaller components and derives the entire Chinese character by identifying these radical features.

[0096] Specifically, as an important part of Chinese characters, radicals not only carry the meaning of characters, but also help distinguish Chinese characters with similar shapes. Radical-based recognition technology usually includes the following steps:

[0097] Chinese character structure decomposition: Divide handwritten Chinese characters into several radicals or glyph components.

[0098] Radical feature extraction: Extract features from each radical and analyze their shape, position, and combination.

[0099] Combination and matching: Based on the combination and mutual relationship of radicals, Chinese characters are reconstructed and matched to identify complete Chinese characters.

[0100] The advantage of this technology is that it can reduce the overall recognition difficulty by decomposing complex Chinese characters. Especially when dealing with complex or similar characters, the radical-based recognition method helps to improve the robustness and accuracy of the system.

[0101] Contrastive Learning: Contrastive Learning is a self-supervised learning method. Its core idea is to learn effective feature representations by comparing the similarities and differences between samples. In contrastive learning, the model is trained to bring similar samples closer and push dissimilar samples further away. It is often used to represent learning tasks, allowing the model to learn useful features in the absence of a large amount of labeled data.

[0102] The process of contrastive learning generally includes the following steps:

[0103] Construction of positive and negative sample pairs: For a sample, generate a "positive sample" (usually a data-augmented version) and compare it with other irrelevant samples as "negative samples".

[0104] Similarity metrics: By calculating the similarity between pairs of samples (e.g., cosine similarity), the model is trained to maximize the similarity of positive sample pairs while minimizing the similarity with negative samples.

[0105] Loss function: Common contrastive learning loss functions are contrastive loss or InfoNCE loss, which are used to measure the distance difference between positive and negative sample pairs.

[0106] Contrastive learning is widely used in feature learning tasks in fields such as images and texts, and performs well in unsupervised and semi-supervised learning. For example, SimCLR, MoCo, etc. are classic methods based on contrastive learning.

[0107] In this way, the model can learn features with the same semantics in similar contexts, thereby improving performance in tasks such as classification and clustering.

[0108] The above are only preferred embodiments of the present invention. It should be understood that the description of the above embodiments is only used to help understand the method and core ideas of the present invention, and is not used to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, etc. made within the ideas and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal handwritten Chinese character error correction network architecture based on contrastive learning, characterized in that: include: Pre-training module based on contrastive learning, ideographic description sequence generation module and Chinese character error correction module; The pre-training module based on contrastive learning means: in the pre-training part, the handwritten Chinese character image is used as the input of the image encoder, and the ideographic character description sequence is used as the input of the text encoder, and the pre-training module based on contrastive learning is trained by the contrastive learning method to obtain an image encoder with the function of recognizing the ideographic character description sequence of the handwritten Chinese character image; and a text encoder for extracting the features of the corresponding ideographic character description sequence; The ideographic character description sequence generation module refers to: in the ideographic character description sequence generation part, using the image encoder to realize the generation of the ideographic character description sequence corresponding to the handwritten Chinese character image; The Chinese character error correction module is used to determine the error type of the character and find the correct Chinese character corresponding to the incorrect Chinese character for error correction; after the ideographic character description sequence generation module, the Chinese character error correction module will match the generated ideographic character description sequence with the ideographic character description sequence dictionary.

2. According to the multimodal handwritten Chinese character error correction network architecture based on contrastive learning according to claim 1, it is characterized in that: The structure of the pre-training module based on contrastive learning includes: an image encoder taking a handwritten Chinese character image as input; a text encoder taking an ideographic character description sequence as input; and a contrastive learning model for aligning features outputted by the two; The ideographic character description sequence generation module structure includes: an image encoder with frozen weights, a multimodal decoder, and the multimodal decoder contains the features of the handwritten Chinese character image and the ideographic character description sequence; and finally outputs a generated ideographic character description sequence.

3. According to the multimodal handwritten Chinese character error correction network architecture based on contrastive learning as described in claim 1, it is characterized in that: The handwritten Chinese character images are other target characters, including: English, Russian, Latin, Japanese and Korean.

4. According to the multimodal handwritten Chinese character error correction network architecture based on contrastive learning as described in claim 1, it is characterized in that: The image encoder includes: 12 layers of ViT, used to extract handwritten Chinese character image features: Where: H represents the height of the image, W represents the width of the image, and C represents the channel dimension; In order to use a one-dimensional vector to represent the handwritten Chinese character image features, an end identifier is added to the end of the image block sequence; since the end identifier can focus on the features of all the previous image blocks, the end identifier can represent the global features of the image. represents the projection matrix and C′ is the alignment dimension.

5. According to the multimodal handwritten Chinese character error correction network architecture based on contrastive learning as described in claim 1, it is characterized in that: The text encoder includes: 12 layers of Transformer encoders and 1 embedding layer, through which the ideographic character description sequence is encoded as: The text encoder also projects the ideographic description sequence features into represents the projection matrix; Among them: T is the feature output after the ideographic description sequence passes through the text encoder, and the feature after being projected into the unified feature space; D is the "dimension" of the feature output after the ideographic description sequence passes through the text encoder.

6. According to the multimodal handwritten Chinese character error correction network architecture based on contrastive learning as described in claim 4, it is characterized in that: Using contrastive loss function Aligning the handwritten Chinese character image features and the ideographic character description sequence features; For a batch of N training samples, the contrast loss function The design is as follows: Where: I j , T j They respectively represent the handwritten Chinese character image features and ideographic description sequence features of the j-th sample in the image block. In order to adapt to the problem of the same Chinese character samples in the same image block, the subsequent labels are aligned with the label at the position where the Chinese character first appeared during calculation.

7. The multimodal handwritten Chinese character error correction network architecture based on contrastive learning according to claim 1, characterized in that: For each handwritten Chinese character image-ideographic description sequence pair, the ideographic description sequence generation module is trained using the loss of the multimodal language model, and the loss formula is designed as: Where: I represents the input handwritten Chinese character image features, T i It is an ideographic character description sequence identifier, T0 is the start identifier, T N+1 is the end identifier; CE is the cross entropy loss.

8. The multimodal handwritten Chinese character error correction network architecture based on contrastive learning according to claim 1, characterized in that: The matching includes the following three steps: Step 1: Exact matching; If there is a sequence in the ideographic description sequence dictionary that is completely consistent with the ideographic description sequence of the target character, the target character is classified as a "correct character"; if there is no completely consistent sequence, the second step is entered; Step 2: Unordered matching; If there is a sequence in the ideographic description sequence dictionary, each element of which is identical to the ideographic description sequence of the target character, but only the order is different, then the target character is classified as a "disordered character", and the sequence is the correct character corresponding to the target character; Step 3: If no result can be obtained in the unordered matching, the ideographic description sequence of the target character is input into the text encoder, and the output ideographic description sequence features are compared with the ideographic description sequence features in the ideographic description sequence dictionary by cosine similarity, and the character with the largest cosine similarity is selected as the correct character of the target sequence and classified as a "radical error character".

Images