Multi-stage ideographic image classification method based on contrastive learning of glyphs and semantics

Abstract

The present invention relates to a multi-stage ideographic character image classification method based on glyph and semantic contrast learning, and relates to the technical field of ideographic character recognition, comprising: inputting an original character image into a structured recognition network to extract the glyph features of the input image; wherein the structured recognition model is obtained based on training a convolutional neural network model; label encoding the original character image; utilizing a visual semantic encoding network to extract the image features of the original character image, and mapping and aligning the image features with the label encoding to obtain the semantic features of the input image; wherein the visual semantic encoding network is obtained based on training a convolutional neural network model; fusing the glyph features and semantic features, integrating glyph and semantic information, and obtaining a character classification result. The present invention addresses the challenges of Chinese character glyph complexity and long-tail distribution, and its core technology breakthrough is achieved through confusion community contrast learning and semantic feature alignment, which significantly improves the recognition performance of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of ideogram recognition, and particularly to a multi-stage ideogram image classification method based on glyph and semantic contrast learning. Background Art

[0002] Current deep learning-based ideogram recognition technologies (such as Chinese character OCR, oracle bone script digitization, etc.) mainly rely on convolutional neural networks to construct a mapping relationship from pixels to character categories. Such methods learn the visual feature representations of characters through end-to-end training and have achieved high recognition accuracies on balanced datasets. However, the characteristics of the ideogram system (such as the existence of easily confused characters like "chicken - bird - crow" in Chinese characters, and the long-tailed distribution of high-frequency and rare characters) pose severe challenges to the existing technologies: on the one hand, it is difficult to distinguish characters with highly similar structures solely relying on visual features; on the other hand, traditional methods cannot effectively alleviate the problem of insufficient representation learning of low-frequency characters.

[0003] The existing technologies have the following key defects: (1) Insufficient utilization of semantic information: The glyph design of ideograms contains semantic information (such as characters with the "wood" radical are mostly related to plants), but existing visual models have not effectively integrated such semantic features to form a "form - meaning" association on the visual model, that is, the feature images of semantically similar characters may not be close. (2) Mismatch in feature learning granularity: For easily confused characters (such as "end - not yet", "self - already"), existing methods have not systematically established a contrast learning mechanism for easily confused characters, resulting in the feature distances of easily confused characters not being fully optimized. Summary of the Invention

[0004] The purpose of the present invention is to propose a multi-stage ideogram image classification method based on glyph and semantic contrast learning to solve the problems existing in the above-mentioned existing technologies. In response to the challenges of the complexity of Chinese character glyphs (such as similar-shaped characters like "self - already - oneself") and the long-tailed distribution, the core achieves a technical breakthrough through confusion community contrast learning and semantic feature alignment, significantly improving the recognition performance of the model.

[0005] To achieve the above purpose, the present invention provides the following solutions:

[0006] A multi-stage ideogram image classification method based on glyph and semantic contrast learning, comprising:

[0007] Input the original text image into a structured recognition model to extract the glyph features of the input image; wherein, the structured recognition model is obtained by training a convolutional neural network model.

[0008] Perform label encoding on the original text image.

[0009] Extracting image features of the original text image using a visual semantic coding network, and mapping and aligning the image features with the label code to obtain semantic features of the input image; wherein the visual semantic coding network is constructed based on a convolutional neural network model;

[0010] The glyph features and semantic features are fused to integrate glyph and semantic information to obtain a text classification result.

[0011] Optionally, training a convolutional neural network model to obtain the structured recognition model includes:

[0012] Preprocess the character image samples and construct a training set;

[0013] Based on the training set, a convolutional neural network model is trained using a classification loss function to obtain a classification model; wherein the classification model includes a backbone network and a fully connected layer, and the backbone network is an original structured recognition network;

[0014] Using the classification model, obtaining a confusing word list on the training set;

[0015] Based on the obfuscated word table, constructing an obfuscated word graph;

[0016] Performing community mining on the obfuscated word graph using a community mining algorithm to obtain a set of obfuscated communities;

[0017] Based on the confusion community set, comparative learning is performed to obtain the structured recognition model.

[0018] Optionally, preprocessing the character image sample includes: size normalization, skeletonization and etching, and foreground extraction;

[0019] The size normalization is as follows: all character images are adjusted to a uniform size of 64×64 pixels;

[0020] The skeletonization etching is as follows: applying a skeletonization algorithm to remove the thickness of the text, and retaining a text skeleton of a preset width of pixels as the foreground;

[0021] The foreground extraction is to separate the foreground part containing only text information from the processed image and discard the background information.

[0022] Optionally, using the classification model to obtain the confusion word table on the training set includes:

[0023] For each category in the training set, the classification model is used to perform classification prediction to obtain a probability distribution vector for each category, wherein each component represents the confidence level of each category;

[0024] For each category, calculate the average confidence vector of all samples;

[0025] Remove the confidence of each category from the average confidence vector of the sample calculated for each category and obtain the confidence of the remaining categories;

[0026] Select the top several categories with the highest confidence scores from the remaining categories as the candidate sets of easily confused categories for the corresponding categories;

[0027] Integrate the several easily confused categories of each category screened out to form a complete confusing word list.

[0028] Optionally, constructing a confusing word graph based on the confusing word table includes:

[0029] Converting the obfuscated word table into an undirected graph; the undirected graph includes: a node set and an edge set;

[0030] Each category is regarded as a node in an undirected graph, and the set of categories is the node set;

[0031] The obfuscated word table is traversed, and an undirected edge is established between each pair of categories and the corresponding easily confused category to construct the edge set.

[0032] Optionally, performing comparative learning based on the confusion community set to obtain the structured recognition model includes:

[0033] For each community in the confusion community set, extract specific samples from each category in the community to form a confusion subset;

[0034] A batch of samples is sampled from each confusion subset, and after being processed by the classification model backbone (original structured recognition model) and the linear projection layer proj, the confusion features are obtained;

[0035] For the confusion features obtained from all samples in a batch, the same category in the same community is used as positive samples, and different categories in the community are used as negative samples, and the category-level contrastive learning loss is calculated;

[0036] Based on contrastive learning loss, the confused subset is used to train the backbone network of the classification model to obtain the structured recognition model; when all samples in the confused subset participate in the training, other untraversed confused subsets are selected for training, and one round of training is completed until all confused subsets participate in the training.

[0037] Optionally, a pre-trained word embedding model is used to perform label encoding on the original text image.

[0038] Optionally, mapping and aligning the image features with the label encoding includes:

[0039] The label encoding is compared with the image features to calculate the learning loss, realizing the joint mapping of visual features and text semantic space.

[0040] Optionally, the label encoding is compared with the image features. The learning loss calculation includes:

[0041] Calculate the image similarity between the image features:

[0042] Calculating text similarity between the label encodings;

[0043] The MSE loss is calculated for the image similarity and the text similarity.

[0044] Optionally, fusing the glyph features and semantic features includes:

[0045] splicing the glyph features and semantic features into a joint feature vector;

[0046] The joint feature vector is fine-tuned using a fully connected layer.

[0047] The beneficial effects of the present invention are:

[0048] Traditional methods rely on visual features and have difficulty distinguishing highly similar characters. However, the present invention can effectively optimize the feature distance between easily confused characters through confusion community comparative learning, significantly improving the model's ability to distinguish similar characters.

[0049] To address the long-tail distribution between high-frequency and low-frequency characters, this paper uses a community-independent sampling strategy in contrastive learning to increase the exposure of low-frequency characters during training, thereby improving their representation quality. Furthermore, semantic knowledge is introduced during the semantic feature alignment phase, enhancing the model's understanding of low-frequency characters.

[0050] Existing visual models fail to effectively integrate semantic features, resulting in poor performance when processing complex characters. This paper uses semantic feature alignment to jointly map text semantic features with image features, enhancing the model's understanding of character meaning, especially when context and semantic reasoning are required. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1Schematic diagram of the flow of a multi-stage ideographic character image classification method based on contrastive learning of glyphs and semantics according to an embodiment of the present invention;

[0053] Figure 2 Schematic diagram of the character feature learning stage according to an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of the confusion community comparison learning stage in an embodiment of the present invention;

[0055] Figure 4 Schematic diagram of the semantic feature alignment stage according to an embodiment of the present invention;

[0056] Figure 5 This is a flow chart of the dual-stream fusion classification stage in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] like Figure 1 As shown, this embodiment proposes a multi-stage ideographic character image classification method based on character shape and semantic comparison learning, including:

[0060] Input the original text image into the structured recognition model to extract the glyph features of the input image; wherein the structured recognition model is obtained based on the training of the convolutional neural network model;

[0061] Label encoding of the original text image;

[0062] The visual semantic coding network is used to extract the image features of the original text image and map and align the image features with the label code to obtain the semantic features of the input image. The visual semantic coding network is built based on a convolutional neural network.

[0063] The glyph features and semantic features are fused, the glyph and semantic information are integrated, and the text classification results are obtained.

[0064] Specifically, the core process in this embodiment includes: First, through glyph feature learning, the input image is preprocessed and standardized, and the skeleton features are extracted, and the confusion relationship within the current data is automatically mined to form a confusion character table; Second, combined with confusion community contrast learning, specialized contrast learning is performed on the easily confused categories to strengthen the distinguishability of the easily confused features; Subsequently, through semantic feature alignment, the joint mapping of visual features and text semantic space is realized to alleviate the long-tailed distribution bias; Finally, based on the dynamic decision-making mechanism of two-stream feature fusion, the glyph and semantic information are integrated to output the classification result. The four-level progressive pre-training method for ideographic character recognition models proposed in this embodiment addresses the challenges of the complexity of Chinese character glyphs (such as similar-shaped characters like "己-已-巳") and long-tailed distribution. The core achieves technological breakthroughs through confusion community contrast learning and semantic feature alignment, significantly improving the recognition performance of the model.

[0065] Furthermore, training the convolutional neural network model to obtain a structured recognition model includes:

[0066] Preprocess the character image samples to construct a foreground-based training set;

[0067] Based on the foreground training set, use the classification loss function to train the convolutional neural network model to obtain a classification model; where the classification model includes a backbone network and a fully connected layer, and the backbone network is the original structured recognition model;

[0068] Use the classification model to obtain the confusion character table on the foreground training set;

[0069] Based on the confusion character table, construct a confusion character graph;

[0070] Use the community mining algorithm to perform community mining on the confusion character graph to obtain a set of confusion communities;

[0071] Based on the set of confusion communities, perform contrast learning to obtain a structured recognition model.

[0072] Among them, the visual semantic encoding network is also a convolutional neural network, and the training process is the process of aligning with the label encoding. The specific process is as Figure 4 shown. An untrained convolutional neural network is used as the visual semantic encoding network SEN. The similarities within the encoded batch are calculated with each other, and then the similarities calculated with the corresponding label encoding are used to calculate the MSE loss. This is equivalent to introducing a distribution relationship similar to the label encoding in the image encoding space to indirectly perform semantic alignment.

[0073] Furthermore, preprocessing the character image samples includes: size standardization, skeleton etching, and foreground extraction.

[0074] Specifically, in this embodiment, in the glyph feature learning stage, as Figure 2As shown in the figure, the main purpose is to automatically generate a confusion word table for the training dataset. Specifically, this stage includes the following processes:

[0075] First, for each character image to be recognized, perform the following operations to prepare the dataset for training;

[0076] Size normalization: resize all character images to a uniform size of 64×64 pixels to ensure consistency of model input. This step can be achieved through interpolation algorithms;

[0077] Skeletonization Etching: Then, a skeletonization algorithm is applied to remove the thickness of the text until a 3-pixel-wide text skeleton is retained as the foreground. This process helps reduce the impact of font style changes and allows the model to focus more on the basic form of the characters. Skeletonization can be done using the erosion algorithm in morphological operations. The skeletonization process steps are as follows:

[0078] Perform the following operations on the input text image:

[0079] Grayscale conversion and binarization: The OTSU algorithm (threshold 150) is used to generate a binary image, where the text area is 1 and the background is 0;

[0080] Single-pixel skeleton extraction: Generate the initial skeleton based on the morphological skeletonization algorithm;

[0081] Skeleton enhancement: Use a circular kernel with a diameter of 1 pixel to perform an expansion operation to repair broken skeletons;

[0082] The formal expression is: , where S represents the result of skeletonization, () is the skeleton extraction and enhancement operation, For images that have been size-standardized, The width of the skeleton after enhancement (width is 3).

[0083] Foreground extraction: Finally, the foreground portion containing only text information is separated from the processed image while discarding background information to simplify subsequent processing steps and highlight key features.

[0084] Specifically, in this embodiment, based on the foreground training set, a classification loss function is used to train the convolutional neural network model, and obtaining the classification model includes:

[0085] After completing the above preprocessing steps, this example obtains a series of images containing only character skeletons as input to the SRN. These images undergo preprocessing steps such as size normalization, skeletonization, and foreground extraction to remove irrelevant background information, thereby focusing on differences in character structure and improving classification accuracy.

[0086] In this example, the SRN uses a standard, untrained ResNet18 model (excluding the final fully connected layer; ResNet18 can also be replaced with other networks). Because the input data has been streamlined and preprocessed to contain only the skeleton information of the characters, the SRN is able to more effectively learn the subtle structural features between characters. Specifically:

[0087] Input data: preprocessed character images, which have been resized, skeletonized, and foreground extracted to remove unnecessary background information.

[0088] Model architecture: The standard ResNet18 architecture is used (ResNet18 can also be replaced by other networks), but no pre-trained weights are used and training is started directly from scratch.

[0089] Training goal: Use a classification loss function (such as cross-entropy loss) to train a classification model (SRN+FC) to effectively distinguish the structural features of different characters. FC is the classification head (fully connected layer) of the classification model.

[0090] Furthermore, using the classification model (SRN+FC), the confusion word table on the foreground training set is obtained, including:

[0091] For each category in the training set, the classification model is used to make classification predictions and obtain the probability distribution vector of each category; each component represents the confidence level of each category;

[0092] For each category, calculate the average confidence vector of all samples;

[0093] Remove the confidence of each category from the average confidence vector of the sample calculated for each category and obtain the confidence of the remaining categories;

[0094] Select the top several categories with the highest confidence scores from the remaining categories as the candidate sets of easily confused categories for the corresponding categories;

[0095] Integrate the several easily confused categories of each category screened out to form a complete confusing word list.

[0096] Specifically, in this embodiment, the purpose of the confusable character table construction phase is to automatically evaluate the easily confusing relationships between categories in the current dataset and generate a confusable character table (CCT) for subsequent targeted comparative learning. The specific steps are as follows:

[0097] For each category in the training set in , is the total number of categories, A sample j under category is , use the structured recognition network (SRN) trained in step 2 to perform classification prediction and obtain the probability distribution vector of each category, where each component represents the confidence of each category. , where FC is the classification head (fully connected layer) of the classification model, x j is a sample j in the training data.

[0098] For each category, calculate the average confidence vector of all samples, , here Representation category The number of samples in is the probability distribution vector calculated according to step 1 above.

[0099] Filter out confused categories and calculate the average confidence vector of samples from each category In the example, remove the confidence of itself and get , select the top ten categories with the highest confidence among the remaining categories as the easily confused category candidate set of this category , .

[0100] Construct a confusing word table and list the top 10 easily confused categories of each category filtered out above Integrate to form a complete Confusable Characters Table (CCT). .

[0101] Through this series of steps, this embodiment can automatically evaluate the confusing relationships of various categories in the current dataset and generate a table of similar characters, providing a basis for subsequent confusing community comparative learning.

[0102] Furthermore, based on the confusion community set, comparative learning is performed to obtain a structured recognition model including:

[0103] For each community in the confusion community set, specific samples are extracted from each category in the community to form a confusion subset;

[0104] A batch of samples is sampled from each confusion subset and processed by the backbone network of the classification model, i.e. the original structured recognition model, to obtain the confusion features.

[0105] For the confusion features obtained from all samples in a batch, the same category in the same community is used as positive samples, and different categories in the community are used as negative samples, and the category-level contrastive learning loss is calculated;

[0106] Based on contrastive learning loss, the original structured recognition model is trained using the confusion subset to obtain the structured recognition model. After all samples in the confusion subset participate in the training, other confusion subsets that have not been traversed are selected for training. After all confusion subsets participate in the training, one round of training is completed.

[0107] Specifically, in this embodiment, Figure 3 As shown in Figure 2, the confusion community comparison learning phase specifically includes:

[0108] Constructing the obfuscated word graph:

[0109] First, convert the obfuscated word table into an undirected graph. represents an undirected graph, where is a collection of nodes, is the edge set.

[0110] Node collection: Each category As a node in the graph, the node set .

[0111] Edge set: For each category , which corresponds to an entry in the obfuscated word table , indicating the top 10 easily confused categories of the category. Traverse the confusion word table CCT, for each pair of categories and its easily confused categories ,exist and Create an undirected edge between ,Right now:

[0112] ;

[0113] According to experiments, under normal circumstances, the confidence differences between easily confused characters are small, so the confused characters of each character are not sorted or distinguished, but are all uniformly considered to be easily confused characters.

[0114] Community mining of obfuscated characters:

[0115] Next, we use existing community mining algorithms (such as the Girvan-Newman algorithm, Louvain method, or spectral clustering) to mine closely connected subclusters in the confusion graph. The categories within each cluster are highly connected and more difficult for the model to distinguish. In this invention, these subclusters are called confusion communities. The specific process is as follows:

[0116] Use the Louvain method to perform community detection on the confusing word graph obtained in step 1 to obtain all closely connected community sets. ;

[0117] Represents the set of all confused communities, where each community It is a subset composed of multiple categories, m=1, 2, ..., M, where M is the number of confusion communities. In the present invention, the Louvain algorithm divides the original data set into 21 disjoint sets.

[0118] Contrastive learning based on confusion communities. After obtaining communities, category-level contrastive learning is performed for each community. To ensure that the training samples can fully reflect the confusion characteristics between characters within the community, the sampling strategy and specific training process are as follows:

[0119] Community-independent sampling strategy:

[0120] For each community , extract specific samples from each category in the community to form a confusion subset :

[0121] ;

[0122] in is a category The set of all samples of is a category All the confusion subsets together constitute the training set S, which is recorded as:

[0123] ;

[0124] Feature extraction and processing:

[0125] For a batch of samples sampled from each confusion subset, the features are obtained after SRN (Structure Recognize Network) and linear projection layer proj processing :

[0126] ;

[0127] Where: B is the current community The batch size is set to 128 in this paper. The SRN is a structured recognition network used to extract the glyph features of the characters (the same as the SRN used in the glyph feature learning phase). proj() represents a linear projection layer used to reduce the 512-dimensional feature vector output by the SRN to 128 dimensions. In this paper, the dimension of proj() is 512×128.

[0128] Category-level contrastive learning loss:

[0129] The features obtained for all samples in a batch , taking the same categories within the same community as positive samples and different categories within the community as negative samples (mined similar characters as negative samples) to calculate the category-level contrastive learning loss, specifically as follows:

[0130] Feature Normalization: , yes The L2 norm of ; where, Represents the characteristics of a sample i in the batch;

[0131] Similarity calculation: , represents the similarity score between sample i and sample j; where T represents the matrix transpose, represents the characteristics of sample j that are different from sample i, represents the temperature parameter used to smooth the distribution;

[0132] Mask matrix: ;in, represents the label of sample i, represents the label of sample j;

[0133] Loss calculation: .

[0134] Training rounds:

[0135] when After all the samples in participate in the training, select other confusion subsets that have not been traversed for training, until all confusion subsets participate in the training, which is considered to be a completed round of training.

[0136] After this stage, we get a SRN (Structure Recognize Network) that has been well pre-trained on the confusable graph and can more accurately identify categories with similar glyph structures.

[0137] Furthermore, a pre-trained word embedding model is used to perform label encoding on the original text image.

[0138] Mapping and aligning image features with label encodings involves:

[0139] The label encoding is compared with the image features to calculate the learning loss, realizing the joint mapping of visual features and text semantic space.

[0140] Comparing the label encoding with the image features, the learning loss calculation includes:

[0141] Calculate the image similarity between image features; that is, the cosine similarity between images i and j;

[0142] Calculate the text similarity between label encodings; that is, the cosine similarity between text labels i and j;

[0143] For image similarity and text similarity, calculate the MSE loss.

[0144] The fusion of glyph features and semantic features includes:

[0145] Concatenate glyph features and semantic features into a joint feature vector;

[0146] The joint feature vector is fine-tuned using a fully connected layer.

[0147] Specifically, in this embodiment, Figure 4 As shown in the figure, the main purpose of the semantic feature alignment stage is to align the text encoding features of each image label with the image features extracted from each image. Assume that the text encoder is a Text Semantic Encoder (TSE is usually a pre-trained language model such as BERT), and the SEN (Semantic Encode Network) used to extract image encoding features can have various architectures (this embodiment uses Resnet18 with the same architecture as the SRN in the previous stage, and also does not include the final fully connected layer. ResNet18 can also be replaced by other networks). Specifically, this stage mainly includes the following processes:

[0148] Image feature extraction:

[0149] For each batch of samples X, the image features are extracted through SEN, and the feature dimensions are transformed through a fully connected projection layer proj to obtain the features V. In this invention, the dimension of proj is 512x128 and the batch size is 128.

[0150] Image similarity calculation:

[0151] , the feature V is normalized using the L2 norm.

[0152] , represents the cosine similarity between images i and j. represents the normalized transpose of the sample features extracted from image i, Represents the normalized features of the sample features extracted from image j.

[0153] Semantic feature extraction:

[0154] , where Y represents the text corresponding to the labels of all samples in a batch, It represents a text encoder (TSE) pre-trained on a large amount of text, which extracts semantic features T from text tags. In this invention, TSE uses pre-trained Chinese BERT, and the dimension of the output feature T is 768.

[0155] , use L2 norm to normalize text features. Among them, Represents the normalized label text features.

[0156] , Represents the cosine similarity between text labels i and j. represents the transpose of the normalized text label feature of sample i (the superscript T represents the transpose), represents the normalized feature of text label of sample j.

[0157] Semantic loss calculation:

[0158] , calculate the MSE loss for the similarity calculated by image features and the text similarity extracted by label features. Represents the similarity matrix calculated by all sample text label features in a batch, Represents the similarity matrix calculated from all sample image features in a batch.

[0159] Specifically, in this embodiment, the dual-stream fusion classification stage, such as Figure 5 As shown in Figure 1, this stage combines the SRN (Structure Recognize Network) trained on glyph structure and the SEN (Semantic Encode Network) pre-trained on semantics. The resulting network is named SECN (Semantic Enhanced Confusion Community Contrastive Net). This model uses a two-stream architecture to extract glyph and semantic features separately and fuses these features to achieve more accurate character classification. Specifically:

[0160] Sample a batch:

[0161] Sample a batch of samples from the original dataset , each sample Represents an image with height H, width W and number of channels C. Where i=1, 2, ..., n, n represents the number of samples in a batch.

[0162] Feature extraction:

[0163] Use the trained SRN and SEN to extract features respectively:

[0164] Structural specialization features of SRN: ,in ;in, The dimensional space representing the structural specialization features ( represents the dimension of the vector output by SRN, and n represents the number of samples in a batch).

[0165] Semantic specialization features of SEN: ,in ;in, The dimensional space representing semantically specialized features ( represents the dimension of the vector output by SEN, and n represents the number of samples in a batch).

[0166] Feature fusion:

[0167] Concatenate the outputs of the two branches into a joint feature vector:

[0168] ,in ;in, represents the joint eigenvector, represents the dimension of the vector output by SEN, The dimension of the vector representing the output of SEN.

[0169] Fully connected layer fine-tuning:

[0170] ;

[0171] in: is the fully connected layer weight matrix; is the bias of the fully connected layer; classes is the number of final classifications;

[0172] Classification loss calculation:

[0173] Using classification loss Optimizing the SECN network:

[0174] ;

[0175] Among them, Y is the label corresponding to each feature.

[0176] In addition, at this stage, the parameters of SRN and SEN can be selectively frozen or fine-tuned. The preferred solution is:

[0177] Unfreeze SRN / SEN parameters: Fine-tune the entire network through layer-by-layer learning rate (the learning rate of the backbone layer is set to 1 / 10 of the fully connected layer) and weight decay (coefficient 1e-4) to make the model dynamically adapt to the visual-semantic distribution characteristics of downstream tasks.

[0178] Freeze SRN / SEN parameters: When the target dataset size is less than 10% of the pre-training data, fix the SRN / SEN parameters and only fine-tune the fully connected layer to reduce the risk of overfitting.

[0179] Through this dual-stream architecture, the method SECN (Semantic Enhanced Confusion Community Contrastive Net) of this embodiment can comprehensively utilize glyph and semantic information to significantly improve the accuracy and robustness of character recognition.

[0180] To verify the effectiveness of the multi-stage ideographic image classification pre-training method proposed in this example, experiments were conducted on the HWDB1.1 dataset (containing 3755 classes). A subset was manually partitioned with an imbalance ratio of 100 (i.e., a head-to-tail class sample size ratio of 100:1, with 80% of the training set and 20% of the test set). The method (SECN) of this example was compared with several traditional contrastive learning schemes. The comparison results are shown in Table 1; the ablation results are shown in Table 2.

[0181] Table 1. Comparison of experimental results

[0182] All baseline methods (SimCLR / MoCo, etc.) and SECN-Freezed use the same freezing strategy (freezing the backbone and only training the classification head).

[0183] Table 2. Ablation results

[0184]

[0185] In multiple independent experiments, the average accuracy of SECN is higher than that after removing SEN, and the difference is stable. The ablation experiment is set to fine-tune the entire backbone network.

[0186] SECN-NoCommunity refers to removing the confusing community comparison learning phase and retaining other processes (such as semantic alignment).

[0187] Experimental setup:

[0188] The following are the specific experimental settings for each stage in each SECN:

[0189] In the glyph feature learning stage, the batch size is set to 128 and the training epochs is set to 200.

[0190] In the confusion community contrast learning phase, the batch size is set to 128 and the training epochs is set to 100.

[0191] In the semantic feature alignment stage, the batch size is set to 128 and the training epochs is set to 200.

[0192] During the dual-stream fusion fine-tuning stage, the batch size is set to 128 and the training epochs are set to 200.

[0193] This embodiment proposes a multi-stage ideographic character image classification method based on contrastive learning of glyphs and semantics. It aims to improve the recognition accuracy of ideographic characters (such as Chinese characters) and is particularly suitable for handling complex character structures and long-tail distributions. This method first preprocesses the character images (including size normalization, skeletonization, and foreground extraction). It then uses a structured recognition network (SRN) to learn the glyph features of the characters and automatically generates a table of confusing characters. Next, it constructs confusion communities based on the table and performs targeted contrastive learning to optimize the feature distances between easily confused characters, significantly improving the ability to distinguish similar characters. Furthermore, a text encoder (TSE) and a semantic encoding network (SEN) are used to extract semantic features from text labels and images. Feature alignment is achieved by minimizing the difference in similarity between the two, effectively mitigating the long-tail effect. Finally, the output features of the SRN and SEN are concatenated and fine-tuned through a fully connected layer to integrate glyph and semantic information, further improving the accuracy and robustness of character recognition. Experimental results show that the method (SECN) in this embodiment achieves an accuracy of 83.55% on the HWDB1.1 dataset with an imbalance ratio of 100, significantly outperforming traditional contrastive learning methods on long-tail text datasets. This method has broad application prospects and is suitable for high-precision character recognition tasks such as ancient book digitization, handwriting recognition, and OCR system improvement.

[0194] The multi-stage ideographic character image classification method based on contrastive learning of glyphs and semantics proposed in this embodiment can:

[0195] Solve the difficulty of distinguishing similar characters:

[0196] Traditional methods rely on visual features and have difficulty distinguishing highly similar characters. However, the present invention can effectively optimize the feature distance between easily confused characters through confusion community comparative learning, significantly improving the model's ability to distinguish similar characters.

[0197] Alleviate the long-tail distribution problem:

[0198] To address the long-tail distribution between high-frequency and low-frequency characters, this paper uses a community-independent sampling strategy in contrastive learning to increase the exposure of low-frequency characters during training, thereby improving their representation quality. Furthermore, semantic knowledge is introduced during the semantic feature alignment phase, enhancing the model's understanding of low-frequency characters.

[0199] Make full use of semantic information:

[0200] Existing visual models fail to effectively integrate semantic features, resulting in poor performance when processing complex characters. This paper uses semantic feature alignment to jointly map text semantic features with image features, enhancing the model's understanding of character meaning, especially when context and semantic reasoning are required.

[0201] Improve generalization and robustness:

[0202] The proposed multi-stage ideographic image classification method based on contrastive learning of glyphs and semantics achieved a recognition accuracy of 83.55% on the HWDB dataset with an imbalance ratio of 100. This method outperforms traditional contrastive learning methods on long-tail image datasets (such as SimCLR and MoCo). This demonstrates the method's strong generalization and robustness, making it suitable for diverse application scenarios.

[0203] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A multi-stage ideographic character image classification method based on contrastive learning of glyphs and semantics, characterized by: include: Inputting the original text image into a structured recognition model to extract the glyph features of the input image; wherein the structured recognition model is obtained based on training a convolutional neural network model; and training the convolutional neural network model to obtain the structured recognition model includes: Preprocess the character image samples and construct a foreground-based training set; Based on the foreground training set, a convolutional neural network model is trained using a classification loss function to obtain a classification model; wherein the classification model includes a backbone network and a fully connected layer, and the backbone network is the original structured recognition model; Using the classification model, obtaining a confusing word list on the training set; Based on the obfuscated word table, constructing an obfuscated word graph; Performing community mining on the obfuscated word graph using a community mining algorithm to obtain a set of obfuscated communities; Performing comparative learning based on the confusion community set to obtain the structured recognition model; Performing label encoding on the original text image; Extracting image features of the original text image using a visual semantic coding network, and mapping and aligning the image features with the label codes to obtain semantic features of the input image; wherein the visual semantic coding network is constructed based on a convolutional neural network model; mapping and aligning the image features with the label codes includes: performing comparative learning loss calculation on the label codes and the image features to achieve joint mapping of visual features and text semantic space; The glyph features and semantic features are fused to integrate glyph and semantic information to obtain a text classification result.

2. The multi-stage ideographic character image classification method based on contrastive learning of glyphs and semantics according to claim 1, characterized in that: The preprocessing of character image samples includes size normalization, skeletonization and foreground extraction; The size normalization is as follows: all character images are adjusted to a uniform size of 64×64 pixels; The skeletonization etching is as follows: applying a skeletonization algorithm to remove the thickness of the text, and retaining a text skeleton of a preset width of pixels as the foreground; The foreground extraction is to separate the foreground part containing only text information from the processed image and discard the background information.

3. The multi-stage ideographic character image classification method based on contrastive learning of glyphs and semantics according to claim 1, characterized in that: Using the classification model, obtaining the confusion word table on the training set includes: For each category in the training set, the classification model is used to perform classification prediction to obtain a probability distribution vector for each category, wherein each component represents the confidence level of each category; For each category, calculate the average confidence vector of all samples; Remove the confidence of each category from the average confidence vector of the sample calculated for each category and obtain the confidence of the remaining categories; Select the top several categories with the highest confidence scores from the remaining categories as the candidate sets of easily confused categories for the corresponding categories; Integrate the several easily confused categories of each category screened out to form a complete confusing word list.

4. The multi-stage ideographic character image classification method based on contrastive learning of glyphs and semantics according to claim 3, characterized in that: Based on the obfuscated word table, constructing an obfuscated word graph includes: Converting the obfuscated word table into an undirected graph; the undirected graph includes: a node set and an edge set; Each category is regarded as a node in an undirected graph, and the set of categories is the node set; The obfuscated word table is traversed, and an undirected edge is established between each pair of categories and the corresponding easily confused category to construct the edge set.

5. The multi-stage ideographic character image classification method based on contrastive learning of glyphs and semantics according to claim 1, characterized in that: Performing comparative learning based on the confusion community set to obtain the structured recognition model includes: For each community in the confusion community set, extract specific samples from each category in the community to form a confusion subset; Sampling a batch of samples from each confusion subset and processing them through the backbone network of the classification model to obtain confusion features; For the confusion features obtained from all samples in a batch, the same category in the same community is used as positive samples, and different categories in the community are used as negative samples, and the category-level contrastive learning loss is calculated; Based on contrastive learning loss, the confused subset is used to train the backbone network of the classification model to obtain the structured recognition model. After all samples in the confused subset participate in the training, other untraversed confused subsets are selected for training. After all subcommunities participate in the training, one round of training is completed.

6. The multi-stage ideographic character image classification method based on contrastive learning of glyphs and semantics according to claim 1, characterized in that: The original text image is label-encoded using a pre-trained word embedding model.

7. The multi-stage ideographic character image classification method based on contrastive learning of glyphs and semantics according to claim 1, characterized in that: Comparing the label encoding with the image features, the learning loss calculation includes: Calculate the image similarity between the image features: Calculating text similarity between the label encodings; The MSE loss is calculated for the image similarity and the text similarity.

8. The multi-stage ideographic character image classification method based on contrastive learning of glyphs and semantics according to claim 1, characterized in that: The fusion of the glyph features and the semantic features includes: splicing the glyph features and semantic features into a joint feature vector; The joint feature vector is fine-tuned using a fully connected layer.

Images