Character image processing method, device and ancient Chinese book image recognition method
By clustering and training the single-character image in ancient Chinese books and training on the classification model of small samples, the problem of handwritten character recognition is solved, efficient recognition and labeling is achieved, and the difficulty of labeling is reduced.
Patent Information
- Application Number
- CN202010366808.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-04-30
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2040-04-30
AI Technical Summary
The prior art is difficult to effectively identify and label handwritten character images in ancient Chinese books, especially because the handwritten fonts are complex and the number of different characters are numerous, resulting in poor recognition effect and increasing the difficulty of labeling.
By clustering single-character characters images, filtering high-frequency characters, building a classification model for few samples, and using twin networks and residual neural networks with weight sharing for training, the annotation difficulty is reduced and the recognition accuracy is improved.
It realizes the annotation of characters images of ancient Chinese books with less manual participation, reduces the difficulty of labeling, improves the recognition effect, and can more effectively identify single-character characters in ancient Chinese books.
Smart Images
Figure CN113591857B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and more particularly to a character image processing method, device, electronic device and storage device, and also to a method for recognizing ancient Chinese book images. The present application also relates to two methods for recognizing text images. Background Art
[0002] Currently, a large number of ancient Chinese books are preserved in the form of physical copies or scanned copies. This method is not conducive to the protection of ancient books, and it cannot integrate ancient books into information scenarios. For example, it is impossible to search, extract information, build knowledge graphs, etc., which limits the research work of ancient books.
[0003] Under the existing technology, the basic implementation idea of the more advanced text recognition model is: in the detection stage, the training data required by the detection model is manually labeled, and the line detection model is trained using detection frameworks such as Mask-RCNN, EAST, PixelLink, etc. with the strength of line labeling; in the recognition stage, the line training data is generated by automatic generation method, and the line recognition model is trained using CNN+RNN+CTC or CNN+RNN+Attention method.
[0004] However, since most ancient books are handwritten, the effect of using existing fonts to generate them is often very poor, and the font has a great impact on the recognition model. In addition, there are a large number of variant characters in ancient books, which cannot be generated using existing fonts. Therefore, how to reduce the difficulty of annotating character images of ancient Chinese books and obtain a single-character character image classification model for character images of ancient Chinese books has become a technical problem that needs to be solved urgently. Summary of the invention
[0005] The present application provides a character image processing method to reduce the difficulty of labeling character images of ancient Chinese books.
[0006] The present application provides a character image processing method, comprising:
[0007] Perform clustering operation on single-word character images to obtain clustering results;
[0008] According to the frequency of occurrence of characters, high-frequency single-word character images are screened out from the clustering results, and characters corresponding to the high-frequency single-word character images are used as labels of the high-frequency single-word character images;
[0009] Obtaining a few-sample classification model according to the high-frequency single-word character image and the label of the high-frequency single-word character image;
[0010] Using the few-sample classification model to identify the single-word character image, and obtaining a label of the single-word character image;
[0011] A single-character character image classification model is obtained according to the single-character character image and the label of the single-character character image.
[0012] Optionally, the method further includes:
[0013] The single-word character image classification model is used to recognize the single-word character image to be recognized, and the character corresponding to the single-word character image to be recognized is obtained.
[0014] Optionally, the method further includes:
[0015] Obtaining a text set image to be recognized;
[0016] Perform target detection on the image of the text set to be recognized to obtain a single-word character image.
[0017] Optionally, performing a clustering operation on the single-word character image to obtain a clustering result includes:
[0018] Get a pre-trained neural network;
[0019] Utilizing the pre-trained neural network, obtaining single-word feature data of the single-word character image;
[0020] A clustering operation is performed on the single-word feature data to obtain a clustering result.
[0021] Optionally, the method further includes:
[0022] Using the clustering results as pseudo labels within the clustering clusters;
[0023] Using the pseudo-label and the single-word character image corresponding to the pseudo-label, the pre-trained neural network is updated to obtain an updated neural network;
[0024] The updated neural network is used to obtain single-word feature data of the single-word character image.
[0025] Optionally, the step of selecting a high-frequency single-word character image from the clustering result according to the frequency of occurrence of the characters, and using the character corresponding to the high-frequency single-word character image as a label of the high-frequency single-word character image includes:
[0026] According to the frequency of occurrence of characters, high-frequency single-word character images are screened out from the clustering results;
[0027] Recognize the high-frequency single-word character image according to expert knowledge to obtain the character corresponding to the high-frequency single-word character image;
[0028] The character corresponding to the high-frequency single-word character image is used as a label of the high-frequency single-word character image.
[0029] Optionally, obtaining a few-sample classification model according to the high-frequency single-word character image and the label of the high-frequency single-word character image includes:
[0030] Build an initial few-shot classification model;
[0031] According to the high-frequency single-word character image and the label of the high-frequency single-word character image, the initial few-sample classification model is trained using a weight-sharing twin network to obtain a few-sample classification model.
[0032] Optionally, the using the few-sample classification model to recognize the single-word character image to obtain a label of the single-word character image includes:
[0033] Using the few-sample classification model to recognize the single-word character image, and obtain a classification result of the single-word character image;
[0034] Determining whether the classification result of the single-word character image matches the clustering result;
[0035] If there is a match, the classification result of the matched single-word character image is used as the label of the single-word character image.
[0036] Optionally, the method further comprises:
[0037] Screening the classification results of the single-word character images to obtain single-word character images with low confidence classification results;
[0038] Recognize the single-word character image with the low-confidence classification result according to expert knowledge to obtain the character corresponding to the single-word character image with the low-confidence classification result;
[0039] The few-sample classification model is updated using the characters corresponding to the single-word character images with the low-confidence classification results and the single-word character images with the low-confidence classification results.
[0040] Optionally, obtaining a single-character character image classification model according to the single-character character image and the label of the single-character character image includes:
[0041] Build an initial single-word character image classification model;
[0042] According to the single-character character image and the label of the single-character character image, the initial single-character character image classification model is trained using a residual neural network to obtain a single-character character image classification model.
[0043] The present application provides a character image processing device, comprising:
[0044] A clustering execution unit, used for performing a clustering operation on the single-word character image to obtain a clustering result;
[0045] An image screening unit, used for screening out high-frequency single-word character images from the clustering results according to the frequency of occurrence of the characters, and using the characters corresponding to the high-frequency single-word character images as labels of the high-frequency single-word character images;
[0046] A model acquisition unit, used to obtain a few-sample classification model according to the high-frequency single-word character image and the label of the high-frequency single-word character image;
[0047] A label obtaining unit, used to recognize the single-word character image by using the few-sample classification model to obtain a label of the single-word character image;
[0048] The classification model obtaining unit is used to obtain a single-character character image classification model according to the single-character character image and the label of the single-character character image.
[0049] Optionally, the character image processing device further includes a recognition unit, wherein the recognition unit is used to:
[0050] The single-word character image to be recognized is recognized using the character image classification model to obtain the character corresponding to the single-word character image to be recognized.
[0051] Optionally, the character image processing device further includes a detection unit, wherein the detection unit is used to:
[0052] Obtaining a text image to be recognized;
[0053] Perform target detection on the text image to be recognized to obtain a single-word character image.
[0054] Optionally, the clustering execution unit is specifically used to:
[0055] Get a pre-trained neural network;
[0056] Utilizing the pre-trained neural network, obtaining single-word feature data of the single-word character image;
[0057] A clustering operation is performed on the single-word feature data to obtain a clustering result.
[0058] Optionally, the clustering execution unit is further used for:
[0059] Using the clustering results as pseudo labels within the clustering clusters;
[0060] Using the pseudo-label and the single-word character image corresponding to the pseudo-label, the pre-trained neural network is updated to obtain an updated neural network;
[0061] The updated neural network is used to obtain single-word feature data of the single-word character image.
[0062] Optionally, the image screening unit is specifically used to:
[0063] According to the frequency of occurrence of characters, high-frequency single-word character images are screened out from the clustering results;
[0064] Recognize the high-frequency single-word character image according to expert knowledge to obtain the character corresponding to the high-frequency single-word character image;
[0065] The character corresponding to the high-frequency single-word character image is used as a label of the high-frequency single-word character image.
[0066] Optionally, the model acquisition unit is specifically used to:
[0067] Build an initial few-shot classification model;
[0068] According to the high-frequency single-word character image and the label of the high-frequency single-word character image, the initial few-sample classification model is trained using a weight-sharing twin network to obtain a few-sample classification model.
[0069] Optionally, the label obtaining unit is specifically used to:
[0070] Using the few-sample classification model to recognize the single-word character image, and obtain a classification result of the single-word character image;
[0071] Determining whether the classification result of the single-word character image matches the clustering result;
[0072] If there is a match, the classification result of the matched single-word character image is used as the label of the single-word character image.
[0073] Optionally, the label obtaining unit is further used to:
[0074] Screening the classification results of the single-word character images to obtain single-word character images with low confidence classification results;
[0075] Recognize the single-word character image with the low-confidence classification result according to expert knowledge to obtain the character corresponding to the single-word character image with the low-confidence classification result;
[0076] The few-sample classification model is updated using the characters corresponding to the single-word character images with the low-confidence classification results and the single-word character images with the low-confidence classification results.
[0077] Optionally, the classification model obtaining unit is specifically used to:
[0078] Build an initial single-word character image classification model;
[0079] According to the single-character character image and the label of the single-character character image, the initial single-character character image classification model is trained using a residual neural network to obtain a single-character character image classification model.
[0080] The present application provides an electronic device, including:
[0081] Processor; and
[0082] A memory is used to store a program of the data processing method. After the device is powered on and the program of the single-character image recognition method is run by the processor, the following steps are performed:
[0083] Perform clustering operation on single-word character images to obtain clustering results;
[0084] According to the frequency of occurrence of characters, high-frequency single-word character images are screened out from the clustering results, and characters corresponding to the high-frequency single-word character images are used as labels of the high-frequency single-word character images;
[0085] Obtaining a few-sample classification model according to the high-frequency single-word character image and the label of the high-frequency single-word character image;
[0086] Using the few-sample classification model to identify the single-word character image, and obtaining a label of the single-word character image;
[0087] A single-character character image classification model is obtained according to the single-character character image and the label of the single-character character image.
[0088] The present application provides a storage device storing a program of a data processing method, wherein the program is executed by a processor to perform the following steps:
[0089] Perform clustering operation on single-word character images to obtain clustering results;
[0090] According to the frequency of occurrence of characters, high-frequency single-word character images are screened out from the clustering results, and characters corresponding to the high-frequency single-word character images are used as labels of the high-frequency single-word character images;
[0091] Obtaining a few-sample classification model according to the high-frequency single-word character image and the label of the high-frequency single-word character image;
[0092] Using the few-sample classification model to identify the single-word character image, and obtaining a label of the single-word character image;
[0093] A single-character character image classification model is obtained according to the single-character character image and the label of the single-character character image.
[0094] The present application also provides a method for recognizing ancient Chinese book images, comprising:
[0095] Obtaining ancient Chinese book images to be identified;
[0096] Performing target detection on the ancient Chinese book image to be identified to obtain a single-word character image;
[0097] The single-word character image is recognized using a single-word character image classification model to obtain a character corresponding to the single-word character image, wherein the single-word character image classification model is obtained according to any one of the aforementioned methods.
[0098] The present application provides a text image recognition method, comprising:
[0099] Obtain the text image to be recognized;
[0100] Using an image recognition model, obtaining a single-word character image of the text image;
[0101] Recognize the single-word character image using a single-word character image classification model to obtain a character corresponding to the single-word character image, wherein the single-word character image classification model is obtained according to any one of the aforementioned methods;
[0102] According to the character corresponding to the single-word character image, text data corresponding to the text image to be recognized is obtained.
[0103] The present application provides a text image recognition method, comprising:
[0104] Obtain the text image to be recognized;
[0105] Based on the first recognition model, the text image is recognized as a whole to obtain a first recognition result;
[0106] Segmenting the text image to obtain single-word character images of the text image;
[0107] Based on the second recognition model, the single character image is recognized to obtain a second recognition result;
[0108] Based on the first recognition result and the second recognition result, text data of the text image is obtained.
[0109] Compared with the prior art, this application has the following advantages:
[0110] The present application provides a character image processing method, including: performing a clustering operation on a single-character character image to obtain a clustering result; filtering out a high-frequency single-character character image from the clustering result according to the frequency of occurrence of the character, and using the character corresponding to the high-frequency single-character character image as the label of the high-frequency single-character character image; obtaining a few-sample classification model according to the high-frequency single-character character image and the label of the high-frequency single-character character image; using the few-sample classification model to identify the single-character character image to obtain the label of the single-character character image; and obtaining a single-character character image classification model according to the single-character character image and the label of the single-character character image. Using the method provided by the present application, the character image is first classified by clustering; then, the character corresponding to the high-frequency single-character character image is used as the label of the high-frequency single-character character image; then, the single-character character image is identified by the few-sample classification model to obtain the label of the single-character character image, thereby achieving the labeling of the character image with less manual participation, thereby reducing the difficulty of labeling character images such as ancient Chinese books. BRIEF DESCRIPTION OF THE DRAWINGS
[0111] Figure 1 It is a schematic diagram of character image processing provided in the first embodiment of the present application.
[0112] Figure 2 It is a flowchart of a character image processing method provided in the first embodiment of the present application.
[0113] Figure 3 It is a schematic diagram of the generation process of a single-word character image classification model involved in the first embodiment of the present application.
[0114] Figure 4 It is a schematic diagram of a character image processing device provided in the second embodiment of the present application.
[0115] Figure 5 It is a schematic diagram of an electronic device provided in the third embodiment of the present application.
[0116] Figure 6 This is a flow chart of a method for recognizing ancient Chinese book images provided in the fifth embodiment of the present application. DETAILED DESCRIPTION
[0117] Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited to the specific implementation disclosed below.
[0118] In order to present the present application more clearly, the application scenario of the character image processing method provided in the first embodiment of the present application is briefly introduced.
[0119] The character image processing method provided in the first embodiment of the present application can be applied to the scene where the client and the server interact with each other, such as Figure 1 When it is necessary to recognize a character image, the client usually first establishes a connection with the server. After the connection is established, the client sends the character image to the server. After the server receives the character image, the server inputs the character image into a single-word character image classification model for recognizing characters from the character image, obtains the character corresponding to the character image, provides the character to the client, and the client receives the recognized character. The single-word classification model is obtained by the following method: performing a clustering operation on the single-word character image to obtain a clustering result; according to the frequency of occurrence of the character, a high-frequency single-word character image is screened out from the clustering result, and the character corresponding to the high-frequency single-word character image is used as the label of the high-frequency single-word character image; according to the high-frequency single-word character image and the label of the high-frequency single-word character image, a few-sample classification model is obtained; the single-word character image is recognized by the few-sample classification model to obtain the label of the single-word character image; according to the single-word character image and the label of the single-word character image, a single-word character image classification model is obtained.
[0120] The first embodiment of the present application provides a character image processing method. Figure 2 as well as Figure 3 Provide explanation.
[0121] like Figure 2 As shown, in step S201, a clustering operation is performed on the single-word character image to obtain a clustering result.
[0122] The data processing method further includes:
[0123] Get text image;
[0124] Perform target detection on the text image to obtain a single-word character image.
[0125] The text image may be an ancient book text image or other types of text images. Figure 3 In actual training, the text image may be a text image contained in an ancient book.
[0126] The single-word character image refers to an image that includes only one character. The character image may include a Chinese character image, an English character image, a digital character image, etc. The Chinese character image may include a simplified Chinese character image and an ancient Chinese character image.
[0127] The characteristics of ancient texts are that the background is relatively simple (basically pure background), all Chinese characters, and almost no numbers, English, or symbols, so it is more suitable for single-word detection in the whole image. CenterNet (a target detection method) can be used for target detection of single-word characters. CenterNet takes the center point of each character as the target and regresses the width and height of the character at the same time.
[0128] like Figure 3 As shown, by performing single character detection on a text image, all single character images contained in the text image can be obtained.
[0129] The performing of clustering operation on the single-word character image to obtain a clustering result includes:
[0130] Get a pre-trained neural network;
[0131] Utilizing the pre-trained neural network, obtaining single-word feature data of the single-word character image;
[0132] A clustering operation is performed on the single-word feature data to obtain a clustering result.
[0133] The data processing method further includes:
[0134] Using the clustering results as pseudo labels within the clustering clusters;
[0135] Using the pseudo-label and the single-word character image corresponding to the pseudo-label, the pre-trained neural network is updated to obtain an updated neural network;
[0136] The updated neural network is used to obtain single-word feature data of the single-word character image.
[0137] The clustering operation can use an unsupervised image clustering method to cluster all words into different classes. There are many unsupervised image clustering methods. In this application, a method combining clustering and deep learning can be used. First, deep learning is used to extract image features, and then traditional clustering methods such as k-means are used. The clustering results are used as pseudo labels, and pseudo labels are used to supervise deep learning to update network parameters. These two processes are iterated in sequence.
[0138] Specifically, in step 1, an approximate number of clustering categories is pre-set, such as 2000, and the features extracted by the Imagenet pre-trained deep network N are clustered using the k-means algorithm, and the cluster centers are used as pseudo labels for samples within the cluster.
[0139] Step 2: Use the pseudo-labeled image samples obtained in step 1 as training data to update the parameters of the deep network N to improve the adaptability of the deep network N to the current data.
[0140] Using the updated deep network in step 2, repeatedly alternate steps 1 and 2 until the deep network classification loss function value no longer decreases within a set number of rounds (such as 5 rounds), and then stop the iteration.
[0141] like Figure 2 As shown, in step S202, high-frequency single-word character images are screened out from the clustering results according to the frequency of occurrence of characters, and the characters corresponding to the high-frequency single-word character images are used as labels of the high-frequency single-word character images.
[0142] The step of selecting a high-frequency single-word character image from the clustering result according to the frequency of occurrence of the characters, and using the character corresponding to the high-frequency single-word character image as a label of the high-frequency single-word character image includes:
[0143] According to the frequency of occurrence of characters, high-frequency single-word character images are screened out from the clustering results;
[0144] Recognize the high-frequency single-word character image according to expert knowledge to obtain the character corresponding to the high-frequency single-word character image;
[0145] The character corresponding to the high-frequency single-word character image is used as a label of the high-frequency single-word character image.
[0146] Select 1 to 5 high-frequency single-word character images with the most occurrences of characters in the category from the clustering results, and use expert knowledge to manually identify the high-frequency single-word character images to obtain the characters corresponding to the high-frequency single-word character images, and use the identified characters corresponding to the high-frequency single-word character images as labels for the high-frequency single-word character images.
[0147] like Figure 3 As shown in 3-3, we manually observe that the character image "A" appears the most in class 1. We select 5 character images containing the character "A" from class 1 and annotate them. We use the same method to annotate each cluster and obtain the manual image annotation results.
[0148] like Figure 2 As shown, in step S203, a few-sample classification model is obtained according to the high-frequency single-word character image and the label of the high-frequency single-word character image.
[0149] The obtaining of a few-sample classification model according to the high-frequency single-word character image and the label of the high-frequency single-word character image comprises:
[0150] Build an initial few-shot classification model;
[0151] According to the high-frequency single-word character image and the label of the high-frequency single-word character image, the initial few-sample classification model is trained using a weight-sharing twin network to obtain a few-sample classification model.
[0152] Siamese network is a small sample classification model. The data processing method provided in this application can use the weight-sharing Siamese network to train the initial few-sample classification model to obtain a few-sample classification model, thereby reducing the training parameters of the few-sample classification model and improving the model training efficiency. Since the Siamese network is a prior art, it will not be described in detail here, and you can refer to the relevant papers for details.
[0153] like Figure 2 As shown, in step S204, the single-word character image is recognized using the few-sample classification model to obtain a label of the single-word character image.
[0154] The step of using the few-sample classification model to recognize the single-word character image to obtain a label of the single-word character image includes:
[0155] Using the few-sample classification model to recognize the single-word character image, and obtain a classification result of the single-word character image;
[0156] Determining whether the classification result of the single-word character image matches the clustering result;
[0157] If there is a match, the classification result of the matched single-word character image is used as the label of the single-word character image.
[0158] The data processing method further includes:
[0159] Screening the classification results of the single-word character images to obtain single-word character images with low confidence classification results;
[0160] Recognize the single-word character image with the low-confidence classification result according to expert knowledge to obtain the character corresponding to the single-word character image with the low-confidence classification result;
[0161] The few-sample classification model is updated using the characters corresponding to the single-word character images with the low-confidence classification results and the single-word character images with the low-confidence classification results.
[0162] First, determine whether the classification result of the single-character character image matches the clustering result. If they match, the classification result of the matching single-character character image is used as the label of the single-character character image. If they do not match, the unmatched single-character character image will be iterated again in step S201. At the same time, the expert annotates the single-character character image with the lowest confidence in each category given by the classification model, combines the single-character character images that have been annotated by the expert before, and then continues the iteration of step S204.
[0163] like Figure 2 As shown, in step S205, a single-character character image classification model is obtained according to the single-character character image and the label of the single-character character image.
[0164] The step of obtaining a single-character character image classification model according to the single-character character image and the label of the single-character character image comprises:
[0165] Build an initial single-word character image classification model;
[0166] According to the single-character character image and the label of the single-character character image, the initial single-character character image classification model is trained using a residual neural network to obtain a single-character character image classification model.
[0167] After obtaining the single-word character image and the label of the single-word character image, the single-word character image and the label are used as training samples to train a residual network (resnet) for single-word character image classification, and obtain a trained single-word character image classification model.
[0168] Corresponding to the character image processing method provided in the first embodiment of the present application, the second embodiment of the present application also provides a character image processing device.
[0169] like Figure 4 As shown, the device comprises:
[0170] The clustering execution unit 401 is used to perform a clustering operation on the single-word character image to obtain a clustering result;
[0171] An image screening unit 402 is used to screen out high-frequency single-word character images from the clustering results according to the frequency of occurrence of the characters, and use the characters corresponding to the high-frequency single-word character images as labels of the high-frequency single-word character images;
[0172] A model acquisition unit 403, configured to acquire a few-sample classification model according to the high-frequency single-word character image and the label of the high-frequency single-word character image;
[0173] A label obtaining unit 404 is used to recognize the single-word character image using the few-sample classification model to obtain a label of the single-word character image;
[0174] The classification model obtaining unit 405 is used to obtain a single-character character image classification model according to the single-character character image and the label of the single-character character image.
[0175] In this embodiment, the character image processing device further includes a recognition unit, and the recognition unit is used to:
[0176] The single-word character image to be recognized is recognized using the character image classification model to obtain the character corresponding to the single-word character image to be recognized.
[0177] In this embodiment, the character image processing device further includes a detection unit, and the detection unit is used to:
[0178] Obtaining a text image to be recognized;
[0179] Perform target detection on the text image to be recognized to obtain a single-word character image.
[0180] In this embodiment, the clustering execution unit is specifically used to:
[0181] Get a pre-trained neural network;
[0182] Utilizing the pre-trained neural network, obtaining single-word feature data of the single-word character image;
[0183] A clustering operation is performed on the single-word feature data to obtain a clustering result.
[0184] In this embodiment, the clustering execution unit is further used for:
[0185] Using the clustering results as pseudo labels within the clustering clusters;
[0186] Using the pseudo-label and the single-word character image corresponding to the pseudo-label, the pre-trained neural network is updated to obtain an updated neural network;
[0187] The updated neural network is used to obtain single-word feature data of the single-word character image.
[0188] In this embodiment, the image screening unit is specifically used for:
[0189] According to the frequency of occurrence of characters, high-frequency single-word character images are screened out from the clustering results;
[0190] Recognize the high-frequency single-word character image according to expert knowledge to obtain the character corresponding to the high-frequency single-word character image;
[0191] The character corresponding to the high-frequency single-word character image is used as a label of the high-frequency single-word character image.
[0192] In this embodiment, the model obtaining unit is specifically used to:
[0193] Build an initial few-shot classification model;
[0194] According to the high-frequency single-word character image and the label of the high-frequency single-word character image, the initial few-sample classification model is trained using a weight-sharing twin network to obtain a few-sample classification model.
[0195] In this embodiment, the label obtaining unit is specifically used to:
[0196] Using the few-sample classification model to recognize the single-word character image, and obtain a classification result of the single-word character image;
[0197] Determining whether the classification result of the single-word character image matches the clustering result;
[0198] If there is a match, the classification result of the matched single-word character image is used as the label of the single-word character image.
[0199] In this embodiment, the label obtaining unit is further used for:
[0200] Screening the classification results of the single-word character images to obtain single-word character images with low confidence classification results;
[0201] Recognize the single-word character image with the low-confidence classification result according to expert knowledge to obtain the character corresponding to the single-word character image with the low-confidence classification result;
[0202] The few-sample classification model is updated using the characters corresponding to the single-word character images with the low-confidence classification results and the single-word character images with the low-confidence classification results.
[0203] In this embodiment, the classification model obtaining unit is specifically used to:
[0204] Build an initial single-word character image classification model;
[0205] According to the single-character character image and the label of the single-character character image, the initial single-character character image classification model is trained using a residual neural network to obtain a single-character character image classification model.
[0206] It should be noted that for the detailed description of the device provided in the second embodiment of the present application, reference can be made to the relevant description of the first embodiment of the present application, which will not be repeated here.
[0207] Corresponding to the character image recognition method provided in the first embodiment of the present application, the third embodiment of the present application also provides an electronic device.
[0208] like Figure 5 As shown, the electronic device comprises:
[0209] Processor 501; and
[0210] The memory 502 is used to store the program of the character image processing method. After the device is powered on and the program of the single character image processing method is run by the processor, the following steps are performed:
[0211] Perform clustering operation on single-word character images to obtain clustering results;
[0212] According to the frequency of occurrence of characters, high-frequency single-word character images are screened out from the clustering results, and characters corresponding to the high-frequency single-word character images are used as labels of the high-frequency single-word character images;
[0213] Obtaining a few-sample classification model according to the high-frequency single-word character image and the label of the high-frequency single-word character image;
[0214] Using the few-sample classification model to identify the single-word character image, and obtaining a label of the single-word character image;
[0215] A single-character character image classification model is obtained according to the single-character character image and the label of the single-character character image.
[0216] In this embodiment, the electronic device further performs the following steps:
[0217] The single-word character image classification model is used to recognize the single-word character image to be recognized, and the character corresponding to the single-word character image to be recognized is obtained.
[0218] In this embodiment, the electronic device further performs the following steps:
[0219] Obtaining a text set image to be recognized;
[0220] Perform target detection on the image of the text set to be recognized to obtain a single-word character image.
[0221] In this embodiment, performing a clustering operation on the single-word character image to obtain a clustering result includes:
[0222] Get a pre-trained neural network;
[0223] Utilizing the pre-trained neural network, obtaining single-word feature data of the single-word character image;
[0224] A clustering operation is performed on the single-word feature data to obtain a clustering result.
[0225] In this embodiment, the electronic device further performs the following steps:
[0226] Using the clustering results as pseudo labels within the clustering clusters;
[0227] Using the pseudo-label and the single-word character image corresponding to the pseudo-label, the pre-trained neural network is updated to obtain an updated neural network;
[0228] The updated neural network is used to obtain single-word feature data of the single-word character image.
[0229] In this embodiment, the step of selecting a high-frequency single-word character image from the clustering result according to the frequency of occurrence of the characters, and using the character corresponding to the high-frequency single-word character image as a label of the high-frequency single-word character image includes:
[0230] According to the frequency of occurrence of characters, high-frequency single-word character images are screened out from the clustering results;
[0231] Recognize the high-frequency single-word character image according to expert knowledge to obtain the character corresponding to the high-frequency single-word character image;
[0232] The character corresponding to the high-frequency single-word character image is used as a label of the high-frequency single-word character image.
[0233] In this embodiment, obtaining a few-sample classification model according to the high-frequency single-word character image and the label of the high-frequency single-word character image includes:
[0234] Build an initial few-shot classification model;
[0235] According to the high-frequency single-word character image and the label of the high-frequency single-word character image, the initial few-sample classification model is trained using a weight-sharing twin network to obtain a few-sample classification model.
[0236] In this embodiment, the step of using the few-sample classification model to recognize the single-word character image to obtain the label of the single-word character image includes:
[0237] Using the few-sample classification model to recognize the single-word character image, and obtain a classification result of the single-word character image;
[0238] Determining whether the classification result of the single-word character image matches the clustering result;
[0239] If there is a match, the classification result of the matched single-word character image is used as the label of the single-word character image.
[0240] In this embodiment, the electronic device further performs the following steps:
[0241] Screening the classification results of the single-word character images to obtain single-word character images with low confidence classification results;
[0242] Recognize the single-word character image with the low-confidence classification result according to expert knowledge to obtain the character corresponding to the single-word character image with the low-confidence classification result;
[0243] The few-sample classification model is updated using the characters corresponding to the single-word character images with the low-confidence classification results and the single-word character images with the low-confidence classification results.
[0244] In this embodiment, obtaining a single-character character image classification model according to the single-character character image and the label of the single-character character image includes:
[0245] Build an initial single-word character image classification model;
[0246] According to the single-character character image and the label of the single-character character image, the initial single-character character image classification model is trained using a residual neural network to obtain a single-character character image classification model.
[0247] It should be noted that for the detailed description of the electronic device provided in the third embodiment of the present application, reference can be made to the relevant description of the first embodiment of the present application, which will not be repeated here.
[0248] Corresponding to the character image processing method provided in the first embodiment of the present application, the fourth embodiment of the present application further provides a storage device storing a program of the character image processing method, the program is executed by a processor to perform the following steps:
[0249] Perform clustering operation on single-word character images to obtain clustering results;
[0250] According to the frequency of occurrence of characters, high-frequency single-word character images are screened out from the clustering results, and characters corresponding to the high-frequency single-word character images are used as labels of the high-frequency single-word character images;
[0251] Obtaining a few-sample classification model according to the high-frequency single-word character image and the label of the high-frequency single-word character image;
[0252] Using the few-sample classification model to identify the single-word character image, and obtaining a label of the single-word character image;
[0253] According to the single-word character image and the label of the single-word character image, a single-word character image classification model is obtained. It should be noted that the detailed description of the storage device provided in the fourth embodiment of the present application can refer to the relevant description of the first embodiment of the present application, which will not be repeated here.
[0254] The fifth embodiment of the present application provides a method for recognizing ancient Chinese book images. Figure 6 .
[0255] like Figure 6 As shown, in step S601, an ancient Chinese book image to be identified is obtained.
[0256] like Figure 6 As shown, in step S602, target detection is performed on the ancient Chinese book image to be recognized to obtain a single-word character image.
[0257] like Figure 6 As shown, in step S603, the single-word character image is recognized using a single-word character image classification model to obtain a character corresponding to the single-word character image, wherein the single-word character image classification model is obtained according to any one of the methods provided in the first embodiment of the present application.
[0258] It should be noted that the detailed description of the ancient Chinese book image recognition method provided in the fifth embodiment of the present application can refer to the relevant description of the first embodiment of the present application, and will not be repeated here.
[0259] A sixth embodiment of the present application provides a text image recognition method, comprising:
[0260] Obtain the text image to be recognized;
[0261] Using an image recognition model, obtaining a single-word character image of the text image;
[0262] Recognize the single-word character image using a single-word character image classification model to obtain a character corresponding to the single-word character image, wherein the single-word character image classification model is obtained according to any one of the methods provided in the first embodiment of the present application;
[0263] According to the character corresponding to the single-word character image, text data corresponding to the text image to be recognized is obtained.
[0264] The text image recognition method provided in the sixth embodiment of the present application can be applied to multiple scenarios. For example, it can be used to recognize advertising images. Some unscrupulous businesses may use similar characters to play tricks for economic interests, which may mislead consumers. In order to reduce the occurrence of this situation, the method provided in this embodiment can be used to identify such advertisements. Then, the illegal advertisements can be removed from the shelves to protect the interests of consumers.
[0265] Specifically, first obtain a text image to be identified, which may be an advertisement image to be identified. Then, use an overall recognition model, such as an image recognition model such as CenterNet, to perform target detection on the advertisement image to obtain a single-word character image of the text image; then, use a single-word character image classification model to recognize the single-word character image to obtain the character corresponding to the single-word character image, wherein the single-word character image classification model is obtained according to any one of the methods provided in the first embodiment of the present application; then, based on the character corresponding to the single-word character image, obtain the advertisement text data corresponding to the text image to be identified; finally, the advertisement text data may be compared with the legitimate advertisement text data of the advertisement registration structure to identify whether the advertisement image is in violation of regulations, and perform subsequent processing such as removing the infringing advertisements.
[0266] The text image recognition method provided in the sixth embodiment of the present application can also be applied to the scenario of answering questions. With the popularization of mobile phone applications, more and more mobile phone users use their mobile phones to complete the answers to some questions in image format. For example, a mobile phone user obtains a text image of a knowledge contest question and can obtain the answer to the knowledge contest question with the help of a mobile phone application.
[0267] Specifically, first obtain a text image to be recognized, which can be a question image to be recognized. Then, use an overall recognition model, such as an image recognition model such as CenterNet, to perform target detection on the question image to obtain a single-word character image of the text image; then, use a single-word character image classification model to recognize the single-word character image to obtain the character corresponding to the single-word character image, wherein the single-word character image classification model is obtained according to any one of the methods provided in the first embodiment of the present application; then, based on the character corresponding to the single-word character image, obtain the question text data corresponding to the text image to be recognized; finally, the question text data can be used as an index to retrieve a database storing questions and their answers, obtain the answer corresponding to the question, and provide the answer to the mobile phone user.
[0268] A seventh embodiment of the present application provides a text image recognition method, comprising:
[0269] Obtain the text image to be recognized;
[0270] Based on the first recognition model, the text image is recognized as a whole to obtain a first recognition result;
[0271] Segmenting the text image to obtain single-word character images of the text image;
[0272] Based on the second recognition model, the single character image is recognized to obtain a second recognition result;
[0273] Based on the first recognition result and the second recognition result, text data of the text image is obtained.
[0274] For example, first obtain the advertisement image to be identified (i.e., the text image to be identified), and then obtain the overall features of the advertisement image based on a first recognition model, such as a convolutional neural network. The overall features can be used to determine the category of the text image, such as whether it is an advertisement. Then, use CenterNet to segment the text image to obtain the single-word character image of the text image. Next, based on the single-word character image classification model, recognize the single-character image to obtain the corresponding character of the single-character image. Finally, based on the overall features of the advertisement image and the corresponding characters of the single-character image, obtain the text data of the text image.
[0275] It should be pointed out here that the text image to be recognized in this embodiment may also be a title image to be recognized.
[0276] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.
[0277] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0278] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0279] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory media such as modulated data signals and carrier waves.
[0280] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
Claims
1. A character image processing method, It is characterized in that include: Perform clustering operation on single-word character images to obtain clustering results; According to the frequency of occurrence of characters, high-frequency single-word character images are screened out from the clustering results, and characters corresponding to the high-frequency single-word character images are used as labels of the high-frequency single-word character images; Obtaining a few-sample classification model according to the high-frequency single-word character image and the label of the high-frequency single-word character image; Using the few-sample classification model to recognize the single-word character image, and obtain a classification result of the single-word character image; Determining whether the classification result of the single-word character image matches the clustering result; If there is a match, the classification result of the matched single-word character image is used as the label of the single-word character image; A single-character character image classification model is obtained according to the single-character character image and the label of the single-character character image.
2. The method according to claim 1, It is characterized in that Also includes: Obtain a single character image to be recognized; The single-word character image classification model is used to recognize the single-word character image to be recognized, and the character corresponding to the single-word character image to be recognized is obtained.
3. The method according to claim 1, It is characterized in that Also includes: Get text image; Perform target detection on the text image to obtain a single-word character image.
4. The method according to claim 1, It is characterized in that The performing of clustering operation on the single-word character image to obtain a clustering result includes: Get a pre-trained neural network; Using the pre-trained neural network, obtaining single-word feature data of the single-word character image; A clustering operation is performed on the single-word feature data to obtain a clustering result.
5. The method according to claim 4, It is characterized in that Also includes: Using the clustering results as pseudo labels within the clustering clusters; Using the pseudo-label and the single-word character image corresponding to the pseudo-label, the pre-trained neural network is updated to obtain an updated neural network; The updated neural network is used to obtain single-word feature data of the single-word character image.
6. The method according to claim 1, It is characterized in that The step of selecting a high-frequency single-word character image from the clustering result according to the frequency of occurrence of the characters, and using the character corresponding to the high-frequency single-word character image as a label of the high-frequency single-word character image includes: According to the frequency of occurrence of characters, high-frequency single-word character images are screened out from the clustering results; Recognize the high-frequency single-word character image according to expert knowledge to obtain the character corresponding to the high-frequency single-word character image; The character corresponding to the high-frequency single-word character image is used as a label of the high-frequency single-word character image.
7. The method according to claim 1, It is characterized in that The obtaining of a few-sample classification model according to the high-frequency single-word character image and the label of the high-frequency single-word character image comprises: Build an initial few-shot classification model; According to the high-frequency single-word character image and the label of the high-frequency single-word character image, the initial few-sample classification model is trained using a weight-sharing twin network to obtain a few-sample classification model.
8. The method according to claim 7, It is characterized in that Also includes: Screening the classification results of the single-word character images to obtain single-word character images with low confidence classification results; Recognize the single-word character image with the low-confidence classification result according to expert knowledge to obtain the character corresponding to the single-word character image with the low-confidence classification result; The few-sample classification model is updated using the characters corresponding to the single-word character images with the low-confidence classification results and the single-word character images with the low-confidence classification results.
9. The method according to claim 1, It is characterized in that The step of obtaining a single-character character image classification model according to the single-character character image and the label of the single-character character image comprises: Build an initial single-word character image classification model; According to the single-character character image and the label of the single-character character image, the initial single-character character image classification model is trained using a residual neural network to obtain a single-character character image classification model.
10. A character image processing device, It is characterized in that include: A clustering execution unit, used for performing a clustering operation on the single-word character image to obtain a clustering result; An image screening unit, used for screening out high-frequency single-word character images from the clustering results according to the frequency of occurrence of the characters, and using the characters corresponding to the high-frequency single-word character images as labels of the high-frequency single-word character images; A model acquisition unit, used to obtain a few-sample classification model according to the high-frequency single-word character image and the label of the high-frequency single-word character image; a label obtaining unit, configured to recognize the single-character character image by using the few-sample classification model to obtain a classification result of the single-character character image; determine whether the classification result of the single-character character image matches the clustering result; if so, use the classification result of the matching single-character character image as a label of the single-character character image; The classification model obtaining unit is used to obtain a single-character character image classification model according to the single-character character image and the label of the single-character character image.
11. An electronic device, It is characterized in that include: processor; as well as A memory is used to store a program of a data processing method. After the device is powered on and the program of a single-character image recognition method is run by the processor, the following steps are performed: Perform clustering operation on single-word character images to obtain clustering results; According to the frequency of occurrence of characters, high-frequency single-word character images are screened out from the clustering results, and characters corresponding to the high-frequency single-word character images are used as labels of the high-frequency single-word character images; Obtaining a few-sample classification model according to the high-frequency single-word character image and the label of the high-frequency single-word character image; Using the few-sample classification model to recognize the single-word character image, and obtain a classification result of the single-word character image; Determining whether the classification result of the single-word character image matches the clustering result; If there is a match, the classification result of the matched single-word character image is used as the label of the single-word character image; A single-character character image classification model is obtained according to the single-character character image and the label of the single-character character image.
12. A storage device, It is characterized in that A program storing a data processing method is executed by a processor to perform the following steps: Perform clustering operation on single-word character images to obtain clustering results; According to the frequency of occurrence of characters, high-frequency single-word character images are screened out from the clustering results, and characters corresponding to the high-frequency single-word character images are used as labels of the high-frequency single-word character images; Obtaining a few-sample classification model according to the high-frequency single-word character image and the label of the high-frequency single-word character image; Using the few-sample classification model to recognize the single-word character image, and obtain a classification result of the single-word character image; Determining whether the classification result of the single-word character image matches the clustering result; If there is a match, the classification result of the matched single-word character image is used as the label of the single-word character image; A single-character character image classification model is obtained according to the single-character character image and the label of the single-character character image.
13. A method for recognizing ancient Chinese book images. It is characterized in that include: Obtaining ancient Chinese book images to be identified; Performing target detection on the ancient Chinese book image to be identified to obtain a single-word character image; The single-word character image is recognized using a single-word character image classification model to obtain a character corresponding to the single-word character image, wherein the single-word character image classification model is obtained according to any one of the methods of claims 1-9.
14. A text image recognition method, It is characterized in that include: Obtain the text image to be recognized; Using an image recognition model, obtaining a single-word character image of the text image; Recognize the single-word character image using a single-word character image classification model to obtain a character corresponding to the single-word character image, wherein the single-word character image classification model is obtained according to any one of the methods of claims 1-9; According to the character corresponding to the single-word character image, text data corresponding to the text image to be recognized is obtained.
15. A text image recognition method, It is characterized in that include: Obtain the text image to be recognized; Based on the first recognition model, the text image is recognized as a whole to obtain a first recognition result; Segmenting the text image to obtain single-word character images of the text image; Based on a second recognition model, the single-word character image is recognized to obtain a second recognition result, wherein the second recognition model is obtained according to any one of the methods of claims 1-9; Based on the first recognition result and the second recognition result, text data of the text image is obtained.