Method and system for automatic generation of general scene text recognition data

Abstract

The application relates to the technical field of character recognition, and provides a general-scene character recognition data automatic generation method and system, which comprises the following steps: collecting and establishing a part of character recognition data set A; using a deep learning-based image enhancement method to generate printed data set B; using an adversarial generative network (GAN) to migrate the style of the data set A to the data set B, and expanding to obtain data set C; using a CTC-based character accurate positioning image enhancement method to obtain D and E; and proportionally mixing the data sets A, B, C, D and E for training, wherein the mixed training adopts a feature-based data enhancement method to enrich sample diversity. The application can solve the problem that, in handwriting character recognition, due to the continuous writing of handwriting characters, the handwriting characters are more diverse, meanwhile, diverse fonts and open data sets are lacking, and handwriting recognition is difficult to achieve the same amazing effect as that of typewritten characters, and high-quality and evenly-distributed data are generated through image enhancement, semantic understanding, GAN and other technologies.

Description

Technical Field

[0001] This invention relates to the field of text recognition technology, and in particular to an automatic generation method and system for text recognition data in general scenarios. Background Technology

[0002] With the continuous development of artificial intelligence technology, OCR recognition is widely used in many fields such as banking, logistics, and autonomous driving. Text recognition methods mainly include: character segmentation followed by character classification for recognition, attention-based character alignment recognition, CTC-based maximum probability path recognition, and transformer-based multi-head-attention encoding / decoding recognition. However, due to its faster training and prediction speed, CTC performs better in both short and long texts, especially Chinese text recognition, and is widely used in industry. Currently, there are no publicly available academic papers on character localization methods based on CTC, but character position is crucial in contract comparison, image quality inspection, and other problems.

[0003] Deep learning-based text recognition methods rely on a large amount of data for training. However, in real-world development scenarios, it is often difficult to obtain enough realistic labeled data because manual labeling is very expensive and the labeling speed is unsatisfactory. Text recognition often relies on data augmentation methods to enrich the dataset. Therefore, effective data augmentation methods are key to meeting diverse recognition needs.

[0004] Machine-generated text can generate a wide variety of texts using different fonts, and even images that approximate the background can be produced through simple depth estimation. However, for handwritten text recognition, due to the flowing strokes and greater diversity of handwritten characters, coupled with a lack of diverse fonts and publicly available datasets, it is difficult to achieve the same impressive results as machine-generated text. Therefore, the aim is to generate high-quality and evenly distributed data through techniques such as image enhancement, semantic understanding, and generative adversarial networks. Summary of the Invention

[0005] Commonly used data generation methods in character recognition technology can be divided into three categories: GAN (Generative Adversarial Network) generation, feature transformation-based image enhancement, and deep learning-based image enhancement. This invention proposes a character-accurate localization-based image enhancement method based on Character Transformation (CTC), enriching the data generation methods. Furthermore, it innovatively suggests that for general-purpose character recognition scenarios, a richer and more balanced dataset can be obtained through the cross-application of the above four data generation methods, thereby facilitating model training and improving the model's generalization ability.

[0006] The purpose of this invention is to overcome at least one of the shortcomings of the prior art and to provide a method and system for automatically generating text recognition data for general scenarios.

[0007] The technical concept of this invention is as follows:

[0008] Assuming we have a basic character recognition dataset A, we generate a printed text dataset B using deep learning-based image augmentation methods. We then use a GAN to transfer the style of A to B, expanding the dataset to obtain C. Using dataset A, we obtain a sample set S for each character through precise CTC localization. We generate D by randomly replacing characters in A. We obtain E based on text combinations with semantic information or random combinations of character sets S. Finally, we can train by mixing A, B, C, D, and E in proportion. During the training process, we can use feature-based data augmentation methods to enrich the sample diversity.

[0009] The image enhancement method based on CTC for precise character localization is as follows: Training is performed on a finite training set A using the CTC method, with the accurate prediction being A_. The start and end positions of each character in the feature layer are obtained, and then the receptive field is acquired to determine the start and end points of the characters on the input image. This allows for the acquisition of individual characters, initially completing the basic character collection S. Diversity can be enhanced through manual image cropping, further resulting in a character set S. + Then in S + Use it to generate various character combinations or use S + Replace A_ to obtain a larger dataset with a more balanced data distribution, thus completing data augmentation.

[0010] The present invention adopts the following technical solution:

[0011] On one hand, the present invention provides an automatic generation method for general scene text recognition data, comprising:

[0012] S1. Collect and build a partial character recognition dataset A, where dataset A is an existing dataset;

[0013] S2. Based on dataset A, generate printed data dataset B using deep learning-based image enhancement methods;

[0014] S3. Use Generative Adversarial Network (GAN) to transfer the style of dataset A to dataset B, and expand the dataset to obtain dataset C.

[0015] S4. Based on dataset A, use the CTC-based character precise localization image enhancement method to obtain dataset D and dataset E;

[0016] S5. Datasets A, B, C, D, and E are mixed and trained in proportion. Feature-based data augmentation methods are used to enrich the diversity of samples during mixed training to obtain the final dataset, namely the general scene text recognition dataset.

[0017] In addition to any of the possible implementations described above, another implementation is provided in which, in step S2, the deep learning-based image enhancement method generates dataset B using a method for synthesizing natural scene text. The text generation method includes:

[0018] S2.1 Font Rendering: Randomly select a font and render the text in the image foreground layer along a horizontal direction or a random curve;

[0019] S2.2 Stroke and Shadow: Renders edges or shadows of random width to the foreground layer of the image;

[0020] S2.3 Basic Colorization: Each of the three image layers is filled with a different uniform color collected from the existing dataset A. The uniform color is clustered into three categories (corresponding to the three image layers respectively) by the K-means algorithm for the three channel colors (R, G, B) of each image in dataset A. The three image layers are the image foreground layer, the image background layer, and the edge shadow layer.

[0021] S2.4 Affine Projection Distortion: Randomly projective distortion is applied to the foreground and edge shadow layers of the image to simulate a 3D environment.

[0022] S2.5 Natural Data Mixing: Each image layer is mixed with randomly sampled images from the training datasets of ICDAR 2003 and SVT to obtain dataset B.

[0023] In addition to any of the possible implementations described above, another implementation is provided: In step S3, a generative model is used to generate balanced images applicable to various scenarios from image data of a specific scene obtained through limited channels. The generative model iteratively obtains these images through a game with the discriminative model. The method for expanding the dataset to obtain dataset C is as follows:

[0024] The S3.1 generative model generates a batch of images;

[0025] S3.2 The discriminant model learns to distinguish between generated images and real images;

[0026] The S3.3 generative model improves the generative model based on the feedback results from the discriminative model, iteratively generating new images;

[0027] The S3.4 discriminative model continues to learn to distinguish between generated and real images;

[0028] After S3.5 convergence is complete, image data is generated using the generative model.

[0029] In addition to any of the possible implementations described above, another implementation is provided in which, in step S4, a character-accurate localization-based image enhancement method is used to obtain dataset D and dataset E. The specific method is as follows:

[0030] S4.1 For a limited dataset A, a text recognition model based on CTC is used for training;

[0031] S4.2 After the training in step S4.1 is completed, compare the prediction results of dataset A with the true labels, and denote the datasets with the same comparison results as A_.

[0032] S4.3 By grouping and aggregating the output tensors of the CTC-based character recognition model, the starting position s of each character in the image of dataset A in the output feature layer of the CTC-based character recognition model is obtained. i and the end position e i , i>0;

[0033] S4.4 Calculate the starting position S of each character in the dataset A_image within the input image. i and the end position E i ;

[0034] S4.5 Based on the coordinate values ​​of single characters in the dataset A_ image obtained in step S4.4, obtain the single character fragment image dataset, denoted as S;

[0035] S4.6 Count the frequency of each character in S, and use manual segmentation to balance the character distribution to obtain the character set S. + ;

[0036] S4.7 Expands dataset A_ in three ways:

[0037] 1) First, create semantic tags, and then combine character sets based on the tag content to generate an image;

[0038] 2) Based on the obtained single-character coordinates and the obtained character set S + Replace characters in the image of dataset A by pasting a single character image at the coordinate position of the character to be replaced.

[0039] 3) Randomly generated;

[0040] Use 2) to form dataset D, and use 1) and 3) to generate dataset E.

[0041] In addition to any of the possible implementations described above, another implementation is provided in which, in step S4.4, the starting position S of each character in the input image is... i and the end position E i The calculation method is as follows:

[0042] X1. The values ​​of parameters related to the receptive field of the output feature layer of the convolutional neural network are calculated iteratively using the following formula:

[0043] jump:j out =j in *s

[0044]

[0045] Where jump(j) represents the distance between two consecutive feature points, the subscripts in and out represent the input state and output state, start represents the center coordinates of the first feature point, s represents the compensation of the convolution operation, k represents the size of the convolution kernel, and p represents the size of the convolution padding.

[0046] X2. Calculate the starting position S using the following formula. i and the end position E i :

[0047] S i =start+s i *jump

[0048] E i =start+e i *jump.

[0049] In addition to any of the possible implementations described above, another implementation is provided in which, in step S5, a feature-based data augmentation method is used in the hybrid training. The feature-based image augmentation method is to perform feature transformation on the existing data to expand the data volume. The feature transformation methods include: blurring, contrast change, stretching, rotation and random cropping.

[0050] In addition to any of the possible implementations described above, another implementation is provided in which, in step S5, training is performed by mixing datasets proportionally, with the proportion of each dataset determined based on experimental or actual needs.

[0051] On the other hand, the present invention also provides an automatic generation system for general scene text recognition data, including:

[0052] The deep learning-based image enhancement module is used to generate printed data set B based on dataset A using deep learning-based image enhancement methods.

[0053] The Generative Adversarial Network (GAN) module is used to transfer the style of dataset A to dataset B using Generative Adversarial Network (GAN) to expand the dataset to obtain dataset C.

[0054] The image enhancement module based on CTC character precise localization is used to obtain datasets D and E based on dataset A using the CTC-based character precise localization image enhancement method.

[0055] The hybrid training module is used to train datasets A, B, C, D, and E in a proportional manner. Feature-based data augmentation methods are used in the hybrid training to enrich the diversity of the samples and obtain the final dataset.

[0056] The system employs the aforementioned method for automatically generating general scene text recognition data.

[0057] On the other hand, the present invention also provides a terminal, including: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the terminal executes the above-described method for automatically generating general scene text recognition data.

[0058] On the other hand, the present invention also provides a computer storage medium storing a computer program, which is executed by a processor to implement the method for automatically generating general scene text recognition data as described in any one of claims 1-7.

[0059] The beneficial effects of this invention are as follows:

[0060] 1. The cross-application of GAN generation method, feature transformation-based image enhancement, deep learning-based image enhancement and CTC-based character precise localization image enhancement method realizes the diverse generation of text recognition data.

[0061] 2. Cross-application of text recognition data generation methods to address different recognition scenarios.

[0062] 3. A precise character localization method based on CTC and receptive field.

[0063] 4. Single-character segmentation of the training set based on precise character localization.

[0064] 5. Data augmentation method: Random character replacement method based on precise character positioning and partial semantics. Attached Figure Description

[0065] Figure 1 The diagram shown is a logic diagram of an automatic generation method for general scene text recognition data according to an embodiment of the present invention. Detailed Implementation

[0066] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that the technical features or combinations of technical features described in the following embodiments should not be considered isolated; they can be combined with each other to achieve better technical effects. In the accompanying drawings of the following embodiments, the same reference numerals appearing in each drawing represent the same features or components, which can be applied to different embodiments.

[0067] The overall implementation logic diagram is as follows: Figure 1 As shown in the figure, an embodiment of the present invention provides a method for automatically generating text recognition data in general scenarios, comprising:

[0068] S1. Collect and build a partial character recognition dataset A;

[0069] S2. Based on dataset A, generate printed data dataset B using deep learning-based image enhancement methods;

[0070] As a specific example, dataset B is generated using a method for synthesizing natural scene text. The text generation method includes:

[0071] S2.1 Font Rendering: Randomly select a font and render the text in the image foreground layer along a horizontal direction or a random curve;

[0072] S2.2 Stroke, add shadow, color;

[0073] S2.3 Basic Colorization: Each of the three image layers is filled with a different uniform color collected from the existing dataset A. The uniform color is clustered into three categories by the K-means algorithm for the three channel colors of each image in dataset A. The three image layers are the image foreground layer, the image background layer, and the edge shadow layer.

[0074] S2.4 Affine Projection Distortion: Randomly projective distortion is applied to the foreground and edge shadow layers of the image to simulate a 3D environment.

[0075] S2.5 Natural Data Mixing: Each image layer is mixed with randomly sampled images from the training datasets of ICDAR 2003 and SVT to obtain dataset B.

[0076] S3. Use Generative Adversarial Network (GAN) to transfer the style of dataset A to dataset B, and expand the dataset to obtain dataset C.

[0077] In one specific embodiment, a generative model is used to generate balanced images applicable to various scenarios from image data of a specific scene obtained through limited channels. The generative model iteratively acquires these images through a game with a discriminative model. The method for expanding the dataset to obtain dataset C is as follows:

[0078] The S3.1 generative model generates a batch of images;

[0079] S3.2 The discriminant model learns to distinguish between generated images and real images;

[0080] The S3.3 generative model improves the generative model based on the feedback results from the discriminative model, iteratively generating new images;

[0081] The S3.4 discriminative model continues to learn to distinguish between generated and real images;

[0082] After S3.5 convergence is complete, image data is generated using the generative model.

[0083] S4. Based on dataset A, use the CTC-based character precise localization image enhancement method to obtain dataset D and dataset E;

[0084] In one specific embodiment, the method is as follows:

[0085] S4.1 For a limited dataset A, a text recognition model based on CTC is used for training;

[0086] S 4.2 After the training in step S4.1 is completed (an open-source pre-trained model can also be used), compare the prediction results of dataset A with the true labels, and denote the datasets with the same comparison results as A_.

[0087] S4.3 By grouping and aggregating the model output tensors, the starting position s of each character in the image of dataset A in the model output feature layer is obtained. i and the end position e i , i>0;

[0088] S4.4 Calculate the starting position S of each character in the dataset A_image within the input image. i and the end position E i ;

[0089] In one specific embodiment, the starting position S of each character in the input image i and the end position E i The calculation method is as follows:

[0090] X1. The values ​​of parameters related to the receptive field of the output feature layer of the convolutional neural network are calculated iteratively using the following formula:

[0091] jump:j out =j in *s

[0092]

[0093] Where jump represents the distance between two consecutive feature points, start represents the center coordinate of the first feature point, s represents the compensation of the convolution operation, k represents the size of the convolution kernel, and p represents the convolution padding size;

[0094] X2. Calculate the starting position S using the following formula. i and the end position E i :

[0095] S i =start+s i *jump

[0096] E i =start+e i *jump.

[0097] S4.5 Based on the coordinate values ​​of single characters in the dataset A_ image obtained in step S4.4, obtain the single character fragment image dataset, denoted as S;

[0098] S4.6 Count the frequency of each character in S, and use manual segmentation to balance the character distribution to obtain the character set S. + ;

[0099] S4.7 Expands dataset A_ in three ways:

[0100] 1) First, create semantic tags, and then combine character sets based on the tag content to generate an image;

[0101] 2) Based on the single character coordinates obtained in step 4 and the obtained character set S + Replace characters in the image of dataset A by pasting a single character image at the coordinate position of the character to be replaced.

[0102] 3) Randomly generated;

[0103] Use 2) to form dataset D, and use 1) and 3) to generate dataset E.

[0104] S5. Datasets A, B, C, D, and E are mixed and trained proportionally. Feature-based data augmentation methods are used to enrich the diversity of samples during mixed training to obtain the final dataset.

[0105] In one specific embodiment, the feature transformation-based image enhancement method expands the data volume by performing feature transformation on existing data. The feature transformation methods mainly include: blurring, contrast change, stretching, rotation and random cropping.

[0106] The above steps outline an automatic method for generating text recognition data in general scenarios. Taking the recognition of mixed printed and handwritten dates as an example, in date recognition scenarios, existing datasets all contain data prior to the current time, making it impossible to obtain datasets for future times. For instance, date data for January 1, 2050, strictly speaking, does not exist. However, the recognition model must be able to identify future times; therefore, it is necessary to obtain future date data through data generation. The strategy adopted for date recognition is to generate random date data for training using data generation methods, while using the original real data as the test set. The table below reflects the recognition accuracy corresponding to different data generation methods for the same original dataset and the same recognition model (CTC+CRNN).

[0107]

[0108] As can be seen from the table above, the image enhancement method based on CTC-based precise character localization proposed in this invention can significantly improve the recognition accuracy (from 56-62% to 92%). If the four recognition methods are combined, the recognition accuracy can be further improved (to 95%). Other experiments have yielded similar results.

[0109] The four generation methods can be flexibly combined to enrich the dataset under different recognition needs.

[0110] This invention addresses the problem that "handwritten text recognition is difficult to achieve the same stunning results as machine-printed text due to the continuous and diverse strokes of handwritten text, coupled with a lack of diverse fonts and publicly available datasets." It generates high-quality and evenly distributed data through techniques such as image enhancement, semantic understanding, and GAN.

[0111] While several embodiments of the present invention have been provided herein, those skilled in the art should understand that modifications can be made to these embodiments without departing from the spirit of the invention. The above embodiments are merely exemplary and should not be construed as limiting the scope of the invention.

Claims

1. A method for automatically generating text recognition data for general scenarios, characterized in that, The method includes: S1. Collect and build a partial character recognition dataset A; S2. Based on dataset A, generate printed data dataset B using deep learning-based image enhancement methods; S3. Use Generative Adversarial Network (GAN) to transfer the style of dataset A to dataset B, and expand the dataset to obtain dataset C. S4. Based on dataset A, use the CTC-based character precise localization image enhancement method to obtain dataset D and dataset E; S5. Datasets A, B, C, D, and E are mixed and trained in proportion. Feature-based data augmentation methods are used in the mixed training to obtain the final dataset, namely the general scene text recognition dataset. In step S2, the deep learning-based image enhancement method generates dataset B by synthesizing natural scene text. The text generation method includes: S2.1 Font rendering: Randomly select a font and render the text in the foreground layer of the image along a horizontal direction or a random curve; S2.2 Stroke and Shadow: Renders edges or shadows of random width to the foreground layer of the image; S2.3 Basic Coloring: Each of the three image layers is filled with a different uniform color collected from the existing dataset A. The uniform color is clustered into three categories by the K-means algorithm for the three channel colors of each image in dataset A. The three image layers are the image foreground layer, the image background layer, and the edge shadow layer. S2.4 Affine Projection Distortion: Randomly project and distort the foreground and edge shadow layers of the image to simulate a 3D environment. S2.5 Natural Data Mixing: Each image layer is mixed with randomly sampled images from the training datasets of ICDAR 2003 and SVT to obtain dataset B; In step S4, image enhancement methods based on CTC for precise character localization are used to obtain datasets D and E. The specific method is as follows: S4.1 For a limited dataset A, a text recognition model based on CTC is used for training; S 4.2 After the training in step S4.1 is completed, compare the prediction results of dataset A with the true labels, and denote the datasets with the same comparison results as A_; S4.3 By grouping and aggregating the output tensors of the CTC-based character recognition model, the starting position s of each character in the image of dataset A in the output feature layer of the CTC-based character recognition model is obtained. i and the end position e i , i > 0; S4.4 Calculate the starting position S of each character in the dataset A_image within the input image. i and the end position E i ; S4.5 Based on the coordinate values ​​of individual characters in the dataset A_ obtained in step S4.4, obtain the single-character fragment image dataset, denoted as... ; S4.6 Statistics The frequency of each character in the character set is used, along with manual segmentation, to balance the character distribution and obtain the character set. ; S4.7 Expanded Dataset There are three ways to expand: 1) First, create semantic tags, and then combine character sets based on the tag content to generate an image; 2) Based on the obtained single-character coordinates and the obtained character set Replace dataset The characters in the image, specifically, are pasted onto the coordinate positions of the character to be replaced; 3) Randomly generated; Use 2) to form dataset D, and use 1) and 3) to generate dataset E.

2. The method for automatically generating general scene text recognition data as described in claim 1, characterized in that, In step S3, the image data of a specific scene obtained by the generative model is used to generate balanced images applicable to various scenes. The generative model iteratively obtains these images through a game with the discriminative model. The method for expanding the dataset to obtain dataset C is as follows: S3.1 Generative model generates a batch of images; S3.2 The discriminant model learns to distinguish between generated images and real images; The S3.3 generative model improves the generative model based on the feedback results from the discriminative model, iteratively generating new images; The S3.4 discriminative model continues to learn to distinguish between generated and real images; After S3.5 converges, the trained generative model is used to generate image data.

3. The method for automatically generating general scene text recognition data as described in claim 1, characterized in that, In step S4.4, the starting position S of each character in the input image i and the end position E i The calculation method is as follows: X1. The values ​​of parameters related to the receptive field of the output feature layer of the convolutional neural network are calculated iteratively using the following formula: Where jump represents the distance between two consecutive feature points, the subscripts in and out represent the input state and output state, start represents the center coordinates of the first feature point, s represents the compensation of the convolution operation, k represents the size of the convolution kernel, and p represents the size of the convolution padding. X2. Calculate the starting position S using the following formula. i and the end position E i : 。 4. The method for automatically generating general scene text recognition data as described in claim 1, characterized in that, In step S5, a feature-based data augmentation method is used in the hybrid training. The feature-based image augmentation method expands the data volume by performing feature transformation on the existing data. The feature transformation methods include: blurring, contrast change, stretching, rotation and random cropping.

5. The method for automatically generating general scene text recognition data as described in claim 1, characterized in that, In step S5, training is performed by mixing datasets proportionally, with the proportions of each dataset determined based on experimental or practical needs.

6. An automatic generation system for general scene text recognition data, characterized in that, The system includes: The deep learning-based image enhancement module is used to generate printed data set B based on dataset A using deep learning-based image enhancement methods. The Generative Adversarial Network (GAN) module is used to transfer the style of dataset A to dataset B using Generative Adversarial Network (GAN) to expand the dataset to obtain dataset C. The image enhancement module based on CTC character precise localization is used to obtain datasets D and E based on dataset A using the CTC-based character precise localization image enhancement method. The hybrid training module is used to train datasets A, B, C, D, and E in a proportional manner. Feature-based data augmentation methods are used in the hybrid training to enrich the diversity of the samples and obtain the final dataset. The system employs the automatic generation method for general scene text recognition data as described in any one of claims 1-5.

7. A terminal, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory, so that the terminal performs the automatic generation method for general scene text recognition data as described in any one of claims 1-5.

8. A computer storage medium, characterized in that, The medium stores a computer program, which is executed by a processor to implement the automatic generation method for general scene text recognition data as described in any one of claims 1-5.

Images