Natural scene text detection and recognition method based on CRAFT and SCRN-SEED framework

Abstract

The present invention discloses a natural scene text detection and recognition method based on CRAFT and SCRN-SEED framework, comprising the following steps: (1) using a real data set and a synthetic data set to establish an image data set, and dividing the image data set into a training set and a test set; (2) using the image data set to train a CRAFT network; (3) using the real data set to train an irregular text correction network SCRN; (4) combining SCRN with a SEED network, and training the combined SCRN-SEED network; (5) connecting the CRAFT network with the SCRN-SEED network, building a complete model and training it. The present invention can fully detect curved and deformed text or long text instances, and achieves the purpose of detection by accurately locating each character, and then connecting the detected characters into a text through an affinity mechanism. It is applicable to curved, deformed or extremely long texts; by correcting irregular text images and using semantic information for global information detection, low-quality text instances can be accurately identified.

Description

Technical Field

[0001] The present invention relates to the technical field of text detection methods, and in particular to a natural scene text detection and recognition method based on CRAFT and SCRN-SEED frameworks. Background Art

[0002] Optical Character Recognition (OCR) traditionally refers to analyzing and processing input scanned document images to identify text information in the image. This type of technology assumes that the input image background is clean, the font is simple, and the text is neatly arranged. It can achieve a high level of recognition if it meets the requirements. Scene Text Recognition (STR) refers to recognizing text information in natural scene images. It is much more difficult than recognizing text in scanned document images. The text in natural scenes is extremely rich in form, and there may be the following situations: text lines are horizontal, vertical, curved, rotated, twisted, etc.; the text area in the image is incomplete or blurred; the background is diverse, such as text appears on a flat, curved or wrinkled surface, there are complex interfering textures near the text area, or there are textures similar to text in the non-text area, such as sand, grass, fences, brick walls, etc.

[0003] The scene text detection method based on neural network has achieved good results, far exceeding traditional technologies in detection and recognition, but it still cannot solve the problems of bending, blurred interference texture and other problems in natural scenes. The existing natural scene text detection methods have the following problems: inaccurate text positioning. Traditional text positioning model frameworks mostly focus on the entire line of text, require a large receptive field, and use a single rectangular box to mark the location of the text, which is not suitable for curved, deformed or extremely long texts, and it is difficult to accurately locate and mark; inaccurate text recognition. Many recognition methods based on the encoder-decoder framework are proposed to process curved text. However, most of them are based on local visual features and ignore global semantic information. Therefore, when faced with situations such as blurred images, uneven lighting and incomplete characters, the recognition accuracy is greatly reduced. Summary of the invention

[0004] Purpose of the invention: In view of the above problems, the purpose of the present invention is to provide a natural scene text detection and recognition method based on CRAFT and SCRN-SEED framework.

[0005] Technical solution: A natural scene text detection and recognition method based on CRAFT and SCRN-SEED framework of the present invention comprises the following steps:

[0006] (1) Use real datasets and synthetic datasets to build an image dataset, and divide the image dataset into a training set and a test set;

[0007] (2) Training the CRAFT network using image datasets:

[0008] (201) The CRAFT network was improved, and the ResNet50 network was used as the backbone network. The images in the synthetic dataset were input into the improved CRAFT network for feature extraction, and the region score and affinity score were output;

[0009] (202) Encoding the two scores through Gaussian heat mapping to generate a Gaussian heat map;

[0010] (203) cutting the complete text in the input image into individual characters according to the watershed algorithm, and generating polygons of arbitrary shapes of text from the characters through post-processing operations;

[0011] (203) Applying the idea of ​​transfer learning, the improved CRAFT network is initialized using the pre-trained model;

[0012] (3) Use real data sets to train the irregular text correction network SCRN;

[0013] (4) Combine SCRN with SEED network and train the combined SCRN-SEED network;

[0014] (5) Connect the improved CRAFT network and SCRN-SEED network to build a complete model and train it.

[0015] Furthermore, the steps of applying the idea of ​​transfer learning and initializing the improved CRAFT network using the pre-trained model include:

[0016] First, we train the CRAFT network using synthetic datasets, optimize the network using the Adam optimizer, and then fine-tune the network using multiple real datasets. During fine-tuning, we use the SynthText dataset at a ratio of 1:5 and online hard example mining at a ratio of 1:3.

[0017] Then, the CRAFT network was trained using a real dataset containing quadrilateral annotations and a SynthText dataset, and a part of it was divided into a test set to adjust the network parameters.

[0018] Furthermore, the steps of combining the SCRN with the SEED network and training the combined SCRN-SEED network include:

[0019] The trained SCRN network is used to replace the image correction module in the SEED network to build the SCRN-SEED network. The pre-trained language model of the semantic model FastText is used to initialize the parameters of the pre-trained model. The SCRN-SEED network is preliminarily trained using the test set, and the network parameters are adjusted according to the training results.

[0020] Furthermore, the improved CRAFT network is connected to the SCRN-SEED network to build a complete model and train it. The steps include: generating a minimum rectangular box containing all characters from the polygon of arbitrary shape text, cropping the rectangular box, adjusting the format of the cropped image, and then inputting it into the SCRN-SEED network to complete the construction of the model, training the model using the validation set, retaining the parameters with the best training effect, inputting pictures in natural scenes into the model, and performing automatic text detection and recognition tasks.

[0021] Furthermore, the real datasets are from ICDAR2013, ICDAR2015, ICDAR2017, MSRA-TD500, TotalText, CTW-1500, etc., and the synthetic dataset is the SynthText dataset;

[0022] Adjust the size of each image in the image dataset and convert the image format in the dataset to mdb format.

[0023] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0024] 1. The present invention can fully detect curved, deformed text or long text instances by accurately locating each character and then connecting the detected characters into a text through an affinity mechanism to achieve the purpose of detection. It only needs to focus on the characters and the distance between the characters, not the entire line of text, and does not require a large receptive field. It is applicable to curved, deformed or extremely long texts.

[0025] 2. The present invention can accurately identify low-quality text instances: the present invention uses the output results of the detector through the correction module of SCRN to correct irregular text images. The SCRN network uses the center line of each text instance and some geometric properties to achieve the purpose of correction with better results. The corrected text image is passed through the recognition module in the SEED network for text recognition, and semantic information is used for the prediction of global information, which greatly improves the accuracy of text recognition with defects, blurs, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is the basic framework diagram of the improved CRAFT model of the present invention;

[0027] Figure 2The basic framework diagram of the irregular text correction network SCRN adopted by the present invention;

[0028] Figure 3 This is a basic framework diagram of the SEED network identification module used in the present invention;

[0029] Figure 4 It is a diagram of the overall model structure of the natural scene text detection and recognition method of the present invention;

[0030] Figure 5 This is a sample effect diagram of the natural scene text detection and recognition method of the present invention. DETAILED DESCRIPTION

[0031] The natural scene text detection and recognition method based on the CRAFT and SCRN-SEED framework described in this embodiment includes the following steps:

[0032] (1) Use real data sets and synthetic data sets to establish an image data set, divide the image data set into a training set and a test set, adjust the size of each image in the image data set, and convert the image format in the data set into mdb format.

[0033] The real data sets come from ICDAR2013, ICDAR2015, ICDAR2017, MSRA-TD500, TotalText, and CTW-1500, and the synthetic data set is the SynthText data set.

[0034] (2) Use the image dataset to train the CRAFT network. The flowchart is as follows: Figure 1 shown.

[0035] (201) The CRAFT network was improved, and the ResNet50 network was used as the backbone network. The images in the synthetic dataset were input into the improved CRAFT network for feature extraction, and the region score and affinity score were output;

[0036] (202) Encoding the two scores through Gaussian heat mapping to generate a Gaussian heat map;

[0037] (203) cutting the complete text in the input image into individual characters according to the watershed algorithm, and generating polygons of arbitrary shapes of text from the characters through post-processing operations;

[0038] (203) Applying the idea of ​​transfer learning, the improved CRAFT network is initialized using the pre-trained model:

[0039] First, the CRAFT network is trained using a synthetic dataset, the network is optimized using the Adam optimizer, and then the network is fine-tuned using multiple real datasets. During fine-tuning, the SynthText dataset is used at a ratio of 1:5 to ensure that the character areas are definitely separated, and online hard example mining is used at a ratio of 1:3. Data augmentation techniques such as cropping, rotation, or color change can also be applied.

[0040] Then, the CRAFT network was trained using a real dataset containing quadrilateral annotations and a SynthText dataset, and a part of it was divided into a test set to adjust the network parameters.

[0041] (3) Use real data sets to train the irregular text correction network SCRN. The network framework is as follows: Figure 2 shown.

[0042] (4) Combine SCRN with SEED network, train the combined SCRN-SEED network and use it as the recognition network. The framework of the recognition module in the original SEED network is as follows: Figure 3 shown.

[0043] The trained SCRN network is used to replace the image correction module in the original SEED network to build the SCRN-SEED network, thereby improving the correction effect and recognition accuracy of irregular text and reducing the difficulty of model training. According to the required recognition language, the pre-trained language model of the semantic model FastText is downloaded, the parameters of the pre-trained model are initialized, the SCRN-SEED network is preliminarily trained using the test set, and the network parameters are adjusted according to the training effect.

[0044] (5) Connect the improved CRAFT network with the SCRN-SEED network to build a complete model and train it.

[0045] The polygon of any shape text is generated into the smallest rectangular box containing all characters, the rectangular box is cropped out, the cropped image format is adjusted, and then input into the SCRN-SEED network to complete the model construction. The model framework after construction is as follows Figure 4 As shown in the figure, it consists of three parts: CRAFT network is used as the detector part, SCRN network is used as the correction network part, and SEED network is used as the recognition part. The model is trained using the validation set, and the parameters with the best training effect are retained. Pictures in natural scenes are input to the model to perform automatic text detection and recognition tasks. Figure 5 This is a diagram of the effect of each part of an image in a natural scene as it is positioned and recognized by the model.

Claims

1. A natural scene text detection and recognition method based on the CRAFT and SCRN-SEED framework, characterized in that: The steps include: (1) Use real datasets and synthetic datasets to build an image dataset, and divide the image dataset into a training set and a test set; (2) Training the CRAFT network using image datasets: (201) The CRAFT network was improved, and the ResNet50 network was used as the backbone network. The images in the synthetic dataset were input into the improved CRAFT network for feature extraction, and the region score and affinity score were output; (202) Encoding the two scores through Gaussian heat mapping to generate a Gaussian heat map; (203) cutting the complete text in the input image into individual characters according to the watershed algorithm, and generating polygons of arbitrary shapes of text from the characters through post-processing operations; (203) Applying the idea of ​​transfer learning, the improved CRAFT network is initialized using the pre-trained model; (3) Use real data sets to train the irregular text correction network SCRN; (4) Combine SCRN with SEED network and train the combined SCRN-SEED network; (5) Connect the improved CRAFT network with the SCRN-SEED network to build a complete model and train it; The steps of combining the SCRN with the SEED network and training the combined SCRN-SEED network include: The trained SCRN network is used to replace the image correction module in the SEED network. The pre-trained language model of the semantic model FastText is used to initialize the parameters of the pre-trained model. The improved SCRN-SEED network is preliminarily trained using the test set, and the network parameters are adjusted according to the training results.

2. The natural scene text detection and recognition method according to claim 1, characterized in that: The steps of applying the idea of ​​transfer learning and initializing the improved CRAFT network using the pre-trained model include: First, we train the CRAFT network using synthetic datasets, optimize the network using the Adam optimizer, and then fine-tune the network using multiple real datasets. During fine-tuning, we use the SynthText dataset at a ratio of 1:5 and online hard example mining at a ratio of 1:

3. Then, the CRAFT network was trained using a real dataset containing quadrilateral annotations and a SynthText dataset, and a part of it was divided into a test set to adjust the network parameters.

3. The natural scene text detection and recognition method according to claim 1, characterized in that: The steps of connecting the improved CRAFT network with the SCRN-SEED network, building a complete model and training it include: generating a minimum rectangular frame containing all characters from a polygon of text of any shape, cropping the rectangular frame, adjusting the format of the cropped image, and then inputting it into the SCRN-SEED network to complete the construction of the model, training the model using a validation set, retaining the parameters with the best training effect, inputting images in natural scenes into the model, and performing automatic text detection and recognition tasks.

4. The natural scene text detection and recognition method according to claim 1, characterized in that: The real datasets are from ICDAR2013, ICDAR2015, ICDAR2017, MSRA-TD500, TotalText, and CTW-1500 databases, and the synthetic dataset is the SynthText dataset; Adjust the size of each image in the image dataset and convert the image format in the dataset to mdb format.

Images