A backbone model pre-training method for scene text detection and recognition

Abstract

The embodiment of the present application relates to the technical field of machine learning, in particular to a backbone model pre-training method for scene text detection and recognition. The method comprises: obtaining a training sample data set D by using a virtual sample generation method and a calibration sample correction method, each sample in the data set D comprising a scene text image I, a target binary image I d and a text list T three parts; randomly selecting a batch of data D1 from the data set D; pre-processing the data D1 to obtain data D2; training a deep neural network model using the data D2; repeating the data selection and model training operations until the pre-training target is completed. The present application aims at the shortcomings of the ODM pre-training algorithm. Firstly, the text positions in the target binary image and the original image are aligned by using a feature point matching and distortion correction algorithm. Compared with the ODM algorithm, the text position alignment error of the present application is smaller, which reduces the complexity of the problem to a certain extent and is beneficial to the rapid training of the model and the improvement of the generalization ability. Then, the mask method is applied to the original image and the target image at the same time to achieve the purpose of increasing the diversity of the samples, which is beneficial to improving the generalization performance of the pre-training model encoder and decoder. The pre-trained model of the present application is beneficial to improving the accuracy of scene text detection and recognition.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a method for pre-training a backbone model for scene text detection and recognition. Background Technology

[0002] In recent years, text-image joint pre-training techniques have shown promising results in various tasks. In optical character recognition (OCR) tasks, aligning text instances with their corresponding text regions in an image is challenging because it requires effective alignment between the text and the OCR-Text (referred to as OCR-Text in the image to distinguish it from text in natural language), rather than a comprehensive understanding of the overall image content. OCR-Text destylization modeling (ODM) pre-training methods can convert text of various styles found in an image into a unified style based on text prompts, achieving good alignment between the text and the OCR-Text and enabling the pre-trained model to adapt to complex and diverse text detection and recognition tasks. However, in the ODM method, there is a significant positional deviation between the target binarized label image of the training image and the text in the original image, increasing the difficulty of network training. Furthermore, the ODM method only performs masking and random perturbation operations on the text, without masking the image itself. These factors result in only a limited performance improvement for models pre-trained using the ODM method. Summary of the Invention

[0003] To address the problems existing in the ODM method, this invention provides a backbone model pre-training method for scene text detection and recognition to improve the aforementioned issues. To achieve the above objective, the technical solution adopted by this invention is as follows:

[0004] Firstly, a method is provided for judging the correctness of answers to handwritten multiple-choice questions in scanned documents, including:

[0005] The training sample dataset D is obtained using virtual sample generation and calibration sample correction methods. Each sample in dataset D includes a scene text image I and a target binary image I. d The three parts are: text list T;

[0006] Randomly select a batch of data D1 from dataset D; preprocess data D1 to obtain data D2; use data D2 to train the deep neural network model once; repeat this step until the pre-training objective is achieved.

[0007] Furthermore, the virtual sample generation method is as follows:

[0008] The SynthText virtual sample generation method is used to generate scene text image I and target binary image I. d And the text list T.

[0009] Furthermore, the calibration sample correction method includes the following steps:

[0010] For a real sample image I labeled with a rectangular or polygonal box, a target binary image I1 is generated using the target binary image generation method in the ODM method;

[0011] Initialize image I d A background image consisting entirely of zeros;

[0012] For each calibrated text word region, the following steps are performed: First, obtain the image blocks corresponding to the word in image I and image I1, and calculate the registration points of the two image blocks using the image feature point matching method; then, correct the registration points using the camera distortion correction method; finally, correct the image block in image I1 using the perspective transformation method and place it into image I. d The corresponding position;

[0013] Treat the labeled text as a text list T.

[0014] Furthermore, the step of preprocessing data D1 to obtain data D2 includes:

[0015] Perform random deletion operations on the text list T;

[0016] Insert random text into text list T;

[0017] Perform a random masking operation on image I, and then apply a random masking operation to image I. d Perform the same masking operation.

[0018] Furthermore, the deep neural network model includes a text encoder module, an image encoder module, and an image decoder module. The text encoder module adopts the commonly used text transformer model, the image encoder module adopts the ResNet50 model, and the image decoder module consists of multiple upsampling layers and convolutional layers.

[0019] Furthermore, the pre-training objective is to reach a predetermined number of training rounds.

[0020] The beneficial effects of this invention are:

[0021] This invention addresses the shortcomings of the ODM pre-training algorithm. First, it aligns the text positions in the target binary image and the original image using feature point matching and distortion correction algorithms. Compared to the ODM algorithm, this invention's method has a smaller text alignment error, reducing the complexity of the problem and facilitating faster model training and improved generalization ability. Then, it simultaneously applies masks to both the original and target images to increase sample diversity, which improves the generalization performance of the pre-trained model's encoder and decoder. The model pre-trained using this invention enhances the accuracy of scene text detection and recognition. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a backbone model pre-training method for scene text detection and recognition provided in an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram illustrating the text alignment effect according to an embodiment of the present invention;

[0025] Figure 3 This is a diagram of a pre-trained neural network model architecture provided in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0027] As mentioned earlier, in related technologies, the target binary label image of the training image in the original ODM method has a large positional deviation from the text in the original image, increasing the difficulty of network training. Furthermore, the ODM method only performs masking and random perturbation operations on the text, without masking the image itself. These factors result in minimal performance improvement for models pre-trained using the ODM method. This invention addresses the shortcomings of the ODM pre-training algorithm by first aligning the text positions in the target binary image and the original image through feature point matching and distortion correction algorithms. Compared to the ODM algorithm, this invention's method has a smaller text alignment error, reducing the complexity of the problem to some extent and facilitating faster model training and improved generalization ability. Then, by simultaneously applying masks to both the original and target images, the diversity of samples is increased, which helps improve the generalization performance of the pre-trained model's encoder and decoder. The model pre-trained using this invention improves the accuracy of scene text detection and recognition. The specific implementation of the above concept is described below.

[0028] This invention provides a backbone model pre-training method for scene text detection and recognition, the method comprising:

[0029] Step 100: Obtain the training sample dataset using a virtual sample generation method. The steps include:

[0030] The SynthText virtual sample generation method is used to generate scene text image I and target binary image I. d and text list T;

[0031] Step 101: Obtain the training sample dataset using the calibration sample correction method. The steps include:

[0032] For a real sample image I labeled with a rectangular or polygonal box, a target binary image I1 is generated using the target binary image generation method in the ODM method;

[0033] Initialize image I d A background image consisting entirely of zeros;

[0034] For each calibrated text word region, the following steps are performed: First, obtain the image patch corresponding to the word in image I and image I1 based on the calibration data. Then, detect feature points in image I and image I1 using the Harris corner detection method, and extract Sift feature descriptors for each feature point. Finally, calculate the registration points of the two image patches using the Sift feature point matching method. After obtaining the registration points, correct the registration points using OpenCV's camera distortion correction method. Finally, correct the image patch in image I1 using the perspective transformation method and place it into image I. d The corresponding position; Figure 2The diagram shows the target binary image obtained using the ODM method and the target binary image obtained using the method of this invention. It can be seen that the text position error in the target binary image obtained by this invention is smaller.

[0035] Treat the labeled text as a text list T.

[0036] Step 102: Merge the datasets obtained in Step 100 and Step 101 to obtain the training sample dataset D. Each sample in dataset D includes a scene text image I and a target binary image I. d It consists of three parts: a text list T, and a text list T.

[0037] Step 103: Randomly select a batch of data D1 from dataset D, and preprocess data D1 to obtain data D2. The steps include:

[0038] Randomly select a batch of data D1 from dataset D;

[0039] Perform random deletion operations on the text list T;

[0040] Insert random text into text list T;

[0041] Perform a random masking operation on image I, and then apply a random masking operation to image I. d Perform the same masking operation.

[0042] Step 104: Train the deep neural network model once using data D2. In this embodiment of the invention, the network structure of the neural network model is as follows: Figure 3 As shown, the text encoder module uses the commonly used text transformer model, the image encoder module uses the ResNet50 model, and the image decoder module consists of multiple upsampling layers and convolutional layers. The target size of the decoder is the same as the input image size. Figure 3 The red dashed box represents a random mask operation, and the green dashed line represents a random text deletion operation.

[0043] Step 105: Repeat steps 103 and 104 to train the model 1000 times.

[0044] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0045] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A backbone model pre-training method for scene text detection and recognition, characterized in that, include: The training sample dataset D is obtained using virtual sample generation and calibration sample correction methods. Each sample in dataset D includes a scene text image I and a target binary image I. d The three parts are: text list T; Randomly select a batch of data D1 from dataset D; preprocess data D1 to obtain data D2; use data D2 to train the deep neural network model once; repeat this step until the pre-training objective is achieved.

2. The method according to claim 1, characterized in that, The virtual sample generation method is as follows: The SynthText virtual sample generation method is used to generate scene text image I and target binary image I. d And the text list T.

3. The method according to claim 1, characterized in that, The calibration sample correction method includes the following steps: For a real sample image I labeled with a rectangular or polygonal box, a target binary image I1 is generated using the target binary image generation method in the ODM method; Initialize image I d A background image consisting entirely of zeros; For each calibrated text word region, the following steps are performed: First, obtain the image blocks corresponding to the word in image I and image I1, and calculate the registration points of the two image blocks using the image feature point matching method; then, correct the registration points using the camera distortion correction method; finally, correct the image block in image I1 using the perspective transformation method and place it into image I. d The corresponding position; Treat the labeled text as a text list T.

4. The method according to claim 1, characterized in that, The step of preprocessing data D1 to obtain data D2 includes: Perform random deletion operations on the text list T; Insert random text into text list T; Perform a random masking operation on image I, and then apply a random masking operation to image I. d Perform the same masking operation.

5. The method according to claim 1, characterized in that, The deep neural network model includes a text encoder module, an image encoder module, and an image decoder module. The text encoder module adopts the commonly used text transformer model, the image encoder module adopts the ResNet50 model, and the image decoder module consists of multiple upsampling layers and convolutional layers.

6. The method according to claim 1, characterized in that, The pre-training objective is to reach a predetermined number of training rounds.

Images