Tibetan recognition method in natural scenes based on baseline localization and sequence modeling

Abstract

The present invention belongs to the technical field of text recognition, and specifically relates to a Tibetan recognition method in a natural scene based on baseline positioning and sequence modeling. The present invention includes: 1) constructing a Tibetan image data set in a natural scene and annotating it; 2) expanding the data set and preprocessing the data set image; 3) constructing a baseline detector to locate the baseline position of the Tibetan text; 4) constructing a feature extractor to obtain the sequence feature representation of the horizontal and vertical text images using different feature extraction methods; 5) constructing a TCN-based sequence modeling network to model the sequence relationship; 6) constructing a CTC transcription module to send the output features representing the sequence relationship into the transcription layer to obtain the result of Tibetan recognition. Through the above technical solution, the present invention greatly enhances the effectiveness of feature extraction and sequence relationship modeling, improves the parallel processing capability of the sequence, and thus improves the accuracy and efficiency of Tibetan recognition in natural scenes.

Description

Technical Field

[0001] The present invention relates to the fields of text recognition, computer vision and deep learning, and in particular to a Tibetan recognition method in natural scenes based on baseline positioning and sequence modeling. Background Art

[0002] In the current digital era, people need to process a large amount of text information in their lives and production, which consumes a lot of energy and time. In this case, intelligent recognition of text content has high research value and application value. Text recognition in natural scenes is widely used in tasks such as document retrieval, reading, information recognition, and machine translation. It is a key task in the field of computer vision. Tibetan is a language with a long history and wide application. The recognition of Tibetan text has important research value and significance. However, the current text recognition methods in natural scenes are mainly aimed at Chinese and English, and there is little research on Tibetan recognition.

[0003] At present, the recognition of Tibetan characters in natural scenes is mainly based on deep learning algorithms, and the mainstream method is to use convolutional recurrent neural networks to complete Tibetan recognition. This type of method extracts features through convolutional neural networks and then sends them to recurrent neural networks to model sequence relationships, and finally transcribes and outputs recognition results. However, this type of method will compress the features along the height direction, and cannot realize the recognition of vertical Tibetan characters. In addition, due to the limitations of the sequence structure characteristics of the recurrent neural network itself, it must be reasoned step by step in chronological order, and large-scale parallel processing cannot be performed, which is time-consuming and labor-intensive to calculate. In addition, due to the limited number of Tibetan characters in natural scenes that can be collected, the model training is insufficient and the performance is poor. Therefore, the efficiency and accuracy of existing Tibetan recognition methods in natural scenes are generally low. Summary of the invention

[0004] In view of the problems in the prior art of insufficient Tibetan image data in natural scenes, low Tibetan recognition accuracy and efficiency, and inability to recognize vertical text, the present invention intends to provide a Tibetan recognition method in natural scenes based on baseline positioning and sequence modeling, with the purpose of providing a synthetic algorithm to expand the data set to alleviate the impact of insufficient data on model training, realize the recognition of vertical text through baseline positioning and window division, and use a parallel and bidirectional TCN network to support parallel processing while ensuring the ability to extract sequence-related information, thereby improving the performance of Tibetan recognition and improving the problems existing in the existing methods.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A Tibetan recognition method in natural scenes based on baseline positioning and sequence modeling, comprising:

[0007] Step 1: Collect natural scene images containing Tibetan text, crop all Tibetan text image blocks, perform perspective transformation on the Tibetan text image blocks, and then annotate the text baseline and content to obtain a basic Tibetan text image dataset;

[0008] Step 2: Synthesize the Tibetan text image augmented dataset through the image synthesis algorithm, and preprocess the Tibetan text image basic dataset and the Tibetan text image augmented dataset;

[0009] Step 3: Build a text baseline detector based on a convolutional neural network, use the text baseline detector to extract Tibetan text image features, predict the baseline height position of the text, and obtain the baseline positioning result after post-processing;

[0010] Step 4: Determine the text type based on the baseline positioning result in step 3, build a feature extraction network corresponding to different text types, and obtain the sequence features of the Tibetan text image;

[0011] Step 5: Construct a sequence modeling network based on two parallel and reverse TCN networks, and feed the sequence features extracted in step 4 into the two TCN networks to obtain output features that characterize the sequence relationship;

[0012] Step 6: Construct a CTC transcription module based on the CTC algorithm, send the output features in step 5 to the CTC transcription module to obtain the Tibetan recognition result, and optimize the model based on the recognition result through the joint loss function.

[0013] Furthermore, the specific steps of step 1 are:

[0014] Step 1.1: Collect natural scene images containing Tibetan text from online maps, social software, and search engines;

[0015] Step 1.2: Crop all the Tibetan text image blocks and transform them into orthographic images. Annotate the baseline height position and content of the text to obtain the basic Tibetan text image data set.

[0016] Furthermore, the specific steps of synthesizing the Tibetan text image expansion data set through the image synthesis algorithm in step 2 are:

[0017] Step 2.1: Collect Tibetan text data from Tibetan websites and remove non-Tibetan characters;

[0018] Step 2.2: randomly divide the text data in step 2.1 into Tibetan texts of different lengths to construct a horizontal text database;

[0019] Step 2.3: Segment the text data in step 2.1 into individual syllables according to Tibetan syllable separators to construct a vertical text database;

[0020] Step 2.4: Collect natural scene images without text and build a natural scene background library;

[0021] Step 2.5: Synthesize Tibetan text images: For horizontal text, randomly select Tibetan text from the horizontal text database and render it on the natural scene image in the natural scene background library; for vertical text, randomly select 3 to 5 Tibetan syllables from the vertical text database and render them from top to bottom on the natural scene image in the natural scene background library;

[0022] Step 2.6: Crop the text area from the Tibetan text image synthesized in step 2.5, and record the baseline height and content of the text to obtain the Tibetan text image augmented dataset.

[0023] Furthermore, the specific steps of preprocessing in step 2 are:

[0024] The resolution of Tibetan text images in the Tibetan text image basic dataset and the Tibetan text image expansion dataset is changed to 320×320, and the training set and test set are divided according to r1:r2. For the images in the training set, the probability p s Randomly add Gaussian noise to the image for data augmentation.

[0025] Furthermore, the specific steps of extracting Tibetan text image features using the text baseline detector in step 3 are as follows:

[0026] Step 3.1: For the input image data with a shape of 3×H×W, use the text baseline detector to extract the Tibetan text block features and transform its shape to C×H×W / 4, where C is the feature dimension, H is the image height, and W is the image width;

[0027] Step 3.2: The Tibetan text block features are further compressed through the global average pooling layer, and then the probability p of the baseline distribution at a certain height in the image is predicted after Sigmoid activation through the fully connected layer. The expression is:

[0028] p = Sigmoid(FC(GAP(X f )))

[0029] Where X f Indicates characteristics;

[0030] Step 3.3: Set the probability threshold to p thres , if the probability p is greater than p thres It is determined that there is a baseline at this height, and the spacing threshold D is set to merge similar pixels to obtain the number of baselines N. L And the location information P N , the baseline detector is optimized based on the binary cross entropy loss as follows:

[0031]

[0032] Among them, y h represents the label value at height h, p h is the probability of the presence of a baseline at height h predicted by the network.

[0033] Furthermore, the specific steps of step 4 are:

[0034] Step 4.1: Determine the type of text in the image based on the baseline positioning result in step 3. If only one baseline is detected, it is determined to be horizontal text, otherwise it is determined to be vertical text;

[0035] Step 4.2: For images judged to be horizontal text, extract text image sequence features based on MobileNetV3; for images judged to be vertical text, first split them into N according to the number of baselines. L The window is scaled to 32×320 size, and the sequence features are extracted based on MobileNetV3 in the divided windows. Then the window features are spliced ​​along the sequence length and the sequence length is adjusted through the global average pooling layer.

[0036] Furthermore, the specific steps in step 5 are:

[0037] Step 5.1: Send the sequence features extracted in step 4 to two parallel and reverse TCN networks to obtain output features that represent the sequence relationship and L represents the sequence length;

[0038] Step 5.2: The output features X of the two TCN networks F and X R Splice to get the sequence modeling module output X c :

[0039] X c =concat(X F ,X R )

[0040] in, and (n=1,2,...,L) corresponding splicing;

[0041] Step 5.3: During training, X F and X R Mapping a Classifier to the annotation space to obtain the probability distribution and Use KL divergence-based consistency loss The two-way TCN network is optimized, and the expression is:

[0042]

[0043] in (i=1,2,...,L) represents the probability distribution of the output character at time step i;

[0044] Two KL divergences and The calculation is as follows:

[0045]

[0046]

[0047] in Indicates that the output at time step i is character x j The probability of N is the total number of characters.

[0048] Furthermore, step 6 specifically includes the following steps:

[0049] Step 6.1: Map the output of step 5 to the annotation space based on the CTC algorithm, and then decode it by merging repeated characters and removing the empty character ε to obtain the final recognition result Y;

[0050] Step 6.2: Calculate the CTC loss using negative log conditional probability, which is defined as follows:

[0051]

[0052] Where Ω is the training set, L i , l i Respectively represent the i-th Tibetan text image and text label, Y i Indicates that L i The output after passing through the feature extraction network and the sequence modeling network;

[0053] Step 6.3: When training, use CTC loss Consistency loss Baseline localization loss The joint loss of the three losses optimizes the parameters of the entire network. The joint loss is as follows:

[0054]

[0055] Among them, α and β are balance coefficients.

[0056] In summary, by adopting the above technical solution, the beneficial effects of the present invention are:

[0057] 1. The present invention synthesizes natural scene text image expansion data set through text rendering, Poisson fusion and other methods, solving the problem of insufficient natural scene Tibetan text image data;

[0058] 2. The present invention constructs a baseline detector to determine the text category, realizes vertical text recognition based on window division and feature splicing, and solves the problem that recurrent convolutional neural network cannot recognize vertical text.

[0059] 3. The present invention adopts a parallel, bidirectional temporal convolutional network to model sequence relationships, obtains sufficient context information, realizes large-scale parallel processing, and improves the accuracy and efficiency of Tibetan recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 This is a schematic diagram of the Tibetan character recognition process;

[0062] Figure 2 It is a flow chart of data synthesis and preprocessing in the present invention;

[0063] Figure 3 It is the structural diagram of the baseline detector in the present invention;

[0064] Figure 4 This is a feature extraction network structure diagram in the present invention;

[0065] Figure 5 This is a diagram of the Tibetan recognition network structure based on TCN in the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] The purpose of the present invention is to provide a Tibetan recognition method in natural scenes based on baseline positioning and sequence modeling, which has the characteristics of high Tibetan recognition accuracy and efficiency.

[0068] In order to make the objects, features and advantages of the present invention more obvious and understandable, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0069] like Figure 1 As shown, the present invention proposes a Tibetan recognition method in a natural scene based on baseline positioning and sequence modeling, comprising the following steps:

[0070] Step 1: Collect natural scene images containing Tibetan text, crop all Tibetan text image blocks, perform perspective transformation on the Tibetan text image blocks, and then annotate the text baseline and content to obtain a basic Tibetan text image dataset;

[0071] Step 2: Synthesize the Tibetan text image augmented dataset through the image synthesis algorithm, and preprocess the Tibetan text image basic dataset and the Tibetan text image augmented dataset;

[0072] Step 3: Build a text baseline detector based on a convolutional neural network, use the text baseline detector to extract Tibetan text image features, predict the baseline height position of the text, and obtain the baseline positioning result after post-processing;

[0073] Step 4: Build a feature extractor to determine the text type (horizontal or vertical) based on the baseline positioning results of step 3. For different types of text, different feature extraction methods are used based on MobileNetV3 to obtain the sequence features of the text image.

[0074] Step 5: Construct a sequence modeling network based on two parallel and reverse TCN networks, and feed the sequence features extracted in step 4 into the two TCN networks to obtain output features that characterize the sequence relationship;

[0075] Step 6: Construct a CTC transcription module based on the CTC algorithm, send the output features in step 5 to the CTC transcription module to obtain the Tibetan recognition result, and optimize the model based on the recognition result through the joint loss function.

[0076] The following is a further detailed description of the above-mentioned steps and features in conjunction with specific embodiments:

[0077] In step 1: collect natural scene images containing Tibetan text from network maps, social software, search engines, etc., crop all Tibetan text image blocks and transform them into orthographic images. Annotate the baseline height position h and content c of the Tibetan text to obtain the basic dataset of Tibetan text images, where the baseline position h is in pixels.

[0078] In step 2: expand the data set and preprocess the image data. First, collect Tibetan text data from Tibetan websites and remove non-Tibetan characters. Randomly divide the Tibetan text data into different lengths (in syllables) to construct a horizontal text database d1; divide the text data into syllables to construct a vertical text database d2. Then collect natural scene images without text from the Internet to construct a natural scene background library d3. For horizontal text image synthesis, randomly select text from d1, select images from d3, and render the text to the image; for vertical text image synthesis, randomly select 3 to 5 syllables from d2, select images from d3, and render the syllables from top to bottom to the image. For the synthesized image, crop the text area and record the baseline position h and text content c to obtain the Tibetan text image expansion data set. Change the resolution of all text images to 320×320, and divide the training set and test set according to r1:r2. In this example, r1:r2=7:3. For the training set images, with probability p s Randomly add Gaussian noise to the image for data enhancement. In this example, p s = 0.5. The detailed image synthesis and preprocessing steps are as follows Figure 2 shown.

[0079] In step 3: construct a text baseline detector, extract text image features and predict the text baseline position in the image. First, send the Tibetan text image X into the convolutional neural network to extract features and obtain the feature map X f Specifically, the feature extraction network consists of four convolution blocks, each of which contains three consecutive 3×3 convolution layers or 5×5 convolution layers, and the number of channels increases with depth. f After passing through a global average pooling layer and then through a fully connected layer after Sigmoid activation, the probability p of the baseline distribution at a certain height in the image is predicted, and its expression is:

[0080] p = Sigmoid(FC(GAP(X f )))

[0081] If the probability value is greater than p thres , it is considered that there is a baseline here, and the spacing threshold D is set to merge the similar pixels, so as to obtain the number of baselines N L And the location information P N In this example, p thres = 0.5, D = 25. The baseline detection network is optimized using binary cross entropy loss as follows:

[0082]

[0083] Among them, y hrepresents the label value at height h (0 or 1, 0 means there is no baseline, 1 means there is a baseline), p h The probability that there is a baseline at the height h predicted by the network;

[0084] In step 4: according to the number of baselines N output by the baseline detector L , if N L =1, the text is considered horizontal text. L ≥2, the text is considered vertical text. According to different texts, the MobileNetV3 network is used to make corresponding adjustments to extract sequence features. For horizontal text images, the feature extraction network structure is: 3×3 convolutional layer (s=(2,1))-BneckBlocks1-1×1 convolutional layer-5×5 maximum pooling layer. For vertical text images, it is divided into N according to the number of baselines. L window, Scaled to 32×320, the features are extracted in different windows and then spliced ​​along the length of the sequence, and the sequence length is adjusted through the global average pooling layer. Specifically, the feature extraction network structure of vertical text is: 3×3 convolution layer (s=(2,2))-BneckBlocks2-1×1 convolution layer-2×2 maximum pooling layer. Both BneckBlocks1 and BneckBlocks2 are composed of a series of bneck basic units. The output of the sequence feature extraction network is C represents the feature dimension, and L represents the sequence length. In this example, C=512, L=64.

[0085] In step 5: Sequence feature X sf Send them to the forward and reverse TCN networks respectively to obtain the output features that represent the sequence relationship and Both the forward and reverse networks are composed of two residual blocks. The difference is that the convolution in the forward TCN is a dilated causal convolution, and the convolution in the reverse TCN is a dilated anti-causal convolution. The specific structure is as follows Figure 5 As shown. F and X R Splice to get module output X c :

[0086] X c =concat(X F ,X R )

[0087] in, and (n=1,2,...,L) corresponding splicing. During training, X F and X R Mapping a Classifier to the annotation space to obtain the probability distribution and Use KL divergence-based consistency loss The above network is optimized and its expression is:

[0088]

[0089] in (i=1,2,...,L) represents the output character probability distribution at time step i.

[0090] Two KL divergences and The calculation is as follows:

[0091]

[0092]

[0093] in Indicates that the output at time step i is character x j The probability of N is the total number of characters.

[0094] In step 6: Based on the CTC algorithm, feature X c Map it to the annotation space, then decode it by merging repeated characters and removing empty characters ε to get the final recognition result Y. The CTC module loss is calculated as follows:

[0095]

[0096] Where Ω is the training set, L i , l i Respectively represent the i-th Tibetan text image and text label, Y i Indicates that L i The output after passing through the feature extraction network and the sequence modeling network. Using CTC loss Consistency loss Baseline localization loss The joint loss of the three losses optimizes the parameters of the entire Tibetan recognition network. The joint loss is as follows:

[0097]

[0098] Wherein α and β are balance coefficients. In this example, α=β=0.5.

[0099] The specific implementation described above is only an optimal implementation mode of the present invention and is not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by utilizing the spirit and principle of the present invention and the contents of the drawings should be included in the patent protection scope of the present invention.

Claims

1. A Tibetan character recognition method in natural scenes based on baseline positioning and sequence modeling, characterized in that: include: Step 1: Collect natural scene images containing Tibetan text, crop all Tibetan text image blocks, perform perspective transformation on the Tibetan text image blocks, and then annotate the text baseline and content to obtain a basic Tibetan text image dataset; Step 2: Synthesize the Tibetan text image augmented dataset through the image synthesis algorithm, and preprocess the Tibetan text image basic dataset and the Tibetan text image augmented dataset; Step 3: Build a text baseline detector based on a convolutional neural network, use the text baseline detector to extract Tibetan text image features, predict the baseline height position of the text, and obtain the baseline positioning result after post-processing; Step 4: Determine the text type based on the baseline positioning result in step 3, build a feature extraction network corresponding to different text types, and obtain the sequence features of the Tibetan text image; Step 5: Construct a sequence modeling network based on two parallel and reverse TCN networks, and feed the sequence features extracted in step 4 into the two TCN networks to obtain output features that characterize the sequence relationship; Step 6: Construct a CTC transcription module based on the CTC algorithm, send the output features in step 5 to the CTC transcription module to obtain the Tibetan recognition result, and optimize the model based on the recognition result through the joint loss function.

2. According to the method for Tibetan recognition in natural scenes based on baseline positioning and sequence modeling as described in claim 1, it is characterized in that: The specific steps of step 1 are: Step 1.1: Collect natural scene images containing Tibetan text from online maps, social software, and search engines; Step 1.2: Crop all Tibetan text image blocks and transform them into orthographic images. Annotate the baseline height position and content of the text to obtain a basic Tibetan text image dataset.

3. The Tibetan character recognition method in natural scenes based on baseline positioning and sequence modeling according to claim 1, characterized in that: The specific steps of synthesizing Tibetan text images to expand the dataset through the image synthesis algorithm in step 2 are: Step 2.1: Collect Tibetan text data from Tibetan websites and remove non-Tibetan characters; Step 2.2: randomly divide the text data in step 2.1 into Tibetan texts of different lengths to construct a horizontal text database; Step 2.3: Segment the text data in step 2.1 into individual syllables according to Tibetan syllable separators to construct a vertical text database; Step 2.4: Collect natural scene images without text and build a natural scene background library; Step 2.5: Synthesize Tibetan text images: For horizontal text, randomly select Tibetan text from the horizontal text database and render it on the natural scene image in the natural scene background library; for vertical text, randomly select 3 to 5 Tibetan syllables from the vertical text database and render them from top to bottom on the natural scene image in the natural scene background library; Step 2.6: Crop the text area from the Tibetan text image synthesized in step 2.5, and record the baseline height and content of the text to obtain the Tibetan text image augmented dataset.

4. The Tibetan character recognition method in natural scenes based on baseline positioning and sequence modeling according to claim 3, characterized in that: The specific steps of preprocessing in step 2 are: The resolution of Tibetan text images in the Tibetan text image basic dataset and the Tibetan text image expansion dataset is changed to 320×320, and the training set and test set are divided according to r1:r2. For the images in the training set, the probability p s Randomly add Gaussian noise to the image for data augmentation.

5. The Tibetan character recognition method in natural scenes based on baseline positioning and sequence modeling according to claim 1, characterized in that: The specific steps of using the text baseline detector to extract Tibetan text image features in step 3 are as follows: Step 3.1: For the input image data with a shape of 3×H×W, use the text baseline detector to extract the Tibetan text block features and transform its shape to C×H×W / 4, where C is the feature dimension, H is the image height, and W is the image width; Step 3.2: The Tibetan text block features are further compressed through the global average pooling layer, and then the probability p of the baseline distribution at a certain height in the image is predicted after Sigmoid activation through the fully connected layer. The expression is: p=Sigmoid(FC(GAP(X f ))) Where X f Indicates characteristics; Step 3.3: Set the probability threshold to p thres , if the probability p is greater than p thres It is determined that there is a baseline at this height, and the spacing threshold D is set to merge similar pixels to obtain the number of baselines N. L And the location information P N , the baseline detector is optimized based on the binary cross entropy loss as follows: Among them, y h represents the label value at height h, p h is the probability of the presence of a baseline at height h predicted by the network.

6. The Tibetan character recognition method in natural scene based on baseline positioning and sequence modeling according to claim 1, characterized in that: The specific steps of step 4 are: Step 4.1: Determine the type of text in the image based on the baseline positioning result in step 3. If only one baseline is detected, it is determined to be horizontal text, otherwise it is determined to be vertical text; Step 4.2: For images judged to be horizontal text, extract text image sequence features based on MobileNetV3; for images judged to be vertical text, first split them into N according to the number of baselines. L The window is scaled to 32×320 size, and the sequence features are extracted based on MobileNetV3 in the divided windows. Then the window features are concatenated along the sequence length and the sequence length is adjusted through the global average pooling layer.

7. The Tibetan character recognition method in natural scenes based on baseline positioning and sequence modeling according to claim 1, characterized in that: The specific steps in step 5 are: Step 5.1: Send the sequence features extracted in step 4 to two parallel and reverse TCN networks to obtain output features that represent the sequence relationship and L represents the sequence length; Step 5.2: The output features X of the two TCN networks F and X R Splice to get the sequence modeling module output X c : X c =concat(X F ,X R ) in, and Corresponding splicing; Step 5.3: During training, X F and X R Mapping a Classifier to the annotation space to obtain the probability distribution and Use KL divergence based consistency loss The two-way TCN network is optimized, and the expression is: in represents the probability distribution of the output character at time step i; Two KL divergences and The calculation is as follows: in Indicates that the output at time step i is character x j The probability of N is the total number of characters.

8. The Tibetan character recognition method in natural scenes based on baseline positioning and sequence modeling according to claim 1, characterized in that: Step 6 specifically includes the following steps: Step 6.1: Map the output of step 5 to the annotation space based on the CTC algorithm, and then decode it by merging repeated characters and removing the empty character ε to obtain the final recognition result Y; Step 6.2: Calculate the CTC loss using negative log conditional probability, which is defined as follows: Where Ω is the training set, L i , l i Respectively represent the i-th Tibetan text image and text label, Y i Indicates that L i The output after passing through the feature extraction network and the sequence modeling network; Step 6.3: When training, use CTC loss Consistency loss Baseline localization loss The joint loss of the three losses optimizes the parameters of the entire network. The joint loss is as follows: Among them, α and β are balance coefficients.

Images