A method for Hindi script recognition based on enhanced visual transformer network

Abstract

The application discloses a Hindi text image recognition method based on an enhanced visual transformer network, and steps of the method comprise the following steps: 1, synthesizing a Hindi text image, and establishing a Hindi text image recognition training data set; 2, constructing a Hindi text image recognition network; 3, calculating a loss function corresponding to each input picture and training the Hindi text image recognition network; 4, using the trained Hindi text image recognition network to perform text recognition on an input picture to be recognized. After the Hindi text image recognition network is trained, the Hindi text image recognition network can perform text recognition on a Hindi text image which does not appear in the training data, and has high practical value.

Description

Technical Field

[0001] This invention relates to problems in the field of Hindi text image recognition, specifically to a Hindi text image recognition method based on an enhanced visual transformer network. Background Technology

[0002] Current Hindi image recognition methods are mostly based on deep learning techniques. Deep learning is a data-driven approach, and the quantity and quality of training data have a crucial impact on the model's performance. Hindi, as a low-resource language, suffers from a scarcity of authentic image data. Furthermore, in terms of recognition models, most current Hindi image recognition networks employ encoders based on convolutional neural networks and decoders based on temporal classification. This design approach limits the performance of Hindi image recognition models. Summary of the Invention

[0003] This invention aims to overcome the shortcomings of existing technologies by proposing a Hindi image recognition method based on an enhanced visual transformer network, which is intended to efficiently complete Hindi image recognition tasks and thus improve the accuracy of Hindi text image recognition.

[0004] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0005] The Hindi image recognition method based on an enhanced visual transformer network of the present invention is characterized by the following steps:

[0006] Step 1: Construct a Hindi image recognition training dataset;

[0007] A Hindi text image is synthesized by combining a background image without text and Hindi text content, and the Hindi text content is used as the text label for the corresponding Hindi text image, thus obtaining a text image set X = [x1, x2, ..., x...]. i , ..., x N ] and its corresponding label set G = [g1, g2, ..., g i , ..., g N ];in, Let g represent the i-th Hindi text image. i Image x represents the i-th Hindi text. i The corresponding text labels are: H represents the image height, W represents the image width, and 3 represents the number of channels.

[0008] Step 2: Construct a text image recognition network, including an encoder and a decoder, to obtain the i-th Hindi text image x. i Predicted probability P i ;

[0009] Step 3: load the pre-trained parameters of the ViTAEv2 base module in the image classification task, and use them to initialize the encoder parameters in the text image recognition network;

[0010] According to the character label g i and the predicted probability P i After constructing the cross-entropy loss function, the initialized text image recognition network is trained by using the back propagation algorithm until the cross-entropy loss function converges, so as to obtain the trained text image recognition network;

[0011] Step 4: using the trained text image recognition network to recognize any inputted to-be-recognized Hindi text image, obtaining the predicted probability vector of each position character of the to-be-recognized image, and then selecting the category corresponding to the maximum probability in the predicted probability vector as the text recognition result of the to-be-recognized image.

[0012] The Hindi text image recognition method based on the enhanced visual transformer network has the characteristics that the encoder module in step 2 is composed of R stacked ViTAEv2 base modules, wherein the rth ViTAEv2 base encoding module includes a main branch, a bypass branch and a feedforward encoding layer with a residual connection;

[0013] The main branch of the rth ViTAEv2 base encoding module includes a multi-scale convolution operation layer, a layer normalization operation layer and a multi-head attention operation layer;

[0014] The bypass branch of the rth ViTAEv2 base encoding module includes a convolution operation layer, a batch normalization operation layer and a sigmoid weighted linear unit;

[0015] When r=1, the ith Hindi text image x i is input into the text image recognition network, and the multi-scale convolution operation layer of the main branch of the rth ViTAEv2 base encoding module in the encoder is used for feature extraction, outputting a multi-channel feature map M i,r After shape change, it is converted into a feature sequence S i，r After processing by the multi-head attention operation layer, a feature sequence O i，r is obtained.

[0016] The bypass branch of the rth ViTAEv2 base encoding module in the encoder processes the ith Hindi text image x i , and obtains a feature map M' i，r After shape change, it is converted into a feature sequence S' i，r .

[0017] Oi，r and S' of the corresponding position elements are added to obtain the fused sequence f i，r i，r and input into the feedforward encoding layer of the rth ViTAEv2 basic encoding module, and after sequentially passing through the layer normalization operation and the processing of the two fully connected layers with linear rectification activation function, the rth visual feature sequence P i，r is output.

[0018] When r = 2, 3, …, R, the (r-1)th visual feature sequence P i，r-1 is input into the rth ViTAEv2 basic encoding module of the encoder for processing, and the rth visual feature sequence P i，r is obtained accordingly; and the Rth visual feature sequence P i,R output by the Rth ViTAEv2 basic encoding module is taken as the visual feature sequence output by the encoder module. wherein, denotes the jth visual feature vector of the ith Hindi text image x i ; n denotes the total number of visual feature vectors.

[0019] The decoder in the step 2 is stacked by a linear classification layer and K identical Transformer basic decoding modules, and any kth Transformer basic decoding module comprises: a masked multi-head self-attention layer, a multi-head attention layer between encoder-decoder, and a feedforward layer with residual connection.

[0020] When k = 1, the literal label gt i is embedded to obtain an embedded representation result seq i , and input into the kth masked multi-head attention layer for processing to obtain the kth self-attention processing result sequence

[0021] The visual feature sequence is input into the decoder, and after being processed by the kth multi-head attention layer between encoder-decoder, , the kth cross-modal processing result sequence

[0022] After being processed by the kth feedforward layer with residual connection, , the kth processing result sequence of the kth Transformer basic decoding module is obtained.

[0023] When k = 2, 3, …, R, the processing result sequence ​The mask multi-head attention layer in the kth Transformer-based decoding module is input for processing, and the output result is input into the kth encoder-decoder interlayer multi-head attention layer and the feedforward layer with a residual connection for processing, to obtain a processing result sequence of the kth Transformer-based decoding module Thus, the kth processing result sequence is obtained from the kth Transformer-based decoding module

[0024] The linear classification layer uses to make a prediction to obtain the prediction probability of the i-th Hindi text image x i wherein represents the probability of the t-th position character in the i-th Hindi text image x i , and T represents the maximum decoding length.

[0025] The electronic device comprises a memory and a processor, and the memory is configured to store a program supporting the processor to execute any of the Hindi text recognition methods, and the processor is configured to execute the program stored in the memory.

[0026] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the Hindi text recognition methods are executed.

[0027] Compared with the prior art, the present application has the following advantages:

[0028] 1. The present application designs a Hindi text recognition network, which can effectively recognize real Hindi text images in real scenes after being trained on synthetic Hindi text images, and improves the recognition accuracy.

[0029] 2. The structure of the enhanced visual transformer network is introduced into the encoder of the Hindi recognition network, which strengthens the modeling ability of the Hindi recognition encoder when extracting image features. This way makes the Hindi text recognition network able to extract more features conducive to text recognition.

[0030] 3. The present application uses the idea of transfer learning to transfer the knowledge learned by part of the parameters in the enhanced visual transformer network on other task image datasets to Hindi text recognition, which makes up for the lack of Hindi real text data resources. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 ​​is a flow chart of an Hindi language recognition method based on an enhanced visual transformer network according to the present application;

[0032] Figure 2 is a structural diagram of a Hindi language recognition network based on an enhanced visual transformer network according to the present application. DETAILED DESCRIPTION

[0033] In this embodiment, a Hindi language recognition method based on an enhanced visual transformer network includes the following steps as shown in the figure: Figure 1

[0034] Step 1: Construct a Hindi language recognition training data set;

[0035] Synthesize Hindi text images with background images without text and Hindi text content, and take the Hindi text content as the text label of the corresponding Hindi text image, so as to obtain a text image set X = [x1, x2,..., x i ,..., x N ] and its corresponding label set G = [g1, g2,..., g i ,..., g N ]; wherein, represents the i-th Hindi text image, g i represents the corresponding text label of the i-th Hindi text image x i , H represents the height of the image, W represents the width of the image, and 3 is the channel number;

[0036] Step 2: As shown in the figure, construct a text image recognition network, which includes an encoder and a decoder; Figure 2

[0037] The encoder is used to extract and process information in the image that is conducive to recognizing characters in the image. The decoder takes the image features constructed by the encoder as input and decodes the characters contained in the image in a self-recurrent manner.

[0038] Step 2.1: The encoder module is composed of R stacked ViTAEv2 basic modules. The simultaneous inclusion of multi-head self-attention operation and convolution operation enables the ViTAEv2 basic module to take advantage of the advantages of convolution operation and the advantages of transformer network long-distance dependence modeling. The r-th ViTAEv2 basic encoding module includes: a main branch, a bypass branch and a feedforward encoding layer with a residual connection;

[0039] ​​The main branch of the rth ViTAEv2 basic encoding module includes a multi-scale convolution operation layer, a layer normalization operation layer, and a multi-head attention operation layer. The multi-head attention operation is used to model long-term dependency information in the image, and its input is a sequence of vectors. If the output of the upper structure is a multi-channel feature map, it needs to be shape-transformed into a sequence of vectors;

[0040] The bypass branch of the rth ViTAEv2 basic encoding module includes a convolution operation layer, a batch normalization operation layer, and a sigmoid weighted linear unit. If the input of the previous layer is a sequence of vectors, the bypass branch needs to be shape-transformed into a multi-channel feature map before processing;

[0041] When r = 1, the i-th Hindi text image x i is input into the text image recognition network and is feature-extracted by the multi-scale convolution operation layer of the main branch of the rth ViTAEv2 basic encoding module in the encoder, outputting a multi-channel feature map M i,r . After shape transformation, it is converted into a feature sequence S i，r . After processing by the multi-head attention operation layer, a feature sequence O i，r is obtained.

[0042] The bypass branch of the rth ViTAEv2 basic encoding module in the encoder processes the i-th Hindi text image x i , obtaining a feature map M' i，r . After shape transformation, it is converted into a feature sequence S' i，r .

[0043] After adding the corresponding position elements of O i,r and S' i，r , a fused sequence f i，r is obtained. The fused sequence f i，r is input into the feedforward encoding layer of the rth ViTAEv2 basic encoding module, and after processing by the layer normalization operation, two fully connected layers with linear rectifier activation functions, the rth visual feature sequence P i，r is output.

[0044] When r = 2, 3,..., R, the r-1th visual feature sequence P i，r-1 is input into the rth ViTAEv2 basic encoding module of the encoder for processing, and the rth visual feature sequence P i，r is obtained accordingly. Thus, the Rth visual feature sequence P i,R output by the Rth ViTAEv2 basic encoding module is the visual feature sequence output by the encoder module. wherein, indicates the i-th Hindi text image x ithe jth visual feature vector of the i-th image; n denotes the total number of visual feature vectors;

[0045] Step 2.2: The decoder is stacked by a linear classification layer and K identical Transformer-based decoding modules. Any kth Transformer-based decoding module contains: a masked multi-head self-attention layer, a multi-head attention layer between encoder-decoder, a feed-forward layer with residual connection.

[0046] The decoder takes the visual feature sequence constructed by the encoder as input and decodes the characters contained in the image in an autoregressive manner. At each step, the character information predicted by the previous step is fed into the decoder as additional input.

[0047] When k = 1, the text label gt i is embedded to obtain an embedded representation result seq i , which is input into the kth masked multi-head attention layer for processing to obtain the kth self-attention processing result sequence When calculating the self-attention weight, a mask is used before the softmax(·) function to prevent information from flowing from right to left in the decoder, causing label leakage.

[0048] The visual feature sequence is input into the decoder and processed by the kth multi-head attention layer between encoder-decoder and to obtain the kth cross-modal processing result sequence Through the attention between encoder-decoder, the decoder calculates an attention weight for each vector in the vector sequence output by the encoder. If a certain region in the image is important for decoding the current character, the vector corresponding to the region has a higher attention weight. Through weighted summation, the multi-head attention suppresses the features in the encoder output that are irrelevant to the decoding of the current character, and highlights the features in the encoder output that are important for the decoding of the current character.

[0049] The kth feed-forward layer with residual connection processes to obtain the processing result sequence of the kth Transformer-based decoding module

[0050] When k = 2, 3, …, R, the processing result sequence output by the k-1th basic decoder module is input into the masked multi-head attention layer in the kth Transformer-based decoding module for processing, and the output result is After the processing of the multi-head attention layer and the feed-forward layer with a residual connection between the kth encoder-decoder pair, the processing result sequence of the kth Transformer-based decoding module is obtained Thus, the kth processing result sequence is obtained by the kth Transformer-based decoding module

[0051] The linear classification layer utilizes to make a prediction to obtain the prediction probability of the i-th Hindi text image x i . wherein, represents the probability of the t-th position character in the i-th Hindi text image x i , and T represents the maximum decoding length. During network training, this probability is used to calculate the loss value. During network testing, this probability is used to select the character prediction result at the current step;

[0052] Step 3: Load the pre-trained parameters of the ViTAEv2 base module in the image classification task, and use them to initialize the encoder parameters in the text image recognition network;

[0053] According to the character label g i and the prediction probability P i , a cross-entropy loss function is constructed, and the initialized text image recognition network is trained using the back propagation algorithm until the cross-entropy loss function converges, thereby obtaining the trained text image recognition network;

[0054] Step 4: Use the trained text image recognition network to recognize any input Hindi text image to be recognized, obtain the prediction probability vector of each position character of the image to be recognized, and select the class corresponding to the maximum probability in the prediction probability vector as the text recognition result of the image to be recognized.

[0055] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above-mentioned Hindi text recognition method, and a processor configured to execute the program stored in the memory.

[0056] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program is executed by a processor to perform the steps of the above-mentioned Hindi text recognition method.

Claims

1. A method for Hindi OCR based on enhanced visual transformer network, characterized in that, The method comprises the following steps: Step 1: Constructing an OCR training data set for Hindi text images; Synthesizing Hindi text images with a background image without text and Hindi text content, and taking the Hindi text content as the text label of the corresponding Hindi text image, so as to obtain a text image set and a corresponding label set thereof ; wherein, denotes the i-th Hindi text image, denotes the i-th Hindi text image corresponding text label, denotes the height of the image, denotes the width of the image, is the number of channels; Step 2: Constructing a text image recognition network, including an encoder and a decoder, and used to get the prediction probability of the i-th Hindi text image of the i-th Hindi text image ​ The encoder is composed of R stacked ViTAEv2 basic modules, wherein the rth ViTAEv2 basic encoding module comprises a main branch, a bypass branch and a feedforward encoding layer with a residual connection; The main branch of the rth ViTAEv2 basic encoding module comprises a multi-scale convolution operation layer, a layer normalization operation layer and a multi-head attention operation layer; The bypass branch of the rth ViTAEv2 basic encoding module comprises a convolution operation layer, a batch normalization operation layer and a sigmoid weighted linear unit; when r = 1, the ith printed Hindi text image The text image recognition network is input, and the feature is extracted through the multi-scale convolution operation layer of the main branch of the rth ViTAEv2 basic encoding module in the encoder, and a multi-channel feature map is output And is converted into a feature sequence after shape change After processing by the multi-head attention operation layer, the feature sequence is obtained ; The bypass branch of the rth ViTAEv2 base encoding module in the encoder encodes the ith Indian language text image The processing is performed to obtain a feature map And is converted into a feature sequence after shape change ; The corresponding position elements of and are added to obtain a fused sequence and input into the feedforward encoding layer of the rth ViTAEv2 basic encoding module, and after sequentially passing through the layer normalization operation and the processing of the two fully connected layers with the linear rectification activation function, the rth visual feature sequence is output. when r = 2, 3, …, R, the (r-1)th visual feature sequence is processed in the rth ViTAEv2 basic encoding module of the input encoder, and the rth visual feature sequence is obtained accordingly ; so that the Rth visual feature sequence output by the Rth ViTAEv2 basic encoding module is the visual feature sequence output by the encoder module , , …, , …, ], wherein represents the jth visual feature vector of the ith Hindi text image ; n represents the total number of visual feature vectors; Step 3: Loading the pre-trained parameters of the ViTAEv2 basic module in the image classification task and using them to initialize the parameters of the encoder in the text image recognition network; According to the text label and the prediction probability After the cross-entropy loss function is constructed, the initialized text image recognition network is trained by using a back propagation algorithm until the cross-entropy loss function converges, so that the trained text image recognition network is obtained. Step 4: Using the trained text image recognition network to recognize any input Hindi text image to obtain a prediction probability vector of each position character in the image, and then selecting the class corresponding to the maximum probability in the prediction probability vector as the text recognition result of the image.

2. The Hindi OCR method based on enhanced visual transformer network according to claim 1, wherein, The decoder in step 2 is stacked by a linear classification layer and K identical Transformer basic decoding modules, and any kth Transformer basic decoding module comprises a masked multi-head self-attention layer, a multi-head attention layer between the encoder and the decoder, and a feedforward layer with a residual connection; When k=1, the literal label The embedding representation is carried out to obtain an embedding representation result , and input into the kth mask multi-head attention layer for processing to obtain a kth self-attention processing result sequence ; The visual feature sequence [ , , …, ] is input into the decoder, and after being processed by the multi-head attention layer between the kth encoder-decoder and [ and [ , , …, ], the kth cross-modal processing result sequence is obtained. The kth feedforward layer pair with residual connection The processing result sequence of the kth Transformer basic decoding module after processing ; The processing result sequence output by the k-1th base decoder module when k = 2, 3, …, R The input is processed in the mask multi-head attention layer in the kth Transformer base decoding module, and the output is processed together with , , …, ] through the kth inter-encoder-decoder multi-head attention layer and the feedforward layer with a residual connection, to obtain the processing result sequence of the kth Transformer base decoding module , so as to obtain the kth processing result sequence from the kth Transformer base decoding module ; The linear classification layer utilizes to make a prediction to obtain a predicted probability of the i-th Hindi text image = wherein, denotes a probability of a t-th position character in the i-th Hindi text image and T denotes a maximum decoding length.​ 3. An electronic device comprising a memory and a processor, characterized in that The memory is used to store a program supporting the processor to execute the Hindi text image recognition method in any one of claims 1-2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the Hindi text image recognition method in any one of claims 1-2.

Images