A method for discriminating the language of text images based on deep learning

Abstract

The present invention discloses a method for discriminating the language of natural scene text images based on deep learning. The steps include: 1. Collect text images and establish a language discrimination data set; 2. Construct a language discrimination network for text images; 3. Calculate the loss function corresponding to each input image and train the language discrimination network; 4. Use the trained language discrimination network for text images to discriminate the language of any input image to be discriminated. After training the language discrimination network, the present invention can effectively discriminate the language of given natural scene text images, and has high practical value.

Description

Technical Field

[0001] The present invention relates to related problems in the field of text image language discrimination, and specifically relates to a method for text image language discrimination based on deep learning. Background Art

[0002] At present, there are mainly two methods for language discrimination: the language discrimination method based on traditional algorithms and the language discrimination method based on deep learning. The traditional language discrimination method first extracts various features using some heuristic rules, and then sends the features into a classifier for classification. For traditional methods, the steps of manually designing feature extraction methods are rather cumbersome. Moreover, in the case where the text content in the text image is blurred, curved, or occluded, it is very difficult for traditional language discrimination methods to cope with these challenges. Compared with traditional methods, deep learning-based methods do not require manual feature design. Based on a large amount of data, a deep neural network can learn a feature extraction method that is beneficial to text language discrimination. The discrimination accuracy and robustness of the language discrimination method based on deep learning far exceed those of traditional language discrimination methods. Summary of the Invention

[0003] In order to overcome the deficiencies of the prior art, the present invention proposes a language discrimination network including multiple modules with different functions, aiming to enhance the ability to model image context information and the ability to extract n-gram information during language discrimination, so as to improve the accuracy of text image language discrimination.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] The present invention is characterized in that a method for text picture language discrimination based on deep learning includes the following steps:

[0006] Step 1: Collect text pictures of multiple languages and perform language category annotation to obtain a set of annotated text pictures, denoted as X = [x1, x2,..., x i ,..., x N , and denote the corresponding label set of X as G = [g1, g2,..., g i ,..., g N , where represents the i-th text picture, and g i represents the text label corresponding to the i-th text picture x i , H represents the height of the picture, W represents the width of the picture, 3 is the number of channels; N represents the number of text pictures;

[0007] Step 2: Construct a text image language discrimination network, including: an image feature extraction module, a sequence modeling module, and a language classification module, and used for the i-th text picture xi Perform recognition to obtain the predicted probability y with a length of S for the language category i ;

[0008] Step 3: Use Equation (1) to construct a text image language discrimination network for the i-th text image x i The label smoothing loss function L of i :

[0009]

[0010] In Equation (1), ce i (g) represents the cross-entropy loss calculated from the corresponding text label g of the i-th text image x i and the language prediction probability y of the i-th image x i ; ce i (j) represents the cross-entropy loss calculated from the j-th language category and the language prediction probability y of the i-th image x i ; ε represents the label smoothing factor; i (j) represents the cross-entropy loss calculated from the j-th language category and the language prediction probability y of the i-th image x i ; ε represents the label smoothing factor; i The cross-entropy loss calculated from the j-th language category and the language prediction probability y of the i-th image x, and ε represents the label smoothing factor;

[0011] Step 4: Based on the labeled text image set X, use the backpropagation algorithm to train the text image language discrimination network and calculate the label smoothing loss function to update the network parameters until the label smoothing loss function converges, so as to obtain the trained text image language discrimination network;

[0012] Step 5: Use the trained text image language discrimination network to perform language discrimination on any text image to be discriminated, obtain the predicted probability vector p of the image to be discriminated, and then select the category corresponding to the maximum probability in the predicted probability vector p as the language discrimination result of the image to be discriminated.

[0013] The characteristics of the text image language discrimination method based on deep learning described in the present invention also lie in that: the image feature extraction module in Step 2 includes M cascaded convolutional layers; among them, the first convolutional layer consists of a main branch, and the main branch is composed of a convolutional operation layer, a batch normalization operation layer, and a rectified linear unit ReLU, and each of the remaining N-1 convolutional layers includes K sub-blocks, and each sub-block includes a main branch and a skip connection branch;

[0014] When m = 1, after inputting the i-th text image x i into the language discrimination network, the m-th convolutional layer in the image feature extraction module processes x i to obtain the feature map G i，m ;

[0015] ​When m=2, k=1, the feature map G output by the m-1th convolutional layer i，m-1 Input the mth convolutional layer, the main branch of the kth sub-block is used to map the feature map G i，m-1 Processing is performed and the processing results of the main branch are obtained accordingly At the same time, the tensor G is connected by the skip connection branch i，m-1 After the projection operation, the number of channels of the feature map is changed to obtain the processing result of the jump connection branch. Add element by element and Perform fusion to obtain the processing result G of the kth sub-block i，k，m ;

[0016] When m=2, k=2, 3, ..., K, the processing result G of the k-1th sub-block is i，k-1，m Input the kth sub-block of the mth convolutional layer for processing, and output the processing result G of the kth sub-block i，k，m ; Thus, the K-th sub-block outputs the processing result G i，K，m ;

[0017] When m=3, 4, ..., M, k=1, 2, 3, ..., K, the Kth sub-block of the m-1th convolutional layer is output as the processing result G i，K,m-1 Input into the mth convolutional layer, and after processing by K sub-blocks, the output is the processing result G i，K，m ; Thus, the K-th sub-block of the M-th convolutional layer outputs the processing result G i，K，M , and recorded as the i-th text picture x i Image feature map Among them, h and w represent the height and width of the image feature map, and c represents the number of channels;

[0018] The image feature extraction module is used to extract G i，out Perform shape transformation to get the i-th text image x i The characteristic sequence E i =[e i 1, ..., e i k ,...,e i w ], where e i k is the kth vector of the feature sequence Ei, with dimension h×c;

[0019] The sequence modeling module in step 2 uses two stacked bidirectional long short-term memory network layers to model the feature sequence [e i 1, ..., e i k ,...,e i wProcess, including: taking [e i 1,..., e i k ,..., e i w as the input to the first bidirectional long short-term memory network layer and performing forward and backward processing. After concatenating the forward and backward processing results, the primary feature sequence [a i 1.,..., a i k ,..., a i w output by the first bidirectional long short-term memory network layer is obtained, where a i k represents the k-th primary feature of the i-th text image x i .;

[0020] After the second long short-term memory network layer performs forward, backward processing and concatenation on [a i 1.,..., a i k ,..., a i w , the image feature sequence V i = [v i 1, v i 2,..., v i k ,..., v i w output by the sequence modeling module is obtained; where v i k represents the k-th feature of the i-th text image x i after context information modeling.

[0021] The language classification module in step 2 includes: a multi-scale one-dimensional convolutional layer, a max pooling layer, and a linear classifier with a softmax activation function;

[0022] The language classification module uses the multi-scale one-dimensional convolutional layer and the max pooling layer to process V i to obtain the language category feature vector I i v ;

[0023] Input I i v into the linear classifier with a softmax activation function for processing, so as to obtain the prediction probability y i = {y i 1 ,..., y i j,..., y i S}; wherein, y i j is a value between 0 and 1, representing the probability that the language category of the text image x i belongs to the j-th category prediction.

[0024] An electronic device according to the present invention includes a memory and a processor, characterized in that the memory is used to store a program for supporting the processor to execute any of the text image language discrimination methods, and the processor is configured to execute the program stored in the memory.

[0025] A computer-readable storage medium according to the present invention, characterized in that when the computer program stored on the computer-readable storage medium is run by a processor, it executes the steps of any of the text image language discrimination methods.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0027] 1. The present invention proposes a text image language discrimination method based on deep learning. Through the trained language discrimination network, the language of natural scene text images can be effectively discriminated, and the recognition accuracy is improved.

[0028] 2. The present invention constructs a feature sequence for language discrimination through a convolutional network and a long short-term memory network. The convolutional network can effectively extract image features, and the long short-term memory network strengthens the context modeling, thereby improving the recognition accuracy.

[0029] 3. The present invention further extracts n-gram features for language discrimination through multi-scale one-dimensional convolution to improve the accuracy of language discrimination of text images.

[0030] 4. The present invention uses label smoothing technology to solve the overfitting problem during the training of the language discrimination network, thereby enhancing the generalization ability of the discrimination network. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flowchart of the use of the language discrimination method based on deep learning of the present invention;

[0032] Figure 2 is a network structure diagram of the language discrimination method based on deep learning of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] In this embodiment, as Figure 1 shown, a language discrimination method based on deep learning includes the following steps:

[0034] Step 1: Collect text images in multiple languages and perform language category annotation to obtain a set of annotated text images, denoted as X = [x1, x2,..., x i ,..., x N , and denote the corresponding label set of X as G = [g1, g2,..., g i ,..., g N , where, represents the i-th text image, and g i represents the text label corresponding to the i-th text image x i . H represents the height of the image, W represents the width of the image, and 3 represents the number of channels; N represents the number of text images;

[0035] Step 2: As Figure 2 shown, construct a text image language discrimination network, including: an image feature extraction module, a sequence modeling module, and a language classification module;

[0036] The image feature extraction module is mainly used for extracting image visual features and suppressing background information. The extracted image features are fed into the sequence modeling module for context information modeling to obtain an image feature sequence. The language classification module takes the image feature sequence as input, first extracts key features helpful for language discrimination, and then uses the key features to predict the language category of the text image.

[0037] Step 2.1: The image feature extraction module contains M cascaded convolutional layers; among them, the first convolutional layer consists of a main branch, and the main branch is composed of a convolutional operation layer, a batch normalization operation layer, and a rectified linear unit ReLU. Each of the remaining N - 1 convolutional layers contains K sub-blocks, and each sub-block includes a main branch and a skip connection branch;

[0038] When m = 1, after inputting the i-th text image x i into the language discrimination network, the m-th convolutional layer in the image feature extraction module processes x i to obtain a feature map G i，m ;

[0039] When m = 2 and k = 1, the feature map G i，m-1 output by the m - 1-th convolutional layer is input into the m-th convolutional layer, and the main branch of the k-th sub-block processes the feature map G i，m-1 to obtain the processing result of the main branch At the same time, the skip connection branch performs a projection operation on the tensor G i，m-1 and then changes the number of channels of the feature map to obtain the processing result of the skip connection branch Add [[ID=--48]] and element-wise.Perform fusion to obtain the processing result G of the k-th sub-block i，k，m ;

[0040] When m = 2 and k = 2, 3,..., K, input the processing result G of the (k - 1)-th sub-block i，k-1，m into the k-th sub-block of the m-th convolutional layer for processing, and output the processing result G of the k-th sub-block i，k，m ; Thus, output the processing result G from the K-th sub-block i，K，m ;

[0041] When m = 3, 4,..., M and k = 1, 2, 3,..., K, input the processing result G output from the K-th sub-block of the (m - 1)-th convolutional layer i，K，m-1 into the m-th convolutional layer, and after processing through K sub-blocks, output the processing result G i，K，m ; Thus, output the processing result G from the K-th sub-block of the M-th convolutional layer i，K，M , and denote it as the image feature map where h and w represent the height and width of the image feature map, and c represents the number of channels;

[0042] The image feature extraction module performs a shape transformation on G i，out to obtain the feature sequence E i = [e i 1,..., e i k ,..., e i w , where e i k is the k vectors of the feature sequence E i , with a dimension of h × c, containing the visual features extracted by the convolution operation;

[0043] Step 2.2. The sequence modeling module uses two stacked bidirectional long short-term memory network layers to process the feature sequence [e i 1,..., e i k ,..., e i w , including: inputting [e i 1,..., e i k ,..., e i w into the first bidirectional long short-term memory network layer and performing forward and backward processing. After concatenating the processing results in the two directions, obtain the primary feature sequence [a i 1,,..., a i k ,..., a iw , where a i k represents the k-th primary feature of the i-th text image x i .

[0044] The second long short-term memory network layer takes the output [a i 1,,..., a i k ,..., a i w of the first long short-term memory network layer as input and processes it in the same way. The processing result is the output of the sequence modeling module, denoted as the image feature sequence V i = [v i 1, v i 2,..., v i k ,..., v i w , where v i k represents the k-th feature of the i-th text image x i after context information modeling. The use of the long short-term memory network strengthens the modeling of context-related information of sequence features, and the resulting V i is more conducive to language discrimination.

[0045] Step 2.3, the language classification module, includes a multi-scale one-dimensional convolutional layer, a max pooling layer, and a linear classifier with a softmax activation function;

[0046] Given the output V i = [v i 1, v i 2,..., v i w of the sequence modeling module, the language classification module processes it using multi-scale one-dimensional convolutional operations. The size of the feature map generated by each convolutional kernel varies according to the sentence length and the size of the convolutional kernel. Therefore, the language classification module performs a max pooling operation on the feature maps generated by different convolutional kernels to obtain the language category feature vector I i v ; in this process, key information of different spans for language discrimination is extracted, and these key information are included in I i v .

[0047] Input I i v into the linear classifier with a softmax activation function for processing, so as to obtain the prediction probability y i = {yi 1 ,..., y i j ,..., y i S}; where y i j is a value between 0 and 1, representing the probability that the language category of the text image belongs to the j-th category prediction; during network training, this probability is used to calculate the loss value. During network testing, this probability is used to select the prediction result;

[0048] Step 3: Use Equation (1) to construct the label smoothing loss function L of the text image language discrimination network i :

[0049]

[0050] In Equation (1), ce i (g) represents the cross-entropy loss calculated from the text label g corresponding to the i-th text image x i and the language prediction probability y of the i-th image x i ; ce i (j) represents the cross-entropy loss calculated from the j-th language category and the language prediction probability yi of the i-th image x i , and ε represents the label smoothing factor. Using the label smoothing technique is mainly to solve the overfitting problem during the training of the language discrimination network; i i

[0051]

[0052] Step 4: Based on the labeled text image set X, use the backpropagation algorithm to train the text image language discrimination network and calculate the loss function to update the network parameters until the loss function converges, thereby obtaining the trained text image language discrimination network;

[0053] Step 5: Use the trained text image language discrimination network to perform language discrimination on any text image to be discriminated, obtain the prediction probability vector p of the image to be discriminated, and then select the category corresponding to the maximum probability in the prediction probability vector p as the language discrimination result of the image to be discriminated. In this process, the network does not need to perform backpropagation and can output the language discrimination result at a very fast speed.

[0053] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor to execute the above text image language discrimination method, and the processor is configured to execute the program stored in the memory.

[0054] In this embodiment, a computer-readable storage medium stores a computer program thereon, and when the computer program is run by a processor, it executes the steps of the above-mentioned text image language discrimination method.

Claims

1. A method for discriminating the language of text pictures based on deep learning, characterized in that, Including the following steps: Step 1: Collect text images in multiple languages and perform language category annotation to obtain a set of annotated text images, denoted as , and denote the corresponding label set as , where , represents the i-th text image, represents the i-th text image corresponding text label, represents the height of the image, represents the width of the figure, is the number of channels; N represents the number of text images; Step 2: Construct a text image language discrimination network, including: an image feature extraction module, a sequence modeling module, and a language classification module, and use it to process the i-th text image for recognition to obtain a prediction probability with a length of S, where S is the number of language categories ; The image feature extraction module includes M cascaded convolutional layers; among them, the first convolutional layer consists of a main branch, and the main branch is composed of a convolutional operation layer, a batch normalization operation layer, and a rectified linear unit ReLU. Each of the remaining N-1 convolutional layers contains K sub-blocks, and each sub-block includes a main branch and a skip connection branch; When m = 1, the i-th text image is input into the language discrimination network, and is processed by the m-th convolutional layer in the image feature extraction module to obtain a feature map ; When m = 2 and k = 1, the feature map output by the (m - 1)-th convolutional layer is input into the m-th convolutional layer, and the main branch of the k-th sub-block processes the feature map to obtain the processing result of the main branch . At the same time, the skip connection branch projects the tensor and then changes the number of channels of the feature map to obtain the processing result of the skip connection branch . The results of and are fused element-wise to obtain the processing result of the k-th sub-block ; When m = 2 and k = 2, 3, …, K, the processing result of the (k - 1)-th sub-block is input into the k-th sub-block of the m-th convolutional layer for processing, and the processing result of the k-th sub-block is output ; thus, the processing result is output from the K-th sub-block ; When \(m = 3, 4, \ldots, M\) and \(k = 1, 2, 3, \ldots, K\), the processing result of the output of the \(K\)-th sub-block of the \((m - 1)\)-th convolutional layer is input into the \(m\)-th convolutional layer and, after being processed by \(K\) sub-blocks, the processing result is output ; thus, the processing result of the output of the \(K\)-th sub-block of the \(M\)-th convolutional layer is obtained and denoted as the image feature map of the \(i\)-th text image , where , , represent the height and width of the image feature map, and represents the number of channels; The image feature extraction module performs a shape transformation on the i-th text picture to obtain the feature sequence E i = [e i 1, ..., e i k , ..., e i w , where e i k is the k-th vector of the feature sequence E i , with the dimension of ; Step 3: Use Equation (1) to construct a text image language discrimination network for the i-th text image with a label smoothing loss function : (1) In formula (1), represents the cross-entropy loss calculated from the text label corresponding to the i-th text image and the language prediction probability of the i-th image ; represents the cross-entropy loss calculated from the language prediction probability of the j-th language category and the i-th image ; represents the label smoothing factor; Step 4: Based on the set of text images after annotation , use the backpropagation algorithm to train the text image language discrimination network, and calculate the label smoothing loss function to update the network parameters until the label smoothing loss function converges, so as to obtain the trained text image language discrimination network; Step 5: Use the trained text image language discrimination network to perform language discrimination on any text image to be discriminated, and obtain the predicted probability vector of the image to be discriminated , and then select the category corresponding to the maximum probability in the predicted probability vector as the language discrimination result of the image to be discriminated.

2. The method for discriminating the language of a text picture based on deep learning according to claim 1, wherein: The sequence modeling module in step 2 uses two stacked bidirectional long short-term memory network layers to process the feature sequence [e i 1, ...,e i k ,...,e i w , including: inputting [e i 1, ...,e i k , ...,e i w into the first bidirectional long short-term memory network layer for forward and backward processing, and after splicing the forward and backward processing results, obtaining the primary feature sequence [a i 1, , ...,a i k , ...,a i w output by the first bidirectional long short-term memory network layer, where a i k represents the k-th primary feature of the i-th text picture . The second long short-term memory network layer processes and concatenates [a i 1, , ..., a i k , ..., a i w in the forward and reverse directions and then obtains the image feature sequence V output by the sequence modeling module i = [v i 1, v i 2, ..., v i k , ..., v i w ; where v i k represents the k-th feature of the i-th text image after context information modeling.

3. The method for discriminating the language of text pictures based on deep learning according to claim 2, wherein: The language classification module in step 2 includes: a multi-scale one-dimensional convolutional layer, a max pooling layer, and a linear classifier with a softmax activation function; The language classification module uses a multi-scale one-dimensional convolutional layer and a max pooling layer to process V i to obtain a language category feature vector I i v ; Input I i v is processed in a linear classifier with a softmax activation function to obtain prediction probabilities of length S, where S is the number of language categories ={y i 1 , …,y i j , …, y i S}; where y i j is a value between 0 and 1, representing the probability that the language category of the text image belongs to the j-th category prediction.

4. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor to execute any one of the text image language discrimination methods in claims 1-3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor, it executes the steps of any one of the text image language discrimination methods in claims 1-3.

Images