A scene text recognition method based on compression and excitation neural network
By introducing the SECGNN model of compression and excitation convolutional neural network and bidirectional GRU network into the CRNN model, the problems of insufficient feature extraction and excessive parameters are solved, more efficient scene text recognition is achieved, recognition accuracy is improved and training costs are reduced.
Patent Information
- Application Number
- CN202211610577.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-12
- Publication Date
- 2025-09-16
- Estimated Expiration
- 2042-12-12
AI Technical Summary
Existing CRNN models suffer from insufficient feature extraction and excessive model parameters in scene text recognition, resulting in poor recognition results and high training costs.
The SECGNN model based on the compression and excitation convolutional neural network SECNN and the bidirectional GRU network is adopted. By adding compression and excitation modules to the convolutional neural network, the model structure is optimized, and the bidirectional GRU network is used in the temporal information learning layer to reduce the model parameters.
It improves the accuracy of scene text recognition, reduces training time cost, and improves the applicability and recognition efficiency of the model.
Smart Images
Figure CN116110055B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of scene text recognition, and in particular relates to a scene text recognition method based on a compression and excitation neural network. Background Art
[0002] With the development of the times, computer vision technology is increasingly being used in everyday life. Images are a crucial carrier of information in the internet age, and text recognition in images has long been a hot topic in the field of computer vision. For ordinary scanned text images, recognition is relatively easy due to their simple image format, clear text information, and regular text features. However, text recognition in natural scenes is currently a research focus in the field. Although many models have been proposed for scene text recognition, recognition performance remains suboptimal due to factors such as the complexity of scene images, environmental uncertainty, and the complexity of text. Currently, academic research in the field of scene text recognition remains highly valued. Scene text recognition holds enormous commercial value due to its wide range of applications. Image text recognition has long been a hot topic in academic research, and deep learning-based text recognition technology has made significant progress in many areas, but significant room for improvement remains. Many current deep learning models fail to fully extract text feature information during the feature extraction phase, and the excessive number of model parameters leads to low training efficiency and poor applicability to electronic devices with limited memory.
[0003] The CRNN model is one of the most popular text recognition models. CRNN classifies image features into segments, making it capable of recognizing relatively long text sequences. The CRNN process begins with feature extraction using a CNN network, then passes the extracted features through a BLSTM network to learn temporal information, and finally translates them through a CTC network to produce the final result. This CRNN approach can mitigate the issue of variable text length and effectively improve text recognition accuracy by leveraging temporal information. While the CRNN model offers numerous advantages for text recognition, it still faces the following challenges:
[0004] (1) When extracting features, the simple CNN network does not take into account the different importance of features in each channel, resulting in the extracted feature map not being able to express the image feature information well.
[0005] (2) In the RNN timing module, CRNN uses a bidirectional long short-term memory network (BLSTM), which results in too many model training parameters, a complex model structure, and excessive time costs for training and recognition.
[0006] In response to the above two problems of the CRNN model, the present invention proposes an efficient text recognition model SECGNN based on the compression and excitation convolutional neural network SECNN and the bidirectional GRU network. To address the problems of CRNN, SECGNN adds a compression and excitation module to the convolutional neural network and uses a bidirectional GRU network in the temporal information learning layer, effectively optimizing the CRNN model structure, and has good recognition effect, low training cost and strong applicability. Summary of the Invention
[0007] The present invention addresses the shortcomings of existing technologies by providing a method for scene text recognition based on a compression and excitation neural network. This method effectively improves scene text recognition accuracy and reduces model training time. This method further enhances the efficiency of existing scene text recognition algorithms, providing new insights for subsequent research and engineering applications of complex scene text recognition algorithms.
[0008] In a first aspect, the present invention proposes a scene text recognition method based on a compression and excitation neural network, comprising the following steps:
[0009] Step 1: Obtain a natural scene image containing text and preprocess it to obtain a text block image;
[0010] Step 2: Extract features of text block images through feature extraction layer;
[0011] The feature extraction module includes a first compression and excitation network SECNN1, a first maximum pooling layer 1, a second compression and excitation network SECNN2, a second maximum pooling layer 2, a third compression and excitation network SECNN3, a fourth compression and excitation network SECNN4, a third maximum pooling layer 3, a fifth compression and excitation network SECNN5, a sixth compression and excitation network SECNN6, a fourth maximum pooling layer 4 and a seventh compression and excitation network SECNN7, which are cascaded in sequence; its input is a picture of size (H, W, 1) and its output is a (1, W / 4, 512) feature map;
[0012] Each compression and excitation network includes a convolution layer, a Global Pooling layer, two fully connected layers FC, a Sigmoid function, and a fusion layer; the convolution layer receives the preprocessed text block image (H0, W0, C0) and converts it into (H, W, C) after convolution; the Global Pooling layer receives the feature map (H, W, C) output by the convolution layer, trains each feature channel as a number of global information of the current feature channel, and each channel corresponds to a number, so a feature map of size (1, 1, C) is obtained; the two fully connected layers FC convert the feature map from (1, 1, C) to (1, 1, C / r), and then to (1, 1, C), where r is a constant; the Sigmoid function is used to calculate the channel weight of the (1, 1, C) feature map; the fusion layer multiplies the channel weight by the convolved feature map (H, W, C) to obtain the compressed and excited feature map;
[0013] Step 3: The feature map extracted by the feature extraction module is transformed into multiple feature sequences arranged in sequence; specifically:
[0014] The (1, W / 4, 512) feature map output by the feature extraction layer is transformed into W / 4 feature sequences through a feature transformation, and each feature sequence is 512 in length, 1 in width and 1 in height, and each feature vector corresponds to the original Figure 1 receptive fields and arranged in order;
[0015] Step 4: Build a time series information learning layer to extract the time series information of the feature graph; specifically:
[0016] The temporal information learning layer includes a double-layer bidirectional gated recurrent neural network GRU and a Softmax activation function;
[0017] The double-layer bidirectional gated recurrent neural network GRU is composed of two GRU networks that perform forward propagation and backward propagation respectively to form a bidirectional GRU network;
[0018] The input of the two-layer bidirectional gated recurrent neural network GRU is T feature sequences, and the length of each feature sequence is m, then the input is as shown in formula (1):
[0019] e=(e 1 ,e 2 ,…,e T ) (1)
[0020] Each of the e t The length is m, it contains m-dimensional information, e t As shown in formula (2):
[0021]
[0022] A Softmax activation function is connected to the output end of the double-layer bidirectional gated recurrent neural network GRU, and the output x is a probability matrix, whose expression is defined as shown in formula (3):
[0023] x=(x 1 ,x 2 ,…,x t ,…,x T ) (3)
[0024] Each column of x contains n values, x t As shown in formula (4):
[0025]
[0026] n is the number of categories of classification characters, each The probability that this input feature is the character at position i in the classification character table is Therefore Subject to the probability assumption: Where k ranges from 1 to n;
[0027] Step 5: Use the Connectionist Temporal Classification (CTC) network to transcribe and translate to obtain the predicted text. Specifically, this is done by finding the label sequence with the highest probability combination from the prediction of each frame of the probability matrix output by the temporal information learning layer, calculating the probability of each path, and taking the path with the highest probability as the inference result.
[0028] Preferably, the preprocessing in step 1 is to perform text detection on a natural scene picture containing text using a text detection algorithm, and obtain an initial recognition image that can be used for text recognition; the size of the initial recognition image is proportionally scaled to (H, W, C), where H is the height, W is the width, C is the number of channels, and H is an integer multiple of 16; and then the (H, W, C) image is grayscale transformed to obtain an image of size (H, W, 1) after the transformation;
[0029] Preferably, the number of convolution kernels of the first to seventh compression and excitation networks SECNN in step 2 is 64, 128, 256, 256, 512, 512, 512 respectively;
[0030] Preferably, in step 5, during the training phase, the output x of the temporal information learning layer, through the CTC output sequence l, is expressed as follows:
[0031]
[0032] where β -1 (l) represents the mapping function that maps the path π to the sequence l;
[0033] In the input feature sequence x, the probability of the path being π is p(π|x), and the specific calculation formula is shown in (6):
[0034]
[0035] Among them, T represents the total time, t represents a certain moment, represents the probability of the character in path π appearing at time t;
[0036] In the prediction stage, the probability of each sequence is calculated, and the sequence with the highest probability among all possible sequences D is taken as the inference result, i.e. * =arg maxp(l|x), where l∈D.
[0037] In a second aspect, the present invention proposes a scene text recognition system, comprising:
[0038] The text preprocessing module is used to obtain natural scene images containing text and preprocess them to obtain text block images;
[0039] A feature extraction module is used to extract features of text block images through a feature extraction layer;
[0040] A feature sequence extraction module is used to transform the feature map extracted by the feature extraction module into multiple feature sequences arranged in sequence;
[0041] A time series information learning module is used to extract the feature sequence time series information output by the feature sequence extraction module using the time series information learning layer;
[0042] The text recognition module is used to perform transcription and translation using the connection time classification network (CTC) to obtain predicted text.
[0043] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described above.
[0044] In a fourth aspect, the present invention proposes a computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described is implemented.
[0045] Beneficial effects of the present invention:
[0046] To address the problems of the CRNN model, this paper proposes an efficient text recognition model, SECGNN, based on a compression and excitation convolutional neural network (SECNN) and a bidirectional GRU network. SECGNN incorporates a compression and excitation module into the convolutional neural network, which can assign weights to each channel and improve the model's sensitivity to channel information, thereby more fully extracting image feature information. The use of a bidirectional GRU network in the temporal information learning layer reduces model parameters and improves training efficiency. The SECGNN model has good recognition effect, low training cost, and strong applicability, providing new ideas for subsequent research on complex scene text recognition algorithms and engineering applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 It is a flow chart of the method of the present invention.
[0048] Figure 2 It is the overall framework diagram of the model in the present invention.
[0049] Figure 3 This is the flow chart of feature extraction in the present invention
[0050] Figure 4 This is a structural diagram of the compression and excitation convolutional neural network in the present invention.
[0051] Figure 5 It is an explanatory diagram of feature transformation in the present invention.
[0052] Figure 6 This is a framework diagram of the temporal information learning process in the present invention, in which a two-layer bidirectional GRU network stack is used. DETAILED DESCRIPTION
[0053] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings.
[0054] like Figure 1 Flowchart and Figure 2 As shown in the model framework diagram, the present invention includes the following steps:
[0055] Step 1: preprocessing the natural scene image containing text to obtain a text block image;
[0056] Step 1 is as follows:
[0057] The preprocessing is to perform text detection on a natural scene image containing text using a text detection algorithm, and obtain an initial recognition image that can be used for text recognition; scale the initial recognition image size to (H, W, C), where H is the height, W is the width, C is the number of channels, and H is an integer multiple of 16; and then perform grayscale conversion on the (H, W, C) image to obtain an image of size (H, W, 1).
[0058] Step 2: Extract features of text block images through feature extraction layer;
[0059] like Figure 3 As shown, the feature extraction module includes a first compression and excitation network SECNN1, a first maximum pooling layer 1, a second compression and excitation network SECNN2, a second maximum pooling layer 2, a third compression and excitation network SECNN3, a fourth compression and excitation network SECNN4, a third maximum pooling layer 3, a fifth compression and excitation network SECNN5, a sixth compression and excitation network SECNN6, a fourth maximum pooling layer 4 and a seventh compression and excitation network SECNN7, which are cascaded in sequence; its input is a picture of size (H, W, 1), and its output is a (1, W / 4,512) feature map;
[0060] like Figure 4 As shown, each compression and excitation network includes a convolution layer, a Global Pooling layer, two fully connected layers FC, a Sigmoid function, and a fusion layer; the convolution layer receives the preprocessed text block image (H0, W0, C0), which is converted into (H, W, C) after convolution; the Global Pooling layer receives the feature map (H, W, C) output by the convolution layer, trains each feature channel as a number of global information of the current feature channel, and each channel corresponds to a number, so a feature map of size (1, 1, C) is obtained; the two fully connected layers FC convert the feature map from (1, 1, C) to (1, 1, C / r), and then to (1, 1, C), where r is a constant; the Sigmoid function is used to calculate the channel weight of the (1, 1, C) feature map; the fusion layer multiplies the channel weight by the convolved feature map (H, W, C) to obtain the compressed and excited feature map;
[0061] The number of convolution kernels of the first to seventh compression and excitation networks SECNN are 64, 128, 256, 256, 512, 512, and 512 respectively.
[0062] Step 3: The feature map extracted by the feature extraction module is transformed into multiple feature sequences.
[0063] Step 3 process is as follows Figure 5 As shown, the feature extraction layer output is (1, W / 4, 512), which is transformed into W / 4 feature sequences after a feature transformation. Each feature sequence is 512 long, 1 in width and 1 in height. Each feature vector corresponds to the original Figure 1 receptive fields and arranged in order.
[0064] Step 4: Send the feature sequence to the time series information learning layer. Figure 6The temporal information learning layer uses a two-layer bidirectional gated recurrent neural network to extract the temporal information of the feature graph.
[0065] The temporal information learning layer uses a deep bidirectional GRU network structure. Its input is W / 4 feature sequences. Each feature sequence has a size of (1, 1, 512), a width and height of 1, and a dimension of 512. Each feature sequence corresponds to a vertical receptive field in the original image. In the bidirectional GRU network, a feature sequence is input at each time step. There are a total of W / 4 time steps. The bidirectional GRU network structure is as follows: Figure 6 As shown in the figure, the two GRU networks are combined into a bidirectional GRU network, one for forward propagation and one for backward propagation, and two layers of bidirectional GRU networks are stacked. This structure can obtain deeper semantic information.
[0066] Assume that the input of the temporal information learning layer is T feature sequences, and the length of each feature sequence is m, then the input is as shown in formula (1):
[0067] e=(e 1 ,e 2 ,…,e T ) (1)
[0068] Each of the e t The length is m, it contains m-dimensional information, e t As shown in formula (2):
[0069]
[0070] After the double-layer bidirectional gated recurrent neural network GRU, its output is connected to a Softmax activation function, then the output x is a probability matrix, and its expression is defined as shown in formula (3):
[0071] x=(x 1 ,x 2 ,…,x t ,…,x T ) (3)
[0072] Each column of x contains n values, x t As shown in formula (4):
[0073]
[0074] n is the number of categories of classification characters, each The probability that this input feature is the character at position i in the classification character table is Therefore Subject to the probability assumption: Where k ranges from 1 to n. In this way, the probability distribution of each feature sequence to the classification character is obtained, and this is used as the input of the transcription module to obtain the final classification result.
[0075] Step 5: Use the connection time classification network (CTC) to transcribe and translate to obtain the predicted text.
[0076] The transcription layer converts the probability matrix obtained from the temporal information learning layer into a label sequence. Mathematically, transcription involves finding the label sequence with the highest probability combination from the per-frame predictions of the temporal information learning layer. The transcription layer is trained using the CTC component, which can handle the alignment of sequences of varying lengths and can be trained end-to-end using the CTC loss function.
[0077] During the training phase, the output x of the temporal information learning layer has the probability of being l through the CTC output sequence as shown in formula (5):
[0078]
[0079] where β -1 (l) represents the mapping function that maps the sequence π to the sequence l.
[0080] The beta function rule is defined as follows:
[0081] (1) If there is no blank character between the characters, they are continuous characters. If the continuous characters are the same characters, the characters are repeated and only one can be retained.
[0082] (2) If there is a blank character between the characters, they are non-continuous characters. Regardless of whether the characters are the same or not, they are non-repeated characters and are retained.
[0083] (3) Finally, delete all blank characters.
[0084] For the β function, examples are shown in formulas (6) to (10), where T is assumed to be 14:
[0085] β(π1)=β(—moot-he—r-)=mother (6)
[0086] β(π2)=β(mot--hh-ee-r-)=mother (7)
[0087] β(π3)=β(-mmott-h-eerr-)=mother (8)
[0088] β(π4)=β(m-ooo-th-e—r-)=mother (9)
[0089] β(π5)=β(mo-ott-hh-er--)=moother (10)
[0090] The above example illustrates the three rules of the beta function, which also shows that the beta function is a many-to-one rule. The beta function is the rule for character mapping in the CTC module.
[0091] In the input feature sequence x, the probability of the path being π is p(π|x), and the specific calculation formula (11) is shown as follows:
[0092]
[0093] Among them, T represents the total time, t represents a certain moment, Represents the probability of the character in path π appearing at time t.
[0094] In the prediction stage, the probability of each sequence is calculated, and the sequence with the highest probability among all possible sequences D is taken as the inference result, that is, l*=argmaxp(l|x), where l∈D.
Claims
1. A scene text recognition method based on compression and excitation neural network, characterized by The method comprises the following steps: Step 1: Obtain a natural scene image containing text and preprocess it to obtain a text block image; Step 2: Extract features of text block images through feature extraction layer; The feature extraction module includes a first compression and excitation network SECNN1, a first maximum pooling layer 1, a second compression and excitation network SECNN2, a second maximum pooling layer 2, a third compression and excitation network SECNN3, a fourth compression and excitation network SECNN4, a third maximum pooling layer 3, a fifth compression and excitation network SECNN5, a sixth compression and excitation network SECNN6, a fourth maximum pooling layer 4 and a seventh compression and excitation network SECNN7, which are cascaded in sequence; its input is a picture of size (H, W, 1) and its output is a (1, W / 4, 512) feature map; Each compression and excitation network includes a convolution layer, a Global Pooling layer, two fully connected layers FC, a Sigmoid function, and a fusion layer; the convolution layer receives the preprocessed text block image (H0, W0, C0) and converts it into (H, W, C) after convolution; the Global Pooling layer receives the feature map (H, W, C) output by the convolution layer, trains each feature channel as a number of global information of the current feature channel, and each channel corresponds to a number, so a feature map of size (1, 1, C) is obtained; the two fully connected layers FC convert the feature map from (1, 1, C) to (1, 1, C / r), and then to (1, 1, C), where r is a constant; the Sigmoid function is used to calculate the channel weight of the (1, 1, C) feature map; the fusion layer multiplies the channel weight by the convolved feature map (H, W, C) to obtain the compressed and excited feature map; Step 3: The feature map extracted by the feature extraction module is transformed into multiple feature sequences arranged in sequence; specifically: The (1, W / 4, 512) feature map output by the feature extraction layer is transformed into W / 4 feature sequences through a feature transformation. Each feature sequence has a length of 512 and a width and height of 1. Each feature vector corresponds to a receptive field of the original image and is arranged in order. Step 4: Build a time series information learning layer to extract the time series information of the feature graph; specifically: The temporal information learning layer includes a double-layer bidirectional gated recurrent neural network GRU and a Softmax activation function; The double-layer bidirectional gated recurrent neural network GRU is composed of two GRU networks that perform forward propagation and backward propagation respectively to form a bidirectional GRU network; The input of the two-layer bidirectional gated recurrent neural network GRU is T feature sequences, and the length of each feature sequence is m, then the input is as shown in formula (1): and=(and 1 ,And 2 ,…,And T ) (1) Each of the e t The length is m, it contains m-dimensional information, e t As shown in formula (2): A Softmax activation function is connected to the output end of the double-layer bidirectional gated recurrent neural network GRU, and the output x is a probability matrix, whose expression is defined as shown in formula (3): x=(x 1 ,x 2 ,…,x t ,…,x T ) (3) Each column of x contains n values, x t As shown in formula (4): n is the number of categories of classification characters, each The probability that this input feature is the character at position i in the classification character table is Therefore Subject to the probability assumption: Where k ranges from 1 to n; Step 5: Use the connection time classification network (CTC) to perform transcription and translation to obtain the predicted text. Specifically, this is done by finding the label sequence with the highest probability combination from the prediction of each frame of the probability matrix output by the temporal information learning layer, calculating the probability of each sequence, and taking the sequence with the highest probability as the inference result.
2. The method according to claim 1, characterized in that The preprocessing described in step 1 is to perform text detection on natural scene pictures containing text through a text detection algorithm, and obtain an initial recognition image that can be used for text recognition; the size of the initial recognition image is proportionally scaled to (H, W, C), where H is the height, W is the width, C is the number of channels, and H is an integer multiple of 16; then the (H, W, C) picture is grayscale transformed to obtain a picture of size (H, W, 1) after the transformation.
3. The method according to claim 1, characterized in that In step 2, the number of convolution kernels of the first to seventh compression and excitation networks SECNN are 64, 128, 256, 256, 512, 512, and 512 respectively.
4. The method according to claim 1, characterized in that In step 5, during the training phase, the output x of the temporal information learning layer, through the CTC output sequence l, is expressed as follows: where β -1 (l) represents the mapping function that maps the path π to the sequence l; In the input feature sequence x, the probability of the path being π is p(π|x), and the specific calculation formula is shown in (6): Among them, T represents the total time, t represents a certain moment, represents the probability of the character in path π appearing at time t; In the prediction stage, the probability of each sequence is calculated, and the sequence with the highest probability among all possible sequences D is taken as the inference result, i.e. * =arg maxp(l|x), where l∈D.
5. A scene text recognition system implementing the method according to any one of claims 1 to 4, characterized in that include: The text preprocessing module is used to obtain natural scene images containing text and preprocess them to obtain text block images; A feature extraction module is used to extract features of text block images through a feature extraction layer; A feature sequence extraction module is used to transform the feature map extracted by the feature extraction module into multiple feature sequences arranged in sequence; A time series information learning module is used to extract the feature sequence time series information output by the feature sequence extraction module using the time series information learning layer; The text recognition module is used to perform transcription and translation using the connection time classification network (CTC) to obtain predicted text.
6. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 5.
7. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 5 is implemented.