A natural scene text recognition method based on representation batch normalization

Abstract

The present invention discloses a natural scene text recognition method based on representation batch normalization, comprising: using a spatial transformation network to regularize a text image into a horizontal text image; introducing representation normalization into an existing network ResNet-50 to construct a feature extraction network; using the feature extraction network to extract the horizontal text image to obtain a plurality of visual feature maps of different resolutions to form a feature pyramid F r ; Through the feature enhancement network, the feature pyramid F r Perform feature extraction, and fuse the extracted feature maps of different sizes to obtain feature maps of the same size; use the visual-semantic joint network to extract the semantic information of the feature map, fuse the extracted semantic information with the visual features to obtain a new feature space; use the decoder to predict the feature space to obtain the final recognition result. The present invention can improve the accuracy of scene text recognition and enhance the generalization of the model.

Description

Technical Field

[0001] The present invention belongs to the field of recognition technology, and in particular relates to a natural scene text recognition method based on representation batch normalization. Background Art

[0002] Scene text recognition has a wide range of applications in smart city construction and has attracted the interest of many researchers. Although many deep learning-based recognition methods have been proposed, most of the natural scene text detection and recognition methods based on deep learning far exceed traditional methods such as feature extraction in terms of positioning accuracy and recognition accuracy, but there is still a lot of room for improvement in positioning accuracy and recognition accuracy. In recent years, several research teams have proposed to use text correction models to correct the original text images into images with regular text. However, the corrected images are prone to introduce new noise, which will interfere with the accuracy of text recognition. In addition, some research teams have proposed introducing a pluggable super-resolution module in the recognition model to process low-resolution images, but adding a super-resolution module will cause the model training time to be too long. Summary of the invention

[0003] Purpose of the invention: To solve the above problems, the present invention proposes a natural scene text recognition method based on representation batch normalization.

[0004] Technical solution: To achieve the purpose of the present invention, the technical solution adopted by the present invention is: a natural scene text recognition method based on representation batch normalization, the method specifically comprises the following steps:

[0005] Step 1: Regularize the natural scene text image to be recognized into a horizontal text image using a spatial transformer network;

[0006] Step 2: introduce representation normalization into the existing network ResNet-50 to construct a feature extraction network; use the feature extraction network to extract the horizontal text image to obtain a number of visual feature maps with different resolutions; the visual feature maps with different resolutions form a feature pyramid F r ;

[0007] Step 3: Enhance the feature pyramid F through the feature enhancement network r Perform up-scale enhancement and down-scale enhancement, and combine with the feature pyramid F described in step 2 r The elements are connected to obtain the final feature pyramid F e , for the feature pyramid F e Each feature map in is upsampled to obtain a feature map F of the same size f ;

[0008] Step 4: Use the visual-semantic joint network to extract the feature map F described in step 3 f The extracted semantic information is combined with the feature map to obtain a new feature space.

[0009] Step 5: Use the decoder to predict the feature space described in step 4 to obtain the final recognition result.

[0010] Furthermore, the space transformation network in step 1 includes a positioning network, a grid generator and a sampler; the method in step 1 is specifically as follows:

[0011] Step 1.1, resize the natural scene text image to be recognized to 100 pixels × 32 pixels;

[0012] Step 1.2, predicting a set of control points on the resized natural scene text image to be recognized through the localization network;

[0013] Step 1.3, the grid generator calculates thin plate spline variation parameters using the control points, and generates a sampling grid on the resized natural scene text image to be recognized;

[0014] Step 1.4, inputting the sampling grid and the resized natural scene text image to be recognized into the sampler at the same time, and obtaining a regularized horizontal text image by sampling on the sampling grid.

[0015] Furthermore, the method of step 2 is as follows:

[0016] Step 2.1, adding representation normalization after each convolution layer of the existing network ResNet-50 to obtain the feature extraction network;

[0017] Step 2.2, taking the horizontal text image as the input of the feature extraction network, and outputting four visual feature maps with different resolutions, respectively, wherein the resolutions of the visual feature maps are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input horizontal text image resolution, respectively;

[0018] Step 2.3: Use a 1×1 convolutional layer to convert the number of channels of the visual feature map in step 2.2 into 128 to form a feature pyramid F. r .

[0019] Furthermore, the representation normalization described in step 2 is specifically expressed as follows:

[0020] Given a feature map X∈R A×B×H×W , where A, B, H and W represent the batch size, number of channels, height and width of the feature map respectively;

[0021] First, perform a calibration operation on the feature map X to obtain the calibrated feature map X cm , which is expressed as follows:

[0022] X cm =X+ω m .K m

[0023] In the formula, ω m Represents the weight value, K m Represents the statistics of feature map X;

[0024] Then, for the feature map X cm Perform the centralization operation to obtain the centralized feature map X m , which is expressed as follows:

[0025] X m =X cm -E(X cm )

[0026] In the formula, E(X cm ) represents the feature map X cm The mean of is expressed as follows:

[0027] E(X cm )←mE(X cm )+(1-m)μ c

[0028] In the formula, m represents the cumulative addition, ← represents the function mapping, μ c Represents the mean of each batch during training, expressed as follows;

[0029]

[0030] In the formula, h represents the height value of the feature map X, and w represents the width value of the feature map X;

[0031] Next, for the feature map X m Perform the scaling operation to obtain the scaled feature map X s , which is expressed as follows:

[0032]

[0033] In the formula, ∈ is used to avoid zero variance, Var() represents the variance function, and the expression is as follows;

[0034]

[0035] In the formula, Represents the variance of each batch during training, expressed as follows:

[0036]

[0037] Secondly, for the feature map X s Scaling correction feature strength is performed to obtain the corrected feature map X cs , which is expressed as follows:

[0038] X cs =X s ·R(ω v ·K s +ω b )

[0039] In the formula, ω v and ω b represents the trainable parameters, K s Represents the feature map X s Statistics of ; R() represents the correction function;

[0040] Finally, the corrected feature map X cs Perform affine transformation to obtain the feature Y after radial transformation, which is expressed as follows:

[0041] Y=X cs γ+β

[0042] Where γ is the scaling parameter and β is the translation parameter.

[0043] Furthermore, the method of step 3 is as follows:

[0044] Step 3.1: transform the feature pyramid F described in step 2 r Input to the cascaded upscale convolution, perform upscale enhancement with steps of 32, 16, 8 and 4 respectively, and output the upscale enhanced feature pyramid;

[0045] Step 3.2, the feature pyramid after upscale enhancement in step 3.1 is subjected to cascaded downscale convolution, and downscale enhancement is performed with a step size of 4, 8, 16 and 32 in sequence, and the feature pyramid after downscale enhancement is output;

[0046] Step 3.3: combine the downscale enhanced feature pyramid with the feature pyramid F in step 2 r The elements are connected to obtain the final feature pyramid F e ;

[0047] Step 3.4: The feature pyramid F described in step 3.3 e All feature maps in the image are upsampled to 1 / 4 of the original image to obtain feature map F f , the feature map F f The size of is H / 4×W / 4×512, where H is the height of the feature map and W is the width of the feature map.

[0048] Furthermore, the visual-semantic joint network is composed of a two-layer BiLSTM network; the method of step 4 is specifically as follows:

[0049] First, the feature map F described in step 3 is transformed into f Extracted as context feature sequence V;

[0050] Secondly, the feature map F f and the context feature sequence V to form a new feature space C = (F f , V).

[0051] Furthermore, the method of step 5 is as follows:

[0052] The output y predicted by the decoder at time t t , which is expressed as follows:

[0053] y t =softmax(W y s t +b y )

[0054] In the formula, softmax is the activation function, W y and b y is a trainable parameter, s t is the hidden state of LSTM at time t, and its expression is as follows:

[0055] s t =LSTM(y t-1 , g t ,s t-1 )

[0056] Among them, y t-1 is the output predicted by the decoder at time t-1, s t-1 is the hidden state of LSTM at time t-1, g t is the glimpse vector, which is composed of the feature space C = (c1, c2, ..., c j , …, c N ) is composed of the weighted sum of the elements of each column, that is,

[0057]

[0058] Among them, c j represents the j-th column element value of feature space C, N represents the total number of columns in feature space C, α t，j is the attention weight, expressed as follows:

[0059]

[0060] Among them, e t，j =v T tanh(Fs t-1 +Gc j +b), v, F, G and b are all trainable parameters, the superscript T stands for transpose, and tanh() stands for the hyperbolic tangent function.

[0061] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0062] The method for natural scene text recognition based on representation batch normalization of the present invention improves the performance of the recognizer through two aspects: feature representation and feature enhancement. In terms of feature representation, representation normalization is used to reduce the model's dependence on training data and solve the differences in feature representation between different instances. At the same time, in order to further improve the quality of features, a feature enhancement module is used in combination with feature maps of multiple resolutions to include refined feature information. The method of the present invention can improve the accuracy of scene text recognition and enhance the generalization of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a flow chart of a natural scene text recognition method based on representation batch normalization under an embodiment;

[0064] Figure 2 is a schematic diagram of a convolutional block model of a feature extraction network described in an embodiment;

[0065] Figure 3 It is a schematic diagram of the structure of the feature enhancement network described in an embodiment. DETAILED DESCRIPTION

[0066] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments. The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and the features in the embodiments can be combined with each other without conflict.

[0067] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0068] The present invention discloses a natural scene text recognition method based on representation batch normalization, wherein the natural scene text recognition network structure comprises: a space transformation network, a feature extraction network, a feature enhancement network, a visual-semantic joint sequence modeling, and a decoder; Figure 1 , specifically including the following steps:

[0069] (1) Use the spatial transformation network to correct the shape of the text region and regularize the irregular natural scene text image into a horizontal text image;

[0070] (2) Introducing representation normalization into the backbone network ResNet-50 to construct a feature extraction network with generalization capabilities, which can extract stable feature distributions in different data sets; using the feature extraction network to extract visual feature maps of different resolutions to form a feature pyramid;

[0071] (3) The feature enhancement module uses feature maps to extract deeper and more expressive feature maps, and fuses feature maps of different sizes into feature maps of the same size (H / 4×W / 4×512);

[0072] (4) Visual-semantic joint modeling extracts the semantic information of the image and combines the extracted semantic information with the visual features to obtain a new feature fusion.

[0073] (5) The decoder obtains the final recognition result by predicting the feature fusion.

[0074] Furthermore, the specific process of step (1) is as follows:

[0075] (11) The size of the spatial transformation network input image X is 100 × 32. First, a set of control points are predicted on the image X through the localization network;

[0076] (12) Then, the control points are used in the grid generator to calculate the Thin-Plate-Spline (TPS) variation parameters and generate a sampling grid P in the image X;

[0077] (13) The sampler simultaneously inputs the sampling grid P and the image X, and obtains the corrected image X' by sampling on the grid.

[0078] Furthermore, the specific process of step (2) is as follows:

[0079] (21) The representation batch normalization is introduced after the convolutional layer of the ResNet50 network structure to form the feature extractor we use;

[0080] (22) The regular text image obtained by the spatial transformation network is used as the input of the feature extractor. The four stages conv_2, conv_3, conv_4 and conv_5 of the feature extraction network output four feature maps with different resolution sizes, respectively. The resolution of the feature maps is 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image, respectively. The convolution block of the feature extraction network refers to Figure 2 ;

[0081] (23) Use a 1×1 convolutional layer to change the number of channels of each feature map to 128, and obtain a feature pyramid F containing all feature maps. r .

[0082] Furthermore, the structure of the feature enhancement network refers to Figure 3 , the specific process of step (3) is as follows:

[0083] (31) The feature pyramid F r The input is cascaded upscaled convolution, and upscaled with steps of 32, 16, 8 and 4 respectively;

[0084] (32) Then, the down-scale enhancement is performed continuously with a step size of 4, 8, 16 and 32 in the cascaded down-scale convolution stage, and the output feature pyramid is connected with the input feature pyramid elements of the cascaded up-scale convolution to obtain the final feature pyramid F e .

[0085] (33) The feature pyramid F e All feature maps of are upsampled to 1 / 4 of the original image to obtain the final visual feature map F f , whose size is H / 4×W / 4×512.

[0086] Furthermore, the specific process of step (4) is as follows:

[0087] Visual feature map F f It is extracted as a context feature sequence V through two layers of BiLSTM;

[0088] Visual feature map F f and the context feature sequence V to form a new feature space C = (F f , V).

[0089] Furthermore, the specific process of step (5) is as follows:

[0090] At time t, the decoder predicts the output yt:

[0091] y t =softmax(W y s t +b y )

[0092] Among them, softmax is the activation function, W y and b y is a trainable parameter, s t is the hidden state of LSTM at time t, and its expression is as follows:

[0093] s t =LSTM(y t-1 , g t ,s t-1 )

[0094] Among them, g t is the glimpse vector, which is composed of the feature space C = (c1, c2, ..., c N ) is composed of the weighted sum of each column, that is,

[0095]

[0096] Among them, α t，j is the attention weight, calculated as follows:

[0097]

[0098] Among them, e t，j =v T tanh(Fs t-1 +Gc j +b), v, F, G and b are all trainable parameters, the superscript T stands for transpose, and tanh() stands for the hyperbolic tangent function.

Claims

1. A natural scene text recognition method based on representation batch normalization, characterized in that: The specific steps include: Step 1: Regularize the natural scene text image to be recognized into a horizontal text image using a spatial transformer network; Step 2: introduce batch normalization into the existing network ResNet-50 to construct a feature extraction network; use the feature extraction network to extract the horizontal text image to obtain a number of visual feature maps with different resolutions; the visual feature maps with different resolutions form a feature pyramid F r ; Step 3: Enhance the feature pyramid F through the feature enhancement network r Perform up-scale enhancement and down-scale enhancement, and combine with the feature pyramid F described in step 2 r The elements are connected to obtain the final feature pyramid F e , for the feature pyramid F e Each feature map in is upsampled to obtain a feature map F of the same size f ; Step 4: Use the visual-semantic joint network to extract the feature map F described in step 3 f The extracted semantic information is combined with the feature map to obtain a new feature space. Step 5, using the decoder to predict the feature space described in step 4 to obtain the final recognition result; The characterization batch normalization described in step 2 is specifically expressed as follows: Given a feature map X∈R A×B×H×W , where A, B, H and W represent the batch size, number of channels, height and width of the feature map respectively; First, perform a calibration operation on the feature map X to obtain the calibrated feature map X cm , which is expressed as follows: X cm =X+ω m .K m In the formula, ω m Represents the weight value, K m Represents the statistics of feature map X; Then, for the feature map X cm Perform the centralization operation to obtain the centralized feature map X m , which is expressed as follows: X m =X cm -E(X cm ) In the formula, E(X cm ) represents the feature map X cm The mean of is expressed as follows: E(X cm )←mE(X cm )+(1-m)μ c In the formula, m represents the cumulative addition, ← represents the function mapping, μ c Represents the mean of each batch during training, expressed as follows; In the formula, h represents the height value of the feature map X, and w represents the width value of the feature map X; Next, for the feature map X m Perform the scaling operation to obtain the scaled feature map X s , which is expressed as follows: In the formula, ∈ is used to avoid zero variance, Var() represents the variance function, and the expression is as follows; In the formula, Represents the variance of each batch during training, expressed as follows: Secondly, for the feature map X s Scaling correction feature strength is performed to obtain the corrected feature map X cs , which is expressed as follows: X cs =X s ·R(ω v ·K s +oh b ) In the formula, ω v and ω b represents the trainable parameters, K s Represents the feature map X s Statistics of ; R() represents the correction function; Finally, the corrected feature map X cs Perform affine transformation to obtain the feature Y after radial transformation, which is expressed as follows: Y=X cs c+b Where γ is the scaling parameter and β is the translation parameter.

2. The natural scene text recognition method based on representation batch normalization according to claim 1, characterized in that: The spatial transformation network in step 1 includes a positioning network, a grid generator and a sampler; the method in step 1 is specifically as follows: Step 1.1, resize the natural scene text image to be recognized to 100 pixels × 32 pixels; Step 1.2, predicting a set of control points on the resized natural scene text image to be recognized through the localization network; Step 1.3, the grid generator calculates thin plate spline variation parameters using the control points, and generates a sampling grid on the resized natural scene text image to be recognized; Step 1.4, inputting the sampling grid and the resized natural scene text image to be recognized into the sampler at the same time, and obtaining a regularized horizontal text image by sampling on the sampling grid.

3. The natural scene text recognition method based on representation batch normalization according to claim 1, characterized in that: The method of step 2 is as follows: Step 2.1, adding representation normalization after each convolution layer of the existing network ResNet-50 to obtain the feature extraction network; Step 2.2, taking the horizontal text image as the input of the feature extraction network, and outputting four visual feature maps with different resolutions, respectively, wherein the resolutions of the visual feature maps are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input horizontal text image resolution, respectively; Step 2.3: Use a 1×1 convolutional layer to convert the number of channels of the visual feature map in step 2.2 into 128 to form a feature pyramid F. r .

4. The natural scene text recognition method based on representation batch normalization according to claim 1, characterized in that: The method of step 3 is as follows: Step 3.1: transform the feature pyramid F described in step 2 r Input to the cascaded upscale convolution, perform upscale enhancement with steps of 32, 16, 8 and 4 respectively, and output the upscale enhanced feature pyramid; Step 3.2, the feature pyramid after upscale enhancement in step 3.1 is subjected to cascaded downscale convolution, and downscale enhancement is performed with a step size of 4, 8, 16 and 32 in sequence, and the feature pyramid after downscale enhancement is output; Step 3.3: combine the downscale enhanced feature pyramid with the feature pyramid F in step 2 r The elements are connected to obtain the final feature pyramid F e ; Step 3.4: The feature pyramid F described in step 3.3 e All feature maps in the image are upsampled to 1 / 4 of the original image to obtain feature map F f , the feature map F f The size of is H / 4×W / 4×512, where H is the height of the feature map and W is the width of the feature map.

5. The natural scene text recognition method based on representation batch normalization according to claim 1, characterized in that: The visual-semantic joint network is composed of a two-layer BiLSTM network; the method of step 4 is as follows: First, the feature map F described in step 3 is transformed into f Extracted as context feature sequence V; Secondly, the feature map F f and the context feature sequence V to form a new feature space C = (F f , V).

6. The natural scene text recognition method based on representation batch normalization according to claim 1, characterized in that: The method of step 5 is as follows: The output y predicted by the decoder at time t t , which is expressed as follows: y t =softmax(W y s t +b y ) In the formula, softmax is the activation function, W y and b y is a trainable parameter, s t is the hidden state of LSTM at time t, and its expression is as follows: s t =LSTM(y t-1 ,g t ,s t-1 ) Among them, y t-1 is the output predicted by the decoder at time t-1, s t-1 is the hidden state of LSTM at time t-1, g t is the glimpse vector, which is composed of the feature space C = (c1, c2, ..., c j , …, c N ) is composed of the weighted sum of the elements of each column, that is, Among them, c j represents the j-th column element value of feature space C, N represents the total number of columns in feature space C, α t，j is the attention weight, expressed as follows: Among them, e t，j =v T tanh(Fs t-1 +Gc j +b), v, F, G and b are all trainable parameters, the superscript T stands for transpose, and tanh() stands for the hyperbolic tangent function.

Images