An end-to-end text extraction and recognition head method based on character feature abstraction

Abstract

The application discloses an end-to-end text extraction and recognition head method based on character feature abstract, and belongs to the fields of deep learning, computer vision and end-to-end text extraction. The method gradually extracts global features of characters in multiple stages, and abstracts historical features in each stage, so that the character features are better extracted. The method predicts the text recognition result in each stage, better converges the network, improves the accuracy of the final result under the premise of increasing less calculation amount, and has the characteristics of simplicity, rapidness and accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning, computer vision and end-to-end text spotting, and designs a fast and high-precision text extraction and recognition head algorithm based on character feature abstraction for a two-stage end-to-end text extraction algorithm. BACKGROUND

[0002] End-to-end text spotting is a basic research topic in the field of computer vision, and its main task is to detect the position of text in a given image and recognize the text content of each text region in an end-to-end manner. At present, the mainstream end-to-end text spotting algorithm adopts a shared image feature extraction backbone network, and uses a network architecture in which the detection head for detecting the position of text and the recognition head for recognizing the content of text are separated to perform end-to-end text spotting. In the inference process, the feature spectrum of the input image is first extracted using the backbone network. Then, the feature spectrum is used as input to locate the bounding box position of the text using the detection head. Finally, the text features are cut out from the corresponding region on the feature spectrum according to the obtained text bounding box position, and after shape alignment of the text features of different text instances, the recognition head is used to recognize the text content, thereby realizing end-to-end text spotting. The present application is committed to proposing a fast and high-precision recognition head algorithm based on character feature abstraction to improve the accuracy and inference speed of the end-to-end text spotting network.

[0003] At present, a fast and high-precision recognition head algorithm is one of the key factors for the success of end-to-end text spotting. According to the different text information extraction methods, the existing recognition heads for end-to-end text spotting can be mainly divided into two categories: sequence generation-based methods and visual feature extraction-based methods. The sequence generation-based method regards the text recognition problem as an image translation problem, takes the predicted output of the previous character as the input for predicting the next character, uses an attention module to automatically find the region that should be focused on when predicting the next character, and outputs the text content in a sequence until a special end symbol is output. However, this method needs to generate text sequences in series, which requires a lot of attention calculation and is very time-consuming. The visual feature extraction-based method takes the extraction of character features in each region as the target, and uses a recurrent neural network or a long short-term memory network to interact information between characters, then uses a simple linear classifier to perform parallel prediction of fixed-length characters, and finally removes duplicate characters through a CTC decoder to obtain the text content. Although this method can perform character prediction in parallel and has a faster inference speed, it is difficult to capture character features well, and the accuracy of character prediction is low.

[0004] To solve the above problems, the application provides a novel end-to-end text extraction and recognition head algorithm based on character feature abstract. The algorithm reduces the difficulty of character feature extraction by gradually extracting character features from the global features of the text in multiple stages. At each stage, in addition to further character feature extraction on the output of the previous stage of character feature extraction, the current stage character feature extraction result is also abstracted and fused with the historical character feature abstract result of the previous stage to fully fuse the character feature information of the past stage and generate the historical character feature abstract result of the current stage. In order to accelerate the convergence of the model and better supervise the network, the historical character feature abstract result of each stage will be used for text content prediction. In the training stage, the prediction result will be used for network convergence. In the test stage, the result with the highest confidence will be selected from the multiple prediction results as the final result, thereby effectively improving the inference speed and accuracy of the end-to-end text extraction network. SUMMARY

[0005] The application provides a text extraction and recognition head method based on character feature abstract, which aims to solve the problems that the recognition head used in the existing two-stage end-to-end text extraction technology uses sequence generation or character visual feature extraction method for text recognition, on the one hand, sequence generation needs to be performed in series for character prediction, and the inference speed is slow, on the other hand, the visual features of characters are difficult to extract, and the recognition performance is poor.

[0006] To achieve the above purpose, the application adopts the following technical scheme: an end-to-end text extraction and recognition head method based on character feature abstract, which includes a backbone network detection head feature shape alignment module, a feature down-sampling module, N character feature abstract modules, N character prediction modules, and a result selection module. The method steps include:

[0007] Step 1: feature down-sampling;

[0008] First, the input image is subjected to backbone network detection head feature shape alignment, and the shape of the text feature F obtained after alignment is C1xh xw t , wherein C1 is the number of feature channels, h is the height, and w is the width; then M+1 ConvBN modules composed of convolution layers, batch normalization layers and ReLU activation functions are cascaded, wherein the first ConvBN module is used for channel down-sampling to down-sample the number of feature channels to C2, and the last M ConvBN modules are used for height down-sampling to down-sample the height to 1, and the specific process is as follows:

[0009]

[0010]

[0011] , wherein Indicates that the convolution kernel size is k h ×k w , the step size is s h ×s w The convolution layer, BN(·) is the batch normalization layer, ReLU(·) is the ReLU activation layer, In addition, the output of the last ConvBN module Renamed to F r , as the output of this module, F r The shape is C2×1×w;

[0012] Step 2: Concatenate character feature summaries;

[0013] Since the text feature F after channel height downsampling r The local features of the text image are difficult to represent the global features of the characters. This module extracts the global summary features of the characters from the local features of the text in N stages by cascading N character feature summary modules. The character feature summary module contains a global feature extraction branch and a historical character feature summary branch. For the kth module, the input is the character global feature extraction result of the k-1th module. and historical character summary features First, the Further extract the global character features of the current stage Subsequently, in order to make full use of the features extracted by the previous modules, and Perform summary fusion to generate two historical character feature summary results of the current stage respectively and in Used as the historical character summary feature input for the next module, Used for character prediction; the specific process is as follows, for the kth stage:

[0014]

[0015]

[0016]

[0017]

[0018] in, is the global feature extraction result of the character in the kth stage, is the historical character summary feature of the kth stage used for the historical character feature summary of the next stage, For the historical character summary features used for character prediction in the current stage, Attn(·) is a multi-head self-attention layer, MLP(·) is a multi-layer perceptron layer, and Concat(·,·) is a channel concatenation operation; in particular, for the first stage, the input of the character feature summary module is and are the channel height down-sampled text features F r ;

[0019] Step 3: character prediction;

[0020] The character prediction module takes as input the historical character summary features used for character prediction in each of the N stages of the concatenated character feature summary module, and uses a fully connected network for the classification prediction of characters for each historical character summary feature to obtain N strings of length w; in the training stage, the N prediction results are respectively used to calculate the recognition loss using the CTC loss function for the iterative training of the network; in the test stage, the N prediction results are respectively decoded using the CTC decoder to obtain N text prediction results and the confidence of each result; the specific process is as follows: for the kth historical character summary feature

[0021]

[0022]

[0023]

[0024]

[0025]

[0026] wherein FC(·) is a fully connected layer for the classification prediction of characters, is the character feature at the i-th position in the kth stage, with a shape of C2×1×1, is the classification prediction result of the i-th character in the kth stage, and the confidence of the character is denoted as T k is the set of all predicted characters in the kth stage, CTCLoss(·) is a CTC loss calculation function, is the text recognition loss value in the kth stage, L rec is the recognition loss value of the entire recognition head, CTCDecode(·) is a CTC decoder, is the text prediction result after removing duplicate characters after decoding;

[0027] Step 4: result selection;

[0028] After the character prediction module, N text prediction results are obtained k=1,..., N; In order to better select one result as the final text recognition result, the result selection module selects the result with the highest confidence as the final result, and the specific process is as follows:

[0029]

[0030]

[0031] Where, Conf k is the confidence of the kth recognition result, T F is the final text recognition result, and is the result with the highest confidence among the k stage text recognition results.

[0032] The character feature abstract-based text extraction recognition head method for the two-stage end-to-end text extraction network provided by the present application better extracts character features by gradually extracting character global features in multiple stages and performing historical feature abstraction at each stage. And by performing text recognition result prediction at each stage, the network is better converged and the final result accuracy is improved under the premise of increasing less computational amount, which has the characteristics of simplicity, rapidness and accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 It is the overall process framework diagram of the text extraction recognition algorithm based on character feature abstraction of the present application.

[0034] Figure 2 It is a feature down-sampling module schematic diagram of the present application.

[0035] Figure 3 It is a character feature abstraction module schematic diagram of the present application.

[0036] Figure 4 It is a character prediction module schematic diagram of the present application. DETAILED DESCRIPTION

[0037] The present application is implemented on the Pytorch deep learning platform, uses the backbone network of the two-stage end-to-end text extraction network such as PAN++ or ABCNet v2 to generate image feature spectrum, and uses the detection head to detect the text area. The text features after being intercepted and shape-aligned on the feature spectrum using the detection result are sent to the recognition head proposed by the present application for text recognition, which specifically includes the following steps:

[0038] Step one: using linear interpolation to align the shape of the text features to a shape with a width of 32 and a height of 8, and using a 3x3 convolution kernel to align the channel number of the features to 256.

[0039] Step two: input the aligned text feature processed in step one into the text feature channel height down-sampling module. First, use a ConvBN module to reduce the channel number to 128, and then use three ConvBN modules to down-sample the height to 1, to obtain a channel height down-sampled text feature F r .

[0040] Step three: cascade three character feature abstract modules. Each module takes the global character feature output by the previous module and the historical character abstract feature used for abstract fusion as input, and outputs the global character feature of the current stage, the historical character abstract feature used for abstract fusion, and the historical character abstract feature used for character prediction, thereby obtaining three historical character abstract features used for character prediction with the shape of 128x1x32. In particular, the two inputs of the first module are both the feature F r .

[0041] Step four: for the abstract feature F pk of the kth stage, regard it as 32 character vectors embedded in a 128-dimensional space, use a fully connected network to classify each character vector into 36 characters including 26 English letters and 10 digits, thereby obtaining three text recognition results, each containing 32 characters, and the confidence of each text recognition result is the average of the classification confidences of its 32 characters.

[0042] Step five: in the training stage, the CTG loss value will be calculated using the three recognition results and the text label respectively to train the network, the stochastic gradient descent algorithm is selected for network parameter optimization, the batch size is set to 8, the training iteration is set to 240,000 times, the initial learning rate is set to 0.01, and the learning rate is reduced to 0.001 at the 160,000th iteration and to 0.0001 at the 220,000th iteration. In the test stage, CTC decoding will be used on the three recognition results to remove repeated characters. Finally, compare the confidences of the three text recognition results, and select the one with the highest confidence as the final output.

Claims

1. An end-to-end text extraction and recognition head method based on character feature summarization, comprising a backbone network detection head feature shape alignment module, a feature downsampling module, an N-character feature summary module, an N-character prediction module, and a result selection module. The method steps include: Step 1: Feature downsampling; First, the input image is aligned with the backbone network detection head feature shape, and the text feature F with a shape of C1×h×w is obtained after alignment. t , where C1 is the number of feature channels, h is the height, and w is the width; then cascade M+1 ConvBN modules consisting of convolutional layers, batch normalization layers, and ReLU activation functions. The first ConvBN module is used for channel downsampling, downsampling the number of feature channels to C2, and the next M ConvBN modules are used for height downsampling, downsampling the height to 1. The specific process is as follows: in, Indicates that the convolution kernel size is k h ×k w , the step size is s h ×s w The convolution layer, BN(·) is the batch normalization layer, ReLU(·) is the ReLU activation layer, In addition, the output of the last ConvBN module Renamed to F r , as the output of this module, F r The shape is C2×1×w; Step 2: Concatenate character feature summaries; Since the text feature F after channel height downsampling r The local features of the text image are difficult to represent the global features of the characters. This module extracts the global summary features of the characters from the local features of the text in N stages by cascading N character feature summary modules. The character feature summary module contains a global feature extraction branch and a historical character feature summary branch. For the kth module, the input is the character global feature extraction result of the k-1th module. and historical character summary features First, the Further extract the global character features of the current stage Subsequently, in order to make full use of the features extracted by the previous modules, and Perform summary fusion to generate two historical character feature summary results of the current stage respectively and in Used as the historical character summary feature input for the next module, Used for character prediction; the specific process is as follows, for the kth stage: in, is the global feature extraction result of the character in the kth stage, is the historical character summary feature of the kth stage used for the historical character feature summary of the next stage, is the historical character summary feature used for character prediction in the current stage, Attn(·) is the multi-head self-attention layer, MLP(·) is the multi-layer perceptron layer, and Concat(·,·) is the channel-by-channel cascade operation; For the first stage, the input of the character feature summary module is and All are channel height downsampled text features F r ; Step 3: Character prediction; The character prediction module takes the historical character summary features used for character prediction in each stage of the N stages of the cascaded character feature summary module as input, and uses a fully connected network to perform character classification prediction for each historical character summary feature to obtain N strings of length w; in the training phase, these N prediction results will be respectively calculated using the CTC loss function to calculate the recognition loss for iterative training of the network; in the testing phase, these N prediction results will be respectively decoded using the CTC decoder to obtain N text prediction results and the confidence of each result; the specific process is as follows: for the kth historical character summary feature Among them, FC(·) is a fully connected layer, which is used to predict the classification of characters. is the character feature of the i-th position in the k-th stage, with a shape of C2×1×1, is the classification prediction result of the i-th character in the k-th stage, and the confidence of the character is recorded as T k is the set of all predicted characters in the kth stage, CTCLoss(·) is the CTC loss calculation function, and L reck is the text recognition loss value of the kth stage, L rec is the recognition loss value of the entire recognition head, CTCDecode(·) is the CTC decoder, The text prediction result after decoding and removing repeated characters; Step 4: Result selection; After the character prediction module, N text prediction results are obtained k = 1, ..., N; in order to better select a result as the final text recognition result, the result selection module selects the result with the highest confidence as the final result. The specific process is as follows: Among them, Conf k is the confidence of the kth recognition result, T F is the final text recognition result, which is the result with the highest confidence among the text recognition results of these k stages.

Images