A text image recognition method with multiple reading directions

Abstract

The present invention discloses a method for recognizing text images with multiple reading directions. The present invention preprocesses a natural scene image containing text to obtain a preprocessed text block image; after extracting image features from the preprocessed text block image in a convolutional backbone network, a cross-self-attention module is used to obtain the direction information of texts in different reading orders; finally, the cross-self-attention module is used to decode and output the texts in different reading orders, thereby achieving decoding and recognition of text sequences in different reading orders. The present invention realizes the recognition of text images arranged in different directions with only one network feedforward with a slightly increased delay, and the recognition performance is still improved compared to the ordinary model.

Description

Technical Field

[0001] The present invention relates to a method for recognizing text images, and in particular to a method for recognizing text images in multiple reading directions. Technical Background

[0002] With the booming development and growing popularity of the short video and live streaming industries, the review and understanding of their content ecosystem has become increasingly important. If the images and text content of short videos and live streaming are not restricted, they can easily lead to negative mental orientation in young people and have a negative impact and significant impact on social order and stability. Because many up-loaders and anchors use the front-facing camera to shoot in selfie mode, their images may appear mirrored. Current text recognition models default to decoding from left to right, according to the normal reading order. At the same time, considering industrial performance limitations, adding a module to determine the direction would introduce unnecessary process logic and complexity, thus posing a significant challenge to text recognition in images with multiple reading orders.

[0003] Currently, there is little research in the academic community on text recognition that focuses on the recognition of text in multi-directional reading order. At the same time, the simple logic of first classifying and then recognizing is difficult to implement in the industry with strict speed requirements. In addition, since the classification network itself has an upper limit on accuracy, this method will also have a negative impact on recognition performance. Summary of the Invention

[0004] In order to solve the problems and needs existing in the background technology, the present invention provides a method for text image recognition in multiple reading directions. The present invention aims to use a lightweight cross-self-attention mechanism on the output sequence after extracting image features through a convolutional network. On the one hand, the input sequence can be contextually modeled to improve its recognition performance. On the other hand, through the characteristics of the cross-self-attention mechanism, the query vector part is defined as the position encoding corresponding to each output character, and the correct reading order is learned from the attention feature map, so that with a slightly increased network delay, text strings arranged in different directions can be recognized and decoded through only one feedforward reasoning. That is, the algorithm model in the present invention can recognize the text character information in text images arranged in different reading orders through only one network feedforward reasoning.

[0005] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0006] 1) Preprocessing a natural scene image containing text to obtain a preprocessed text block image;

[0007] 2) The preprocessed text block image is input into the backbone convolutional neural network for feature extraction and downsampling to obtain a one-dimensional visual feature map;

[0008] 3) The one-dimensional visual feature map is input into the cross-self-attention module for parallel cross-self-attention correction to obtain the corrected text vector;

[0009] 4) After the corrected text vector is input into the character decoding module for decoding, the recognized text string is output.

[0010] The step 1) is specifically as follows:

[0011] The text detection algorithm is used to capture the text image of a natural scene image containing text to obtain the original text block image. The original text block image is then enhanced by performing equal-probability random direction rotation. The height of the original text block image after random direction transformation is interpolated to a preset height, and the width of the original text block image after random direction transformation is scaled according to the aspect ratio of the original text block image to obtain a standard text block image. Finally, the standard text block image is normalized with a mean of 0 and a variance of 1 to obtain the preprocessed text block image.

[0012] The backbone convolutional neural network in step 2) is mainly composed of a convolution downsampling module, which is sequentially passed through a 6-layer depth convolution module, a first separable convolution downsampling module, a 12-layer depth convolution module, and a second separable convolution downsampling module, followed by a mean pooling downsampling module. The preprocessed text block image is input into the convolution downsampling module, and the mean pooling downsampling module outputs a one-dimensional visual feature map.

[0013] The step 3) is specifically as follows:

[0014] In the cross-self-attention module, the one-dimensional visual feature map is first encoded in the width dimension and subjected to two different linear transformations to obtain a feature query vector, a feature key vector and an eigenvalue vector respectively. Then, the feature query vector, the feature key vector and the eigenvalue vector are sliced ​​in the channel dimension to obtain N feature query sub-vector segments, feature key sub-vector segments and eigenvalue sub-vector segments respectively and number them in sequence. Cross-self-attention correction is performed on each feature query sub-vector segment, feature key sub-vector segment and eigenvalue sub-vector segment with the same number to obtain each numbered corrected text sub-vector segment. Finally, each corrected text sub-vector segment is spliced ​​in the channel dimension according to the number to obtain a corrected text vector.

[0015] In step 3), a matrix dot product is performed on the feature query sub-vector segment and the feature key sub-vector segment with the same number, followed by a Softmax operation to obtain a weight distribution map. The weight distribution map is then subjected to a matrix dot product with the corresponding numbered feature value sub-vector segment to obtain the corrected text sub-vector segment with the current number. The calculation formula is as follows:

[0016]

[0017] Among them, h i represents the i-th corrected text sub-vector segment, b i represents the i-th learning bias parameter, q i represents the i-th feature query subvector segment, k i represents the i-th characteristic key subvector segment, b i represents the i-th eigenvalue subvector segment, v i represents the i-th eigenvalue subvector segment, Softmax(·) represents the Softmax operation, which is to normalize the result of the matrix dot product, C is the feature dimension of the one-dimensional visual feature map, and T represents the transpose operation.

[0018] The step 4) is specifically as follows:

[0019] In the character decoding module, the corrected text vector is first linearly transformed so that the feature dimension of the linearly transformed corrected text vector is equal to the number of text categories. Then, the Softmax operation is performed on the global text feature vector after the linear transformation to obtain the character probability distribution. According to the character probability distribution, the linearly transformed corrected text vector is mapped to the text dictionary to achieve decoding and output the recognized text recognition sequence.

[0020] The beneficial effects of the present invention are mainly manifested in:

[0021] Compared with traditional text recognition methods, the present invention can simultaneously recognize and decode text images with different text arrangement directions. Compared with conventional convolutional recurrent neural networks, it has a certain performance improvement because it has added a cross-self-attention module with stronger context modeling capabilities.

[0022] The present invention can not only recognize normal text and left-right mirrored text, but also can well recognize text images rotated 180 degrees from the original image and text images mirrored up and down. In actual application scenarios, it can be freely selected according to actual data distribution and needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the method of the present invention.

[0024] Figure 2 It is the overall network structure logic diagram in the method of the present invention, which mainly includes three parts: backbone convolutional neural network, cross self-attention module, and character decoding module.

[0025] Figure 3 It is a schematic diagram of the backbone convolutional neural network structure in the method of the present invention.

[0026] Figure 4This is a schematic diagram of the structure of the cross-self-attention module used to determine the direction in the method of the present invention.

[0027] Figure 5 are illustrations of images of text in different reading orders.

[0028] Figure 6 It is a diagram of the pre-processed text images in different reading orders.

[0029] Figure 7 It is a visualization of the attention distribution map of the attention mechanism. DETAILED DESCRIPTION

[0030] In order to more clearly illustrate the purpose and technical solutions of the present invention, the present invention is further described in detail below with reference to the accompanying drawings.

[0031] The embodiments implemented according to the method of the present invention are as follows:

[0032] like Figure 1 As shown, the present invention includes the following steps:

[0033] 1) Preprocessing a natural scene image containing text to obtain a preprocessed text block image;

[0034] Step 1) is specifically as follows:

[0035] The text image of the natural scene image containing text is intercepted by the text detection algorithm to obtain the original text block image, and the original text block image is enhanced by random rotation with equal probability. In the specific implementation, the rotation direction of the image is vertical mirror flip, horizontal mirror flip, rotation 180° or unchanged. The height of the original text block image after random direction transformation is interpolated to the preset height H, and the width of the original text block image after random direction transformation is scaled according to the aspect ratio of the original text block image to obtain the standard text block image x∈R H×W×3 ,Finally, the standard text block image is normalized with a mean of 0 and a variance of 1 to obtain the preprocessed text block image.

[0036] 2) Input the preprocessed text block image into the backbone convolutional neural network for feature extraction and downsampling to obtain a one-dimensional visual feature map with a height of 1;

[0037] like Figure 3 As shown, the backbone convolutional neural network in step 2) is mainly composed of a convolution downsampling module, which is sequentially connected to a 6-layer depth convolution module, a first separable convolution downsampling module, a 12-layer depth convolution module, and a second separable convolution downsampling module, followed by a mean pooling downsampling module. The preprocessed text block image is input into the convolution downsampling module, and the mean pooling downsampling module outputs a one-dimensional visual feature map.

[0038] 3) The one-dimensional visual feature map is input into the cross-self-attention module for parallel cross-self-attention correction to obtain the corrected text vector;

[0039] like Figure 4 As shown, step 3) is specifically as follows:

[0040] In the cross-self-attention module, the one-dimensional visual feature map is first encoded in the width dimension and subjected to two different linear transformations to obtain the feature query vector, feature key vector and eigenvalue vector respectively. Then, the feature query vector, feature key vector and eigenvalue vector are sliced ​​in the channel dimension to obtain N feature query sub-vector segments, feature key sub-vector segments and eigenvalue sub-vector segments respectively and number them in sequence. The feature query sub-vector segments, feature key sub-vector segments and eigenvalue sub-vector segments with the same numbers are cross-self-attention corrected to obtain the corrected text sub-vector segments with different numbers. Finally, the corrected text sub-vector segments are spliced ​​in the channel dimension according to the numbers to obtain the corrected text vector. This module decomposes the original large matrix task operation into N (N is 4 in this embodiment) sub-tasks to be performed in the corresponding subspace, b i It is used to learn the different information contained in different subspaces. The calculation formula is as follows:

[0041]

[0042] K=W k F

[0043] V=W v F

[0044] Among them, Q represents the feature query vector, that is, the position encoding of the character sequence of the one-dimensional visual feature map, K represents the feature key vector, and V represents the feature value vector. Indicates dimensions are L×C, W k Represents the first linear transformation parameter matrix, W v Represents the second linear transformation parameter matrix.

[0045] In this step, the feature query sub-vector segment q i Represents the position encoding of the i-th position, which is regarded as the i-th branch task in multi-task training. The query vector and key vector of the cross attention mechanism input calculate the similarity and weight the output with the value vector. The output vector and the query vector q i There is a one-to-one correspondence between the two, so the mechanism aims to query the sub-vector segment q in the input feature i The vector encoding the i-th position in the output corresponds to the i-th character.

[0046] In step 3), a matrix dot product is performed on the feature query subvector segment and the feature key subvector segment with the same number, followed by a Softmax operation to obtain a weight distribution map (Attention Map), which is used to characterize and determine the decoding direction and obtain the corrected text vector. A matrix dot product is then performed on the weight distribution map and the corresponding numbered feature value subvector segment to obtain the corrected text subvector segment with the current number. The calculation formula is as follows:

[0047]

[0048] Among them, h i represents the i-th corrected text sub-vector segment, b i represents the i-th learning bias parameter, q i represents the i-th feature query subvector segment, k i represents the i-th characteristic key subvector segment, b i represents the i-th eigenvalue subvector segment, v i represents the i-th eigenvalue subvector segment. Softmax(·) represents the Softmax operation, which normalizes the matrix dot product result to between 0 and 1. This allows the local self-attention module to assign higher weights to text areas in the image and lower weights to background areas, thereby eliminating interference from complex backgrounds. C represents the feature dimension of the one-dimensional visual feature map, and T represents the transpose operation.

[0049] 4) After the corrected text vector is input into the character decoding module for decoding, the recognized text string is output.

[0050] Step 4) is specifically as follows:

[0051] In the character decoding module, the corrected text vector is first linearly transformed so that the feature dimension of the linearly transformed corrected text vector is equal to the number of text categories. Then, the global text feature vector after the linear transformation is subjected to a Softmax operation to obtain a character probability distribution. The corrected text vector after the linear transformation is subjected to a character dictionary mapping according to the character probability distribution to achieve decoding and output the recognized text recognition sequence. The number of text categories is specifically 5001, including 5000 commonly used Chinese and English characters and a termination character EOS as a separate text category to mark the end of the current character string decoding. When decoding from left to right, until the last character is decoded or the first termination character EOS is decoded, slicing is performed and the decoding is terminated to obtain the final text recognition sequence. The present invention does not require the use of complex autoregressive decoding. Compared with ordinary CTC decoding, the decoding efficiency is further improved due to the absence of blank characters.

[0052] The training and test datasets used in this example are derived from publicly available synthetic datasets in the field of text recognition: SynthText (English), MJ-Synth (English), and SynthText-Chinese (Chinese), totaling approximately 14 million words. The character dictionary used in this paper is set to 5001, including 5000 commonly used Chinese and English characters and one terminating character, EOS. The proposed network model was trained on a 2080Ti GPU using the Adam optimizer with a batch size of 512 and 250,000 iterations.

[0053] Input an original text image, after random direction transformation, as shown below Figure 5 As shown in the figure, (a) is the original image; (b) is the image after left-right mirroring; (c) is the image after top-bottom mirroring; (d) is the image after 180-degree rotation. Figure 6 .in, Figure 6 (a), (b), (c), and (d) are Figure 5 (a), (b), (c), (d) are network input images after scaling and padding.

[0054] The above Figure 6 The four images are processed by the backbone convolutional neural network ( Figure 3 ) to obtain a one-dimensional visual feature map and then feed it into the cross self-attention module ( Figure 4 ) to get the corrected text vector, and the attention distribution diagram in the middle can be visualized as Figure 7 , respectively as Figure 7 (a), (b), (c), and (d) are the intermediate results of the above-mentioned different input images under the cross-self-attention module, representing their different reading orders.

[0055] The output of the corrected text vector after passing through the character decoding module is mapped through the text dictionary to obtain the final recognized string, and the final recognized string is "Available".

[0056] It can be seen from the above embodiments that the present invention can accurately identify text images with characters arranged in different directions, and the reading direction of the text image can be determined through the cross-attention weight map of the intermediate results (by checking whether the highlighted part of the first line is the first pixel block).

Claims

1. A method for recognizing text images in multiple reading directions, characterized in that: The following steps are involved: 1) Preprocessing a natural scene image containing text to obtain a preprocessed text block image; 2) The preprocessed text block image is input into the backbone convolutional neural network for feature extraction and downsampling to obtain a one-dimensional visual feature map; 3) The one-dimensional visual feature map is input into the cross-self-attention module for parallel cross-self-attention correction to obtain the corrected text vector; 4) Inputting the corrected text vector into the character decoding module for decoding, and outputting the recognized text string; The backbone convolutional neural network in step 2) is composed of a convolution downsampling module, which is sequentially connected to a 6-layer depth convolution module, a first separable convolution downsampling module, a 12-layer depth convolution module, and a second separable convolution downsampling module, and then connected to a mean pooling downsampling module. The preprocessed text block image is input into the convolution downsampling module, and the mean pooling downsampling module outputs a one-dimensional visual feature map. The step 3) is specifically as follows: In the cross-self-attention module, the one-dimensional visual feature map is first encoded in the width dimension and subjected to two different linear transformations to obtain a feature query vector, a feature key vector and an eigenvalue vector respectively. Then, the feature query vector, the feature key vector and the eigenvalue vector are sliced ​​in the channel dimension to obtain N feature query sub-vector segments, feature key sub-vector segments and eigenvalue sub-vector segments respectively and number them in sequence. Cross-self-attention correction is performed on each feature query sub-vector segment, feature key sub-vector segment and eigenvalue sub-vector segment with the same number to obtain each numbered corrected text sub-vector segment. Finally, each corrected text sub-vector segment is spliced ​​in the channel dimension according to the number to obtain a corrected text vector.

2. The method for multi-reading direction text image recognition according to claim 1, characterized in that: The step 1) is specifically as follows: The text detection algorithm is used to capture the text image of a natural scene image containing text to obtain the original text block image. The original text block image is then enhanced by performing equal-probability random direction rotation. The height of the original text block image after random direction transformation is interpolated to a preset height, and the width of the original text block image after random direction transformation is scaled according to the aspect ratio of the original text block image to obtain a standard text block image. Finally, the standard text block image is normalized with a mean of 0 and a variance of 1 to obtain the preprocessed text block image.

3. The method for multi-reading direction text image recognition according to claim 1, characterized in that: In step 3), a matrix dot product is performed on the feature query sub-vector segment and the feature key sub-vector segment with the same number, followed by a Softmax operation to obtain a weight distribution map. The weight distribution map is then subjected to a matrix dot product with the corresponding numbered feature value sub-vector segment to obtain the corrected text sub-vector segment with the current number. The calculation formula is as follows: Among them, h i represents the i-th corrected text sub-vector segment, b i represents the i-th learning bias parameter, q i represents the i-th feature query subvector segment, k i represents the i-th characteristic key subvector segment, v i represents the i-th eigenvalue subvector segment, Softmax(·) represents the Softmax operation, which is to normalize the result of the matrix dot product, C is the feature dimension of the one-dimensional visual feature map, and T represents the transpose operation.

4. The method for multi-reading direction text image recognition according to claim 1, characterized in that: The step 4) is specifically as follows: In the character decoding module, the corrected text vector is first linearly transformed so that the feature dimension of the linearly transformed corrected text vector is equal to the number of text categories. Then, the Softmax operation is performed on the linearly transformed corrected text vector to obtain the character probability distribution. According to the character probability distribution, the linearly transformed corrected text vector is mapped to the text dictionary to achieve decoding and output the recognized text recognition sequence.

Images