A method and device for recognizing text lines in arbitrary text directions

Abstract

The present invention relates to the field of image recognition technology, and specifically discloses a method for text line recognition with arbitrary text direction, including: obtaining a text line image training set; constructing an initial text line recognition model; inputting the training text line images in the text line image training set into the initial text line recognition model for training to obtain a trained text line recognition model; inputting the target text line image to be recognized into the trained text line recognition model to output the text content recognition result and the text direction recognition result of the target text line image. The present invention also discloses a text line recognition device with arbitrary text direction. The text line recognition method with arbitrary text direction provided by the present invention can complete text direction recognition while recognizing text lines, and because the features of the recognized text are reused in the process of recognizing the direction, the accuracy of direction recognition is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and more specifically, to a text line recognition method for arbitrary text directions and a text line recognition device for arbitrary text directions. Background Art

[0002] In the process of photographing and scanning documents, in order to obtain better imaging effects, the photographing device or the document to be scanned will be rotated, resulting in the text direction of the obtained document image not necessarily being upright. In order to obtain better recognition effects, traditional text recognition methods often add a text direction determination device before text recognition, first adjust the direction of the document image to the positive direction or manually adjust it to the positive direction before recognition. Manual adjustment cannot be applied to batch recognition. On the one hand, adding a text direction determination device will increase additional consumption. On the other hand, if the text direction determination device makes an error and the document image direction is not adjusted to the positive direction, text recognition will still be wrong. Summary of the invention

[0003] In view of the above-mentioned drawbacks existing in the prior art, the present invention provides a text line recognition method with arbitrary text direction to solve the problem that manual adjustment existing in the prior art cannot be applied to batch recognition, and that adding a text direction determination device will increase extra consumption on the one hand, and on the other hand, if the text direction determination device makes an error, resulting in the document image direction not being adjusted to the positive direction, text recognition will still make mistakes.

[0004] As a first aspect of the present invention, a method for recognizing text lines in any text direction is provided, comprising the following steps:

[0005] Step S1: Obtain a text line image training set;

[0006] Step S2: construct an initial text line recognition model;

[0007] Step S3: inputting the training text line images in the text line image training set into the initial text line recognition model for training, thereby obtaining a trained text line recognition model;

[0008] Step S4: inputting the target text line image to be recognized into the trained text line recognition model to output the text content recognition result and the text direction recognition result of the target text line image.

[0009] Furthermore, the text line image training set includes a plurality of horizontal training text line images and vertical training text line images with a text direction of 0°, wherein the vertical training text line image is rotated 90° counterclockwise to become a horizontal training text line image, and the order of the corresponding text labels remains unchanged;

[0010] The text direction of 0° and the direction rotated 90° counterclockwise are defined as the positive text direction of the training text line image. During the training process, the training text line images with the positive text direction are randomly inverted, and the text labels of the inverted text line images are correspondingly changed to reverse order. The text direction of the inverted text line image is defined as the negative direction; wherein, after the training text line image with the original text direction of 0° is inverted, the text direction becomes 180° counterclockwise; wherein, after the training text line image with the original text direction of 90° counterclockwise is inverted, the text direction becomes 270° counterclockwise.

[0011] Furthermore, a unique number is defined for each character in the horizontal training text line image and the vertical training text line image, and during training, each character in the text label corresponding to the training text line image is converted into a corresponding character number as a sequence label corresponding to the training text line image;

[0012] Among them, the direction label of the training text line image with the positive direction is encoded as 1; the direction label of the training text line image with the negative direction is encoded as 0.

[0013] Furthermore, the constructing of the initial text line recognition model further includes:

[0014] The structure of the Resnet50 neural network is improved by using the SE module, and the improved Resnet50 neural network is used as the basic network, wherein the Resnet50 neural network has multiple convolution stages, and the multiple convolution stages include the first stage, the second stage, the third stage, the fourth stage and the fifth stage, and the SE module is added to the third stage and the fifth stage of the Resnet50 neural network; then the stride of the first convolution in the first stage, the second stage, the third stage, the fourth stage and the fifth stage is changed to (2, 2), (2, 2), (2, 1), (2, 1) and (2, 1) respectively;

[0015] After the basic network, a circulation layer is constructed, and between the basic network and the circulation layer, an escape operation from a two-dimensional feature sequence to a one-dimensional feature sequence is defined to connect the basic network layer and the circulation layer;

[0016] The first fully connected layer FC1 and the second fully connected layer FC2 are respectively connected after the circulation layer, wherein the first fully connected layer FC1 is a text content recognition fully connected layer, and the second fully connected layer FC2 is a text direction recognition fully connected layer.

[0017] Furthermore, the step of inputting the training text line images in the text line image training set into the initial text line recognition model for training to obtain a trained text line recognition model further includes:

[0018] Suppose the given text line image training set is denoted as Among them, I i represents the i-th text line image in the training set X, t i Represents the text line image I i The corresponding text content, y i Represents the text line image I i The corresponding text direction;

[0019] The text line image I i Input into the initial text line recognition model, assuming that the text line image I i The first feature sequence obtained by the basic network based on the Resnet50 neural network is f i cnn , the first feature sequence f i cnn After escaping, it becomes the second characteristic sequence f i ' cnn , the second feature sequence f i ' cnn Input into the LSTM recurrent layer to obtain the third feature sequence f i rnn , the third characteristic sequence f i rnn Input into the first fully connected layer FC1 to obtain the text content recognition result of the feature sequence; in the sequence dimension, the second feature sequence f i ' cnn and the third characteristic sequence f i rnn The average value is calculated, and then the average values ​​are added to obtain a fused feature sequence, and the fused feature sequence is input into the second fully connected layer FC2 to obtain a text direction recognition result of the feature sequence.

[0020] Further, selecting CTC to supervise the text content recognition fully connected layer also includes:

[0021] Let the input text line image be I i , and its corresponding real text content label is t i The probability of each column of the feature sequence obtained by the text content recognition fully connected layer on each character is pt i , CTC expects the following L1 value to be minimized, where π i is the predicted character sequence, p(π i |I i) is the predicted character sequence is π i The probability of CTC can quickly get from pt through the forward and backward algorithms i Calculate p(π i |I i ), the formula is as follows:

[0022]

[0023]

[0024] Among them, β is a mapping method defined in CTC, π i :β(π i ) = t i Represents the character sequence π i After being mapped by method β, it is equal to t i A collection of sequences.

[0025] Further, cross entropy loss is selected to supervise the text direction recognition fully connected layer, and further includes:

[0026] Let the input text line image be I i , the corresponding real text direction label is y i , the probability of the text direction predicted by the text direction recognition fully connected layer is py i , the cross entropy expects the following l2 value to be minimized, the formula is as follows:

[0027]

[0028] Wherein, N is the number of training text line images in the text line image training set.

[0029] As a second aspect of the present invention, a text line recognition device with arbitrary text direction is provided, comprising:

[0030] An acquisition module is used to acquire a text line image training set;

[0031] A construction module for constructing an initial text line recognition model;

[0032] A training module, used for inputting the training text line images in the text line image training set into the initial text line recognition model for training, so as to obtain a trained text line recognition model;

[0033] The output module is used to input the target text line image to be recognized into the trained text line recognition model to output the text content recognition result and the text direction recognition result of the target text line image.

[0034] The text line recognition method of arbitrary text direction provided by the present invention has the following advantages:

[0035] (1) No additional text direction recognition device is required. Text direction recognition is completed while text line recognition is performed. If the text direction recognition is negative, the corresponding text content is output in reverse order, that is, the text content of the original inverted text image;

[0036] (2) Based on the original text line recognition device, only one fully connected layer branch for determining the direction is added, and the additional overhead brought is almost negligible. It is not only simple, but also greatly improves the accuracy of direction recognition because the features of the recognized text are reused in the process of identifying the direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.

[0038] Figure 1 A flowchart of a text line recognition method with arbitrary text direction provided by the present invention.

[0039] Figure 2 A structural diagram of a text line recognition device with arbitrary text direction provided by the present invention.

[0040] Figure 3 A schematic diagram of the structure of the Resnet50 basic network with the SE module provided by the present invention.

[0041] Figure 4 A schematic diagram of the structure of the text line recognition model provided by the present invention. DETAILED DESCRIPTION

[0042] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a text line recognition method and device for arbitrary text direction proposed by the present invention in combination with the accompanying drawings and preferred embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] In this embodiment, a text line recognition method with arbitrary text direction is provided, such as Figure 1 As shown, the text line recognition method of any text direction includes:

[0044] Step S1: Obtain a text line image training set;

[0045] Step S2: construct an initial text line recognition model;

[0046] Step S3: inputting the training text line images in the text line image training set into the initial text line recognition model for training, thereby obtaining a trained text line recognition model;

[0047] Step S4: inputting the target text line image to be recognized into the trained text line recognition model to output the text content recognition result and the text direction recognition result of the target text line image.

[0048] Preferably, the training data in the text line image training set is preprocessed:

[0049] The text line image training set includes a plurality of horizontal training text line images and vertical training text line images with a text direction of 0°, wherein, in order to process vertical text, the vertical training text line image is counterclockwise rotated 90° to become a horizontal training text line image, and the corresponding text label order remains unchanged;

[0050] The text direction of 0° and the direction rotated 90° counterclockwise are defined as the positive text direction of the training text line image. During the training process, the training text line image with the positive text direction is randomly inverted, and the text label of the inverted text line image is changed to the reverse order. The text direction of the inverted text line image is defined as the negative direction; wherein, after the training text line image with the original text direction of 0° is inverted, the text direction becomes 180° counterclockwise; wherein, after the training text line image with the original text direction rotated 90° counterclockwise is inverted, the text direction becomes 270° counterclockwise. Random noise and blur are added to the training text line image, and the image light is randomly adjusted. Some examples and labels of training text line images are shown in Table 1 below:

[0051] Table 1

[0052]

[0053] Preferably, according to the business scenario and the collected training data set, a character set containing 9000 characters is defined, and a unique number is defined for each character. During training, each character in the text label corresponding to the training text line image is converted into a corresponding character number as a sequence label corresponding to the training text line image;

[0054] Among them, the direction label of the training text line image with the positive direction is encoded as 1; the direction label of the training text line image with the negative direction is encoded as 0.

[0055] Preferably, if Figure 3 As shown, the construction of the initial text line recognition model also includes:

[0056] In order to further improve the expressive power of the model, the SE module (Squeeze-and-Excitation) is used to improve the Resnet50 (Residual Network) neural network structure, and the improved Resnet50 neural network is used as the basic network, wherein the Resnet50 neural network has multiple convolution stages, and the multiple convolution stages include the first stage, the second stage, the third stage, the fourth stage and the fifth stage, and the SE module is added to the third stage and the fifth stage of the Resnet50 neural network; the model focuses on the spatial features within different receptive fields while strengthening the information fusion between different channels, and helps capture the spatial relationship between convolutional features and the relationship between convolutional feature channels by integrating the learning mechanism into the network, thereby enhancing the representation power of the model. Like the original CRNN model, all images need to be scaled to the same height (here we set it to 32) before entering the network. After the image passes through the convolution-based basic network, the corresponding spatial feature sequence f is obtained. cnn ∈R B×C×H×W (where B represents the batch size of the input image, C represents the number of sequence feature channels, H and W represent the height and width of the feature map respectively). From the perspective of W dimension, each feature vector ∈ R C They all correspond to each column of the feature map, and to each small square in the original image.

[0057] In order to better convert the deep features into sequential representations, we change H to 1, so the stride of the first convolution in stage1, stage2, stage3, stage4, and stage5 is changed to (2, 2), (2, 2), (2, 1), (2, 1), and (2, 1) respectively. The structures of stage1, stage2, stage3, stage4, and stage5 are as follows: Figure 3 shown.

[0058] After the basic network, a two-layer bidirectional recurrent neural network structure is constructed as a recurrent layer to predict the distribution of character labels corresponding to each sequence feature. Traditional recurrent neural network units are easily affected by the gradient vanishing problem during training, which limits the context range that it can be used and makes the training process difficult. The special design of LSTM (Long Short-Term Memory) can alleviate the gradient vanishing problem while capturing longer-range contextual relationships in the sequence. So here we choose a two-layer LSTM as the recurrent layer. There are two benefits to accessing the recurrent layer: 1) The recurrent layer structure can more effectively capture the contextual information in the sequence and can manipulate sequences of arbitrary lengths; 2) The bidirectional recurrent layer structure can focus on information from the beginning to the end and from the end to the beginning. In addition, between the basic network and the recurrent layer, a two-dimensional feature sequence to one-dimensional feature sequence escape operation is defined to connect the basic network layer and the recurrent layer; specifically, the basic network layer outputs a spatial feature sequence f cnn ∈R B×C×H×W , at this time H=1, for the spatial feature sequence f cnn Reshape to remove the H dimension and rearrange the C and W dimensions to get the loop layer input f' cnn ∈R B×L×C , where L = W, f' cnn After the loop layer, we get the feature sequence f of the fused context rnn .

[0059] The first fully connected layer FC1 and the second fully connected layer FC2 are connected after the circulation layer to respectively predict the text content distribution and the corresponding text direction distribution corresponding to the input image. rnn Input FC1 to get the probability of each character in each column of the feature sequence (which can be imagined as each character block in the original image) as the character set defined above. In practical application, the character with the largest probability value is taken as its corresponding predicted character. cnn and f rnn The average value is calculated in the sequence dimension, and then the added feature input FC2 is obtained to obtain the probability of the input image being in the positive direction and the negative direction. In practical applications, the direction with the largest probability value is taken as the predicted direction. Among them, the first fully connected layer FC1 is a text content recognition fully connected layer, and the second fully connected layer FC2 is a text direction recognition fully connected layer.

[0060] Preferably, if Figure 4 As shown, the step of inputting the training text line images in the text line image training set into the initial text line recognition model for training to obtain a trained text line recognition model further includes:

[0061] Suppose the given text line image training set is denoted as Among them, I i represents the i-th text line image in the training set X, t i Represents the text line image I i The corresponding text content, y i Represents the text line image I i The corresponding text direction;

[0062] The text line image I i Input into the initial text line recognition model, assuming that the text line image I i The first feature sequence obtained by the basic network based on the Resnet50 neural network is f i cnn , the first feature sequence f i cnn After escaping, it becomes the second characteristic sequence f i ' cnn , the second feature sequence f i ' cnn Input into the LSTM recurrent layer to obtain the third feature sequence f i rnn , the third characteristic sequence f i rnn Input into the first fully connected layer FC1 to obtain the text content recognition result of the feature sequence; in the sequence dimension, the second feature sequence f i ' cnn and the third characteristic sequence f i rnn The average value is calculated, and then the average values ​​are added to obtain a fused feature sequence, and the fused feature sequence is input into the second fully connected layer FC2 to obtain a text direction recognition result of the feature sequence.

[0063] Specifically, the third feature sequence f i rnn Input into the text content recognition fully connected layer to obtain the probability pt of the sequence on each character i , combined with its corresponding true label, input into the CTC supervised text content recognition fully connected layer. In the sequence dimension, the second feature sequence f i ' cnn and the third characteristic sequence f i rnn Calculate the average value, then add the average values ​​to get the fused feature sequence, input the fused feature sequence into the text direction recognition fully connected layer to get the predicted text line text direction probability py i , supervise the fully connected layer of text direction recognition with cross entropy loss. The model is trained with the Adam optimizer.

[0064] Preferably, CTC (Connectionist Temporal Classification) is selected to supervise the text content recognition fully connected layer, and CTC can effectively handle the problem of misaligned sequences. It also includes:

[0065] Let the input text line image be I i , and its corresponding real text content label is t i The probability of each column of the feature sequence obtained by the text content recognition fully connected layer on each character is pt i , CTC expects the following L1 value to be minimized, where π i is the predicted character sequence, p(π i |I i ) is the predicted character sequence is π i The probability of CTC can quickly get from pt through the forward and backward algorithms i Calculate p(π i |I i ), the formula is as follows:

[0066]

[0067]

[0068] Among them, β is a mapping method defined in CTC, π i :β(π i ) = t i Represents the character sequence π i After being mapped by method β, it is equal to t i A collection of sequences.

[0069] Preferably, cross entropy loss is selected to supervise the text direction recognition fully connected layer, and further includes:

[0070] Let the input text line image be I i , the corresponding real text direction label is y i , the probability of the text direction predicted by the text direction recognition fully connected layer is py i , the cross entropy expects the following L2 value to be minimized, the formula is as follows:

[0071]

[0072] Wherein, N is the number of training text line images in the text line image training set.

[0073] As another embodiment of the present invention, a text line recognition device with arbitrary text direction is provided. Figure 2 As shown, the text line recognition device of arbitrary text direction includes:

[0074] An acquisition module is used to acquire a text line image training set;

[0075] A construction module for constructing an initial text line recognition model;

[0076] A training module, used for inputting the training text line images in the text line image training set into the initial text line recognition model for training, so as to obtain a trained text line recognition model;

[0077] The output module is used to input the target text line image to be recognized into the trained text line recognition model to output the text content recognition result and the text direction recognition result of the target text line image.

[0078] The present invention provides a text line recognition method for arbitrary text direction based on CRNN (Convolutional Recurrent Neural Network). On the model, a feature extraction network based on SE and Resnet is established, and then a two-layer bidirectional LSTM network is connected to further extract text sequence features based on convolution features, and a text recognition branch based on CTC and a text direction discrimination branch based on cross entropy are added in combination with convolution features and sequence features. The model is trained on a set of 10 million text line images, and the accurate text direction of the text line is obtained while recognizing the character sequence corresponding to the text line image, and the accurate character set order corresponding to the text line image is obtained according to the text direction. The model can be widely used in OCR scenarios where there are rotated images whose text direction is not necessarily in the positive direction.

[0079] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of the technical solution of the present invention.

Claims

1. A method for recognizing text lines in arbitrary text directions, characterized in that: The following steps are involved: Step S1: Obtain a text line image training set; wherein the text line image training set includes a plurality of horizontal training text line images and vertical training text line images with a text direction of 0°, wherein the vertical training text line image is rotated 90° counterclockwise to become a horizontal training text line image, and the corresponding text label order remains unchanged; the direction with a text direction of 0° and a direction rotated 90° counterclockwise is defined as the text positive direction of the training text line image, and during the training process, the training text line images with a positive text direction are randomly inverted, and the text labels of the inverted text line images are correspondingly changed to reverse order, and the text direction of the inverted text line images is defined as a negative direction; wherein the original text direction is After the training text line image with a rotation angle of 0° is inverted, the text direction becomes 180° counterclockwise; wherein, after the training text line image with the original text direction rotated 90° counterclockwise is inverted, the text direction becomes 270° counterclockwise; wherein, a unique number is defined for each character in the horizontal training text line image and the vertical training text line image, and during training, each character in the text label corresponding to the training text line image is converted into a corresponding character number as a sequence label corresponding to the training text line image; wherein, the training text line image with a positive text direction has a direction label encoded as 1; the training text line image with a negative text direction has a direction label encoded as 0; Step S2: constructing an initial text line recognition model; wherein the constructing of the initial text line recognition model further includes: The structure of the Resnet50 neural network is improved by using the SE module, and the improved Resnet50 neural network is used as the basic network, wherein the Resnet50 neural network has multiple convolution stages, and the multiple convolution stages include the first stage, the second stage, the third stage, the fourth stage and the fifth stage, and the SE module is added to the third stage and the fifth stage of the Resnet50 neural network; then the stride of the first convolution in the first stage, the second stage, the third stage, the fourth stage and the fifth stage is changed to (2, 2), (2, 2), (2, 1), (2, 1) and (2, 1) respectively; After the basic network, a circulation layer is constructed, and between the basic network and the circulation layer, an escape operation from a two-dimensional feature sequence to a one-dimensional feature sequence is defined to connect the basic network layer and the circulation layer; A first fully connected layer FC1 and a second fully connected layer FC2 are respectively connected after the circulation layer, wherein the first fully connected layer FC1 is a text content recognition fully connected layer, and the second fully connected layer FC2 is a text direction recognition fully connected layer; The step of inputting the training text line images in the text line image training set into the initial text line recognition model for training to obtain a trained text line recognition model further includes: Suppose the given text line image training set is denoted as Among them, I i represents the i-th text line image in the training set X, t i Represents the text line image I i The corresponding text content, y i Represents the text line image I i The corresponding text direction; The text line image I i Input into the initial text line recognition model, assuming that the text line image I i The first feature sequence obtained by the basic network based on the Resnet50 neural network is The first characteristic sequence After escaping, it becomes the second characteristic sequence The second feature sequence Input into the LSTM loop layer to get the third feature sequence The third characteristic sequence Input into the first fully connected layer FC1 to obtain the text content recognition result of the feature sequence; in the sequence dimension, the second feature sequence and the third characteristic sequence Calculate the average value, then add the average values ​​to obtain a fused feature sequence, input the fused feature sequence into the second fully connected layer FC2, and obtain the text direction recognition result of the feature sequence; Wherein, selecting CTC to supervise the text content recognition fully connected layer also includes: Let the input text line image be I i , and its corresponding real text content label is t i The probability of each column of the feature sequence obtained by the text content recognition fully connected layer on each character is pt i , CTC expects the following L1 value to be minimized, where π i is the predicted character sequence, p(π i |I i ) is the predicted character sequence is π i The probability of CTC can quickly get from pt through the forward and backward algorithms i Calculate p(π i |I i ), the formula is as follows: Among them, β is a mapping method defined in CTC, π i :β(π i ) = t i Represents the character sequence π i After being mapped by method β, it is equal to t i A collection of sequences; Wherein, selecting the cross entropy loss to supervise the text direction recognition fully connected layer also includes: Let the input text line image be I i , the corresponding real text direction label is y i , the probability of the text direction predicted by the text direction recognition fully connected layer is py i , the cross entropy expects the following L2 value to be minimized, the formula is as follows: Wherein, N is the number of training text line images in the text line image training set; Step S3: inputting the training text line images in the text line image training set into the initial text line recognition model for training, thereby obtaining a trained text line recognition model; Step S4: inputting the target text line image to be recognized into the trained text line recognition model to output the text content recognition result and the text direction recognition result of the target text line image.

2. A text line recognition device with arbitrary text direction, used to implement the text line recognition method with arbitrary text direction as claimed in claim 1, characterized in that: The text line recognition device of any text direction comprises: An acquisition module is used to acquire a text line image training set; A construction module for constructing an initial text line recognition model; A training module, used for inputting the training text line images in the text line image training set into the initial text line recognition model for training, so as to obtain a trained text line recognition model; The output module is used to input the target text line image to be recognized into the trained text line recognition model to output the text content recognition result and the text direction recognition result of the target text line image.

Images