Text recognition model training method and device, electronic equipment and medium

Abstract

The invention provides a text recognition model training method and device, electronic equipment and a medium, and relates to the technical field of image processing, the method comprises the steps that multiple pieces of text information are generated, each piece of text information comprises first language text information and second language text information, the reading direction of the first language text information is from left to right, and the reading direction of the second language text information is from left to right; the reading direction of the second language text information is from right to left; for each piece of text information, generating a corresponding text picture, the corresponding text picture comprising the text information, a label for each text character in the text information, and a label for a reading direction corresponding to each text character; determining the plurality of text pictures as a training set; according to the method, the basic model is trained through the training set, the text recognition model is obtained, when the obtained text recognition model faces the bidirectional text, reasoning can be conducted in combination with visual information and direction labeling, precise recognition of the bidirectional text is achieved, and the recognition accuracy in the bidirectional text scene is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a text recognition model training method and device, electronic equipment and medium. BACKGROUND

[0002] In the context of global digital information interaction, multi-lingual documents, interface subtitles, cross-border social platform posts and other media frequently present bidirectional text, such as Arabic and Hebrew (reading direction and writing direction from right to left, i.e. right-to-left, RTL) and English and numbers (reading direction and writing direction from left to right, i.e. left-to-right, LTR) mixed layout scenarios. When RTL language and LTR language are mixed, the reading direction is bidirectional, i.e. the above-mentioned bidirectional text is generated.

[0003] Currently, the traditional text recognition method (used to recognize text from a picture) is designed for text with a single writing direction or reading direction. When facing bidirectional text, character segmentation is prone to error, writing direction discrimination is not accurate, semantic coherence understanding and recognition are wrong, and the recognition accuracy is sharply reduced. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a text recognition model training method, device, electronic equipment and medium, and to improve the recognition accuracy of bidirectional text.

[0005] In a first aspect, the embodiments of the present application provide a text recognition model training method, which comprises:

[0006] Generating a plurality of text information, each of which comprises first language text information and second language text information, the reading direction of the first language text information being from left to right, and the reading direction of the second language text information being from right to left;

[0007] For each of the text information, a corresponding text picture is generated, which comprises the text information, a label for each text character in the text information, and a label for the reading direction corresponding to each of the text characters;

[0008] Determining a plurality of the text pictures as a training set;

[0009] Training a base model using the training set to obtain a text recognition model, the text recognition model being used to output target text in a to-be-recognized picture according to the to-be-recognized picture obtained.

[0010] In some embodiments, the training of the base model using the training set to obtain the text recognition model comprises:

[0011] performing multiple rounds of model parameter updating on the base model by using the training set, to obtain the text recognition model, wherein each round of the model parameter updating comprises updating model parameters of the base model according to a loss value obtained by the following steps:

[0012] performing cutting on the text picture to obtain a plurality of sub-pictures;

[0013] for each of the sub-pictures, converting the sub-picture into a first feature vector, the first feature vector being used to represent image information in the sub-picture;

[0014] determining a position encoding of the sub-picture, and adding the position encoding to the first feature vector, wherein the position encoding is used to represent a position of the sub-picture in the text picture;

[0015] taking the plurality of first feature vectors corresponding to the text picture as a first input tensor;

[0016] performing feature extraction on the first input tensor based on a multi-head self-attention mechanism, to obtain a first output tensor;

[0017] reshaping the first output tensor into an input feature map;

[0018] performing feature extraction on the input feature map in a left-to-right order by using a first branch of a convolutional layer in the base model, to obtain a first feature map;

[0019] performing feature extraction on the input feature map in a right-to-left order by using a second branch of the convolutional layer in the base model, to obtain a second feature map;

[0020] fusing the first feature map and the second feature map, to obtain a first fused feature map;

[0021] performing prediction on each text character in the text picture and a reading direction corresponding to each of the text characters according to the first fused feature map, to obtain a prediction result;

[0022] determining a loss value according to the prediction result and a label of the text picture based on a preset loss function.

[0023] In some embodiments, the first feature vector has a dimension number of M, and the performing feature extraction on the first input tensor based on the multi-head self-attention mechanism to obtain the first output tensor comprises:

[0024] For each of the first feature vectors in the first input tensor, the first feature vector is divided into N first feature sub-vectors, and each first feature sub-vector has a dimension number of M / N, where M, N and M / N are positive integers greater than 1;

[0025] For each of the first feature sub-vectors, a feature distance between the first feature sub-vector and other first feature sub-vectors with the same dimension in the first input tensor is determined, and an attention weight between the first feature sub-vector and the other first feature sub-vectors with the same dimension is determined according to the feature distance between the first feature sub-vector and the other first feature sub-vectors with the same dimension;

[0026] The other first feature sub-vectors with the same dimension and the attention weight are fused to obtain a second feature sub-vector;

[0027] The plurality of second feature sub-vectors are fused to obtain a second feature vector corresponding to the first feature vector;

[0028] The first output tensor is determined according to the plurality of second feature vectors.

[0029] In some embodiments, the predicting each text character in the text picture and a reading direction corresponding to each text character according to the first fused feature map to obtain a prediction result comprises:

[0030] A first weight is determined according to position information of each text character in the first fused feature map;

[0031] The first fused feature map and the first weight are fused to obtain a second fused feature map;

[0032] Each text character in the second fused feature map is classified by using the trained language classification model to determine a language attribution probability of each text character in the second fused feature map, and the language attribution probability is used to represent a probability that the text character belongs to the first language text information or a probability that the text character belongs to the second language text information;

[0033] A second weight is determined according to the language attribution probability of each text character in the second fused feature map;

[0034] The second fused feature map and the second weight are fused to obtain a third feature fused map;

[0035] A confidence score is output according to the third feature fused map, and the confidence score is used to represent a proportion of the first language text information and a proportion of the second language text information in the text picture;

[0036] According to the confidence score and the third feature fusion graph, a reading direction of each text character in the text picture and each text character is determined.

[0037] In some embodiments, the training of the base model using the training set comprises:

[0038] The base model is trained using the training set based on a preset learning rate decay strategy, and the text recognition model is obtained, wherein the learning rate decay strategy is:

[0039] In the process of i+1 round of model parameter update, if i is an integer multiple of a preset decay period, the learning rate of the i+1 round of model parameter update is adjusted to 0.5 times the learning rate of the i round of model parameter update, wherein i is a positive integer.

[0040] In some embodiments, after the training of the base model using the training set to obtain the text recognition model, the method further comprises:

[0041] Obtaining a to-be-recognized picture;

[0042] Using the text recognition model to output target text in the to-be-recognized picture according to the to-be-recognized text picture;

[0043] Using the trained semantic correction model to correct the target text to obtain a corrected text.

[0044] In some embodiments, the plurality of text pictures include a plurality of text backgrounds and a plurality of text fonts.

[0045] In a second aspect, the embodiments of the present application provide a training device of a text recognition model, the device comprising:

[0046] A first generation module is configured to generate a plurality of text information, each of the text information comprising first language text information and second language text information, the reading direction of the first language text information being from left to right, and the reading direction of the second language text information being from right to left;

[0047] A second generation module is configured to generate a corresponding text picture for each of the text information, the corresponding text picture comprising the text information, a label for each text character in the text information, and a label for a corresponding reading direction of each of the text characters;

[0048] A determination module is configured to determine a plurality of the text pictures as a training set;

[0049] The training module is configured to train the base model by using the training set to obtain a text recognition model, the text recognition model being configured to output target text in a to-be-recognized picture according to the to-be-recognized picture.

[0050] In a third aspect, an electronic device is provided, and the electronic device comprises:

[0051] a memory configured to store instructions; and

[0052] a processor configured to invoke the instructions from the memory and implement the method for training the text recognition model according to the first aspect of the present application when the instructions are executed.

[0053] In a fourth aspect, a machine-readable storage medium is provided, and the machine-readable storage medium stores instructions, which, when executed by a processor, cause the processor to implement the method for training the text recognition model according to the first aspect of the present application.

[0054] In the embodiments of the present application, the processor first generates a plurality of pieces of text information, each piece of text information containing first language text (reading direction from left to right) and second language text (from right to left) text information, and renders each piece of text information into a text picture, and at the same time, attaches a character label and a corresponding reading direction label to each text character in the text picture, and finally establishes these text pictures as a training set. Subsequently, the base model is trained by using the training set, and the obtained text recognition model can directly receive a to-be-recognized picture and output target text therein. Since the model is exposed to both character visual features and character direction attributes during the training process and completes joint learning in a unified network structure, the obtained recognition model can combine visual information and direction labels for reasoning when facing bidirectional text with LTR and RTL mixed arrangement, thereby achieving accurate recognition of mixed arrangement characters and their correct arrangement order and improving the recognition accuracy in the bidirectional text scenario. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a flowchart of the method for training the text recognition model provided by the embodiments of the present application;

[0056] Figure 2 is a structural diagram of the training device for the text recognition model provided by the embodiments of the present application;

[0057] Figure 3 is a structural diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0058] With reference to the drawings, the technical solutions in the embodiments of the present application will be clearly described below. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of them. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art are within the scope of protection of the present application.

[0059] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a category and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in an "or" relationship.

[0060] The training method, device, electronic device and medium of the text recognition model provided by the embodiments of the present application will be described in detail below in combination with the drawings and specific embodiments and application scenarios.

[0061] Please refer to Figure 1 , which is a flowchart of the text recognition model training method provided by the embodiments of the present application. As Figure 1 shown, the text recognition model training method includes the following steps S100 to S400.

[0062] Step S100: generating a plurality of text information, each text information including first language text information and second language text information, the reading direction of the first language text information being from left to right, and the reading direction of the second language text information being from right to left.

[0063] In this step, the processor first generates a plurality of text information in batches according to a pre-set rule, each text information being a bidirectional text composed of two parts: one part is the first language text information with a reading direction from left to right (LTR), such as English or numbers; the other part is the second language text information with a reading direction from right to left (RTL), such as Arabic or Hebrew. In order to ensure correct distinction between the two directions during subsequent rendering and recognition, zero-width explicit or implicit bidirectional control characters can also be embedded in the generated text information to simulate the mixed layout scene of real documents.

[0064] For example, a text information is taken as an example, where "Welcome12" is the first language information (LTR), and For the second language information (RTL), and insert a zero-width right-to-left isolate after "12" to ensure rendering The display order from right to left is maintained.

[0065] Specifically, the processor can generate the above-mentioned multiple text information using the Unicode bidirectional algorithm. The processor can first build a text generation rule library through the Unicode bidirectional algorithm, which integrates common left-to-right (LTR) language vocabulary and sentence templates (such as English, French), and right-to-left (RTL) language vocabulary and sentence templates (such as Arabic, Hebrew), and randomly combines these elements according to grammar and semantic logic when generating text, to ensure that the output paragraph contains rich words and phrases and can present multiple bidirectional mixed arrangement structures. For example, the processor can combine an English phrase and an Arabic phrase into the same sentence, and appropriately insert control characters between them to form a bidirectional example such as

[0066] On this basis, various directional formatting characters can be introduced during the generation process to fine-tune the actual rendering order of the text. Implicit directional characters are used to affect bidirectional sorting without changing the visible content; explicit embedding and rewriting characters can impose a mandatory direction setting on the character type within the embedded range; and explicit isolation characters block the bidirectional influence of characters outside the isolation range. In addition, each embedded or rewritten paragraph must end with a corresponding termination character to ensure clear control range.

[0067] Finally, the processor must also perform the basic display algorithm and bidirectional consistency processing to finally determine the visual order of the character sequence. The basic display algorithm first determines the initial bidirectional level and reordering level of each character according to the character type and matching formatting characters; then, through the rules of segmentation level and high-level protocol, the segmentation range and formatting markers are comprehensively processed. Finally, the bidirectional consistency test includes the unification of neutral boundaries, the application of explicit formatting characters, and the overall consistency verification, ensuring that such mixed arrangement texts can have the same and expected display effect under different rendering environments.

[0068] Through the joint operation of the above several stages, the processor can generate a raw text set that is both diverse and compliant with the Unicode bidirectional standard, preserving a rich variety of multilingual mixed arrangement examples while providing strong input guarantees for subsequent image rendering and model training.

[0069] Step S200: For each text information, generate a corresponding text picture, which includes the text information, the label for each text character in the text information, and the label for the reading direction corresponding to each text character.​​

[0070] Next, in step S200, the processor renders each piece of pure text information into a text picture available for model training, and simultaneously attaches two sets of labels to each character in the picture: one set records the specific content of the text character (character label), and the other set records the reading direction of the character (direction label).

[0071] In some embodiments, the plurality of text pictures includes a plurality of text backgrounds and a plurality of text fonts. When generating the text pictures, the processor can use an AIGN (Artificial Intelligence Generated Information) tool to generate and render, and select different fonts, font sizes, and backgrounds (paper texture, electronic screen pixels, poster patterns) or noise styles (tilt, perspective distortion, uneven lighting) in the plurality of text pictures to enrich visual interference, thereby improving the generalization ability of the model.

[0072] For example, the processor can draw "Welcome" in Arial font, "12" in Roboto Mono font, and on a 32x128 pixel grayscale image, respectively. And bind the final picture with the following label pair:

[0073] Step S300: Determine a plurality of text pictures as a training set.

[0074] In this step, the text pictures obtained by step S200 and their corresponding character and direction labels will be uniformly collected and shuffled to determine the training set (which includes the validation set and the test set, and also includes the collection of samples used for model parameter updating).

[0075] When constructing the training set using the plurality of pictures obtained by step S200, the processor can also filter out text pictures that are difficult to identify due to blurring, incorrect labeling, and perform data augmentation using image transformation algorithms (such as rotating the picture by -10° to 10°, scaling the picture by 0.8-1.2 times, translating the picture by ±5 pixels) or filter effects (adjusting the contrast of the picture to 0.7-1.3 times, sharpening / blurring the picture) to expand the diversity of text pictures.

[0076] Step S400: Train the base model using the training set to obtain a text recognition model, which is used to output the target text in the obtained to-be-recognized picture according to the to-be-recognized picture.

[0077] Finally, in step S400, the processor performs end-to-end training on the base model using the training set to obtain a final text recognition model. During the training process, the model iteratively reads each batch of text pictures containing character and direction labels, and can calculate the difference between the predicted results and the labels through cross-entropy loss, and can use a stochastic gradient descent algorithm with momentum to jointly optimize all parameters of the base model.

[0078] Exemplarily, taking one training sample (text picture) as an example, after the model inputs the text picture, it will extract visual features and direction information simultaneously in the forward calculation, and then output the prediction results and the LTR / RTL label of each character, and automatically adjusts the learning rate according to the accuracy on the validation set until convergence. After training is completed, the obtained text recognition model can be used to receive any picture containing bidirectional mixed text to be recognized and output the target text and its correct order.

[0079] Through the above steps S100-S400, the processor first generates multiple pieces of text information, each containing first language text (reading direction from left to right) and second language text (from right to left) information, and renders each piece of text information into a text picture, while attaching the character label and the corresponding reading direction label of each text character in the text picture. Finally, these text pictures are established as a training set. Subsequently, the training set is used to train the base model to obtain a text recognition model that can directly receive a picture to be recognized and output the target text. Since the model is exposed to both character visual features and character direction attributes during the training process, and jointly learns in a unified network structure, the obtained recognition model can combine visual information and direction labels for inference when facing LTR and RTL mixed text, thereby achieving accurate recognition of mixed characters and their correct arrangement order, and improving the recognition accuracy in the bidirectional text scenario.

[0080] In some embodiments, the base model is trained using the training set to obtain a text recognition model, including:

[0081] The base model is updated for multiple rounds of model parameters using the training set to obtain a text recognition model, wherein each round of model parameter update includes updating the model parameters of the base model according to the obtained loss value, and the loss value is obtained by the following steps:

[0082] For each text picture in the training set, the text picture is cut to obtain multiple sub-pictures;

[0083] For each sub-picture, the sub-picture is converted into a first feature vector, and the first feature vector is used to represent the image information in the sub-picture.​

[0084] determine a position code of the sub-picture, and add the position code to the first feature vector, wherein the position code is used to represent a position of the sub-picture in the text picture;

[0085] take the plurality of first feature vectors corresponding to the text picture as a first input tensor;

[0086] perform feature extraction on the first input tensor based on a multi-head self-attention mechanism to obtain a first output tensor;

[0087] reshape the first output tensor into an input feature map;

[0088] perform feature extraction on the input feature map in a left-to-right order using a first branch of a convolutional layer in the base model to obtain a first feature map;

[0089] perform feature extraction on the input feature map in a right-to-left order using a second branch of the convolutional layer in the base model to obtain a second feature map;

[0090] fuse the first feature map and the second feature map to obtain a first fused feature map;

[0091] predict each text character in the text picture and a reading direction corresponding to each text character according to the first fused feature map to obtain a prediction result;

[0092] determine a loss value based on a preset loss function and the prediction result and a label of the text picture.

[0093] In the present embodiment, during the training of the base model, the processor can first perform necessary preprocessing on the text pictures generated in the training set using the preprocessing layer in the base model, including grayscale, denoising, binarization, and cropping to a uniform size, then divide each row of text into a plurality of sub-pictures (patches) of a fixed size, and convert each sub-picture into a D-dimensional first feature vector through a linear mapping layer to represent the visual information of the sub-picture.

[0094] Next, the processor generates a corresponding position code according to the row and column coordinates of the sub-picture in the original text picture and adds it to the feature vector, thereby obtaining an encoded feature that carries both visual and spatial position information. By concatenating all the encoded features corresponding to the sub-pictures in order, a first input tensor with a shape of (N, D) is obtained, which is the input of the Vision Transformer (ViT) module in the base model.

[0095] In the ViT module, the multi-head self-attention mechanism allows each encoded feature vector to focus on itself and interact with all other vectors, calculate attention weights and weighted sum of its corresponding value vector, and then output a first output tensor of the same shape. After reshaping the output tensor into a three-dimensional input feature map (C, H, W) according to the row x column structure before splitting, it can be sent to the Bi-CNN in the base model, i.e., the convolutional layer;

[0096] The Bi-CNN extracts local stroke and edge features in the first branch in the left-to-right (LTR) convolutional order, generating a first feature map; in the second branch, it mines symmetric or reverse arrangement information in the reverse right-to-left (RTL) convolutional order, generating a second feature map; the two are fused in a certain layer by channel concatenation or weighted addition, forming a first fused feature map that retains both the local details of bidirectional convolution and the global context.

[0097] Subsequently, the first fused feature map passes through a layer of fully connected networks (classification layer) in the base model to predict the joint probability distribution of each character and its corresponding reading direction in the row, outputting the final prediction result sequence (specifically, the text character with the maximum probability in the probability distribution of each character position and its corresponding reading direction can be selected as the text character and reading direction of the character position). The processor compares the predicted character + direction label with the true label in the training set, and calculates the loss value of this forward inference by a preset loss function such as cross-entropy, and finally updates all parameters of the base model using the loss value under the framework of stochastic gradient descent + momentum. After multiple iterations of the entire process, an end-to-end text recognition model with both visual feature extraction and direction discrimination capabilities is obtained.

[0098] For example, for a text picture containing mixed bidirectional text , after preprocessing, it is divided into 256 4x4 pixel sub-pictures. Each sub-picture is first mapped to a 768-dimensional feature vector, and the corresponding row and column position encoding is added to distinguish the "Hello" section from . The ViT module integrates cross-region information under multi-head self-attention, outputs a feature sequence of the same shape, and reshapes it into a (768, 8, 32) feature map. The first branch of the Bi-CNN identifies the stroke order of "H–e–l–l–o–", and the second branch captures the Arabic character pattern of in the reverse sliding window. After fusion, a feature map with bidirectional details is obtained, and the joint label of each character is output by the classification layer, such as Finally, the model calculates the loss based on the prediction and annotation and updates the parameters in reverse, gradually optimizing until it accurately reproduces the performance on the validation set.

[0099] In some embodiments, the number of dimensions of the first feature vector is M, and feature extraction is performed on the first input tensor based on a multi-head self-attention mechanism to obtain a first output tensor, including:

[0100] For each first eigenvector in the first input tensor, split the first eigenvector into N first eigensubvectors, where the number of dimensions of each first eigensubvector is M / N, where M, N, and M / N are all positive integers greater than 1;

[0101] For each first feature subvector, determining a feature distance between the first feature subvector and other first feature subvectors of the same dimension in the first input tensor, and determining an attention weight between the first feature subvector and the other first feature subvectors of the same dimension based on the feature distance between the first feature subvector and the other first feature subvectors of the same dimension;

[0102] The first feature sub-vector with the same other dimensions is fused with the attention weight to obtain the second feature sub-vector;

[0103] Fusing multiple second eigenvectors to obtain a second eigenvector corresponding to the first eigenvector;

[0104] A first output tensor is determined based on the plurality of second eigenvectors.

[0105] In this embodiment, the processor first represents the preprocessed and embedded position-encoded sub-image sequence as a first input tensor, where each sub-image corresponds to a first feature vector of dimension M. When performing multi-head self-attention feature extraction, each first feature vector of dimension M is equally divided into N first feature sub-vectors of dimension M / N, where N is the number of self-attention heads.

[0106] In each attention head, the “feature distance” (i.e., the similarity between the query and all keys) is calculated in sequence for all sub-vector sequences corresponding to a specific head - that is, the N sets corresponding to all positions of the same sub-vector index in the sequence, and these distances are converted into attention weights through softmax.

[0107] In each attention head, for each feature sub-vector (the kth first feature sub-vector in the first feature vector to which it belongs) in it, the processor will first take out the first feature sub-vectors (the kth first feature sub-vector in the other first feature vectors) in the first input tensor with the same dimension as it, and then calculate the feature distance between the first feature sub-vector and all the first feature sub-vectors in the set in turn, and then normalize these distance values into attention weights by the softmax function.

[0108] Subsequently, each attention weight is weighted and fused with the first feature sub-vector corresponding to it in the above set to obtain the second feature sub-vector of the head; after completing the parallel processing of the N sub-vector heads, all the second feature sub-vectors are spliced back into a second feature vector with a dimension of M, and a linear mapping and a residual connection are applied to it, thereby outputting an output vector in the first output tensor corresponding to the original first feature vector. The entire process is repeated for all first feature vectors in the first input tensor, thereby obtaining a first output tensor with the same shape.

[0109] The processor can also send the obtained first output tensor into a feed-forward neural network (FFN) for further nonlinear feature transformation. Specifically, the FFN module first expands the input feature dimension from D to a higher dimension (usually 4D) through the first layer of linear mapping, then applies the Relu activation function to introduce nonlinear capability, and finally restores the expanded feature dimension back to D through the second layer of linear mapping.

[0110] In this way, the processor allows the model to capture multiple associations between sub-pictures in different subspaces through parallel multiple attention heads, which not only enhances the modeling ability of long-distance dependencies, but also allows parallel attention to multiple levels of features such as stroke, character structure, and semantic context; such "multi-perspective" fusion not only improves the recognition accuracy of the model for multi-lingual and bi-directional layout structures in mixed layout text, but also significantly enhances the expression ability and robustness of the network, so that it can still accurately locate and distinguish each character and its arrangement order when facing complex lighting, noise or deformation interference.

[0111] For example, assuming that the dimension of each first feature vector (f1, f2, f3) output by the ViT is M = 768, and the number of heads N = 12, then the dimension of the first feature sub-vector processed by each head is 768 / 12 = 64. Taking a first input tensor containing 3 first feature vectors as an example, the first input tensor (Z0) can be represented as:

[0112] Z0 = [f1, f2, f3] ∈ R 3×768

[0113] For the first attention head, the processor extracts the 1st-64th dimension of all first feature vectors and obtains the first feature sub-vector sequence {f1 (1) ,f2 (1) ,f3 (1)}. With f1 (1) With f2 (1) ,f3 (1) The similarity scores are calculated separately, and then the weights [α11, α12, α13] (α12, α13 are f2 (1) ,f3 (1) The corresponding weights of α11 can be padded with 0 to ensure that the number of dimensions of the first output tensor is consistent with the first input tensor), and finally [α11, α12, α13] is used to weight [f1, f2, f3] to generate the second feature sub-vector of the head for f1. After the same operation is performed on the 2nd to 12th heads, we can obtain Then concatenate them into the second eigenvector s1 for f1 with a dimension of 768. In this way, each f in the first input tensor i are mapped to the corresponding s i , which ultimately constitutes the first output tensor

[0114] In some embodiments, the above-mentioned ViT includes multiple encoding layers, each encoding layer performs feature extraction on the first input tensor based on a multi-head self-attention mechanism to obtain a first output tensor, and the first output tensor of the previous encoding layer serves as the first input tensor of the next encoding layer.

[0115] In some embodiments, predicting each text character in the text image and the reading direction corresponding to each text character based on the first fused feature map to obtain a prediction result includes:

[0116] Determine a first weight according to position information of each text character in the first fused feature map;

[0117] Fusing the first fused feature map with the first weight to obtain a second fused feature map;

[0118] Using the trained language classification model, classify each text character in the second fused feature map and determine the language attribution probability of each text character in the second fused feature map, where the language attribution probability is used to represent the probability that the text character belongs to the first language text information or the second language text information;

[0119] determining a second weight according to the language attribution probability of each text character in the second fused feature map;

[0120] The second fusion feature map is fused with the second weight to obtain a third feature fusion map;

[0121] A confidence score is output according to the third feature fusion map, and the confidence score is used to represent the proportion of the first language text information and the proportion of the second language text information in the text picture;

[0122] According to the confidence score and the third feature fusion map, each text character and the reading direction corresponding to each text character in the text picture are determined.

[0123] In this embodiment, when the bidirectional convolution branch of the Bi-CNN is fused into the first fusion feature map at a certain layer, the processor first calculates a first weight according to the spatial position information of each character in the feature map, so as to highlight the area where the character is located in the text line and suppress the noise generated by the background of the picture; after the first weight is multiplied with the first fusion feature map element by element, the second fusion feature map is obtained.

[0124] Then, the model calls the trained language classification model to perform language attribution prediction on the feature vector of each character position in the second fusion feature map, and outputs the probability distribution of the character belonging to the first language text information (LTR) or the second language text information (RTL), and then converts the set of language attribution probabilities into a second weight to fuse with the second fusion feature map, to obtain a third fusion feature map. According to the third fusion feature map, the processor can calculate an overall direction confidence score through a lightweight global pooling and a fully connected layer, to represent the proportion of LTR and RTL text in the text line.

[0125] Finally, the model integrates the local features of each character in the third fusion feature map and the global confidence to output the specific category of each character and the corresponding reading direction, to realize the prediction of the characters and their directions in the bidirectional mixed layout text; specifically, the model sends the local visual features of each character position in the third fusion feature map together with the global confidence of the entire line to the classifier in the model, to obtain a probability distribution of “character + direction”. Then, for each character position, only the item with the highest probability needs to be selected from the distribution of the position, to determine the final recognized character and the reading direction of the position. For example, assuming that the model determines the joint probability distribution of the third character position (Arabic letter ) of this text picture as: 0.92, (“n”, LTR): 0.05, (“o”, LTR): 0.03, the model selects the item with the highest probability as the recognition result of the third character. The other positions are the same, and the maximum probability item is taken from their respective distributions to complete the prediction of the characters and directions in the entire line. ​

[0126] The first weight imposes spatial attention on the model training, and the second weight imposes semantic attention on the model training. The spatial attention can enable the model to automatically "focus" on positions with dense text strokes and obvious character edges on the fused feature map, thereby suppressing background noise and irrelevant regions and improving the accuracy of character positioning and recognition. The semantic attention introduces language prior, and weights the language attribution probability of each character position, helping the model to quickly distinguish English and Arabic scripts and other different scripts according to the context when the visual features are ambiguous or similar, thereby greatly enhancing the ability to distinguish character categories and directions in mixed text. The synergistic effect of the two not only improves the recognition accuracy, but also makes the model more robust under various light, noise or deformation interference.

[0127] Those skilled in the art can understand that the first fused feature map, the second fused feature map and the third fused feature map are also tensors, and the first output tensor output by the ViT is a two-dimensional tensor, and the first fused feature map, the second fused feature map and the third fused feature map are all three-dimensional tensors. All operations in the Bi-CNN are completed on the three-dimensional tensors. The first weight can be a single-channel weight map that tells the model "which pixel position is more important". The processor can use broadcast multiplication to expand the weight map of the first weight, and then perform element-level multiplication with the three-dimensional tensor of the first fused feature map to realize fusion. The second weight can be a set of scalar weights, which is also mapped to a tensor and then multiplied with the three-dimensional tensor of the second fused feature map to realize fusion.

[0128] In some embodiments, the base model is subjected to multiple rounds of model parameter updating using the training set to obtain the text recognition model, including:

[0129] Based on the preset learning rate decay strategy, the base model is subjected to multiple rounds of model parameter updating using the training set to obtain the text recognition model, wherein the learning rate decay strategy is:

[0130] In the process of the i+1th round of model parameter updating, if i is an integer multiple of the preset decay period, the learning rate of the i+1th round of model parameter updating is adjusted to 0.5 times the learning rate of the ith round of model parameter updating, wherein i is a positive integer.

[0131] In this embodiment, in the model training process, the model uses the stochastic gradient descent algorithm with momentum to update all network parameters after each forward inference and loss calculation, and automatically adjusts the learning rate according to the preset decay period during the training process, so as to ensure that the model can quickly converge in the early stage of training and can be finely tuned with a smaller step size in the later stage.

[0132] Specifically, when the training is performed to the i-th round, if i is exactly an integer multiple of the preset decay period P, the learning rate currently used will be directly reduced to half of the last round learning rate when updated in the i+1-th round. For example, when the preset decay period P=10, the initial learning rate is set to 0.01, then in the 1-10th iteration, the learning rate is always updated with 0.01; to the 11th iteration, since 10 is an integer multiple of P, the learning rate will be automatically adjusted to 0.005; then the 21st round is again decayed to 0.0025, and so on, until the performance of the model on the validation set reaches saturation or the training round reaches the upper limit, that is, during the training process, the processor monitors the performance indicators of the model on the validation set in real time, such as recognition accuracy, recall rate, etc., when the performance of the model on the validation data set no longer improves or reaches the preset upper limit of the update round, the training is stopped, and the text recognition model is obtained.

[0133] In some embodiments, after training the base model with the training set to obtain the text recognition model, the method further comprises:

[0134] obtaining a to-be-recognized picture;

[0135] outputting a target text in the to-be-recognized picture according to the to-be-recognized text picture by using the text recognition model;

[0136] correcting the target text by using the trained semantic correction model to obtain a corrected text.

[0137] In the present embodiment, after obtaining the text recognition model, the processor can perform the following operations on any new to-be-recognized picture: first, a to-be-recognized picture containing bidirectional mixed text is read and sent into the end-to-end text recognition model. The model will automatically complete the entire process from image preprocessing, segmentation and position coding, ViT global feature extraction, Bi-CNN bidirectional local feature extraction, spatial and semantic attention fusion, to multi-label classification and direction discrimination, and finally output a preliminary recognized target text sequence and the reading direction of each character. Then, in order to eliminate the slight spelling or order deviation caused by visual noise or direction discrimination error, the processor will also call the previously trained semantic correction model (based on massive text pre-training and learning the grammar rules of bidirectional text) to perform multi-lingual language model-based verification and correction on the output preliminary target text, to generate a semantic coherent and direction correct corrected text for downstream applications.

[0138] Please refer to Figure 2 FIG. 1 is a structural schematic diagram of a training device of a text recognition model provided by the present embodiment, the second aspect of the present embodiment provides a training device 10 of a text recognition model, the device 10 comprises:

[0139] The first generation module 11 is configured to generate a plurality of text information, each of the text information comprising first language text information and second language text information, the reading direction of the first language text information being from left to right, and the reading direction of the second language text information being from right to left.

[0140] The second generation module 12 is configured to generate, for each of the text information, a corresponding text picture, the corresponding text picture comprising the text information, a label for each text character in the text information, and a label for the reading direction corresponding to each text character.

[0141] The determination module 13 is configured to determine the plurality of text pictures as a training set.

[0142] The training module 14 is configured to train the base model by using the training set to obtain a text recognition model, the text recognition model being configured to output target text in a to-be-recognized picture according to the to-be-recognized picture.

[0143] The training device 10 of the text recognition model provided in the second aspect of the embodiments of the present application can implement each process achieved by the method embodiments and achieve the same beneficial effects. To avoid repetition, details are not described herein.

[0144] Please refer to Figure 3 FIG. 1 is a structural schematic diagram of an electronic device provided by the embodiments of the present application. The third aspect of the embodiments of the present application provides an electronic device 1000, which comprises a processor 1100 and a memory 1200. The memory 1200 stores machine executable instructions capable of being executed by the processor 1100. The processor 1100 can execute the machine executable instructions to implement the training method of the text recognition model described above.

[0145] The fourth aspect of the embodiments of the present application provides a machine readable storage medium, which stores instructions. When the instructions are executed by a processor, the processor implements the training method of the text recognition model described above.

[0146] In some embodiments, the embodiments of the present application further provide a computer program product comprising a computer program. When the computer program is executed by a processor, the computer program implements the training method of the text recognition model according to the above embodiments.

[0147] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0148] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0149] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0150] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0151] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carriers.

[0152] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed, or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0153] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

[0154] In addition, any combination of various embodiments of the present application can also be made, as long as it does not deviate from the idea of the present application, it should also be considered as disclosed by the present application.

Claims

1. A training method for a text recognition model, characterized in that: The method comprises: Generate a plurality of text messages, each of the text messages including first language text information and second language text information, the first language text information being read from left to right, and the second language text information being read from right to left; For each piece of text information, generating a corresponding text image, wherein the corresponding text image includes the text information, a label for each text character in the text information, and a label for a reading direction corresponding to each text character; Determine the plurality of text images as a training set; The basic model is trained using the training set to obtain a text recognition model, and the text recognition model is used to output the target text in the image to be recognized based on the acquired image to be recognized.

2. The method according to claim 1, characterized in that The method of training the basic model using the training set to obtain a text recognition model includes: Using the training set, performing multiple rounds of model parameter updates on the base model to obtain the text recognition model, wherein each round of model parameter updates includes updating the model parameters of the base model according to the obtained loss value, and the loss value is obtained by the following steps: For each of the text images in the training set, cutting the text image to obtain multiple sub-images; For each of the sub-pictures, convert the sub-picture into a first feature vector, where the first feature vector is used to represent image information in the sub-picture; Determining a position code of the sub-image, and adding the position code to the first feature vector, wherein the position code is used to represent the position of the sub-image in the text image; Taking the first plurality of feature vectors corresponding to the text image as a first input tensor; Based on a multi-head self-attention mechanism, feature extraction is performed on the first input tensor to obtain a first output tensor; Reshape the first output tensor into an input feature map; Using the first branch of the convolutional layer in the base model, extract features from the input feature map in order from left to right to obtain a first feature map; Using the second branch of the convolutional layer in the base model, extract features from the input feature map in order from right to left to obtain a second feature map; Fusing the first feature map with the second feature map to obtain a first fused feature map; Predicting each text character in the text image and the reading direction corresponding to each text character according to the first fused feature map to obtain a prediction result; Based on a preset loss function, a loss value is determined according to the prediction result and the annotation of the text image.

3. The method according to claim 2, characterized in that The first feature vector has a dimension of M. The multi-head self-attention mechanism is used to extract features from the first input tensor to obtain a first output tensor, including: For each first eigenvector in the first input tensor, divide the first eigenvector into N first eigenvectors, where the number of dimensions of each first eigenvector is M / N, where M, N, and M / N are all positive integers greater than 1; For each of the first feature sub-vectors, determining a feature distance between the first feature sub-vector and other first feature sub-vectors with the same dimension in the first input tensor, and determining an attention weight between the first feature sub-vector and the first feature sub-vectors with the same dimension according to the feature distance between the first feature sub-vector and the first feature sub-vectors with the same dimension; Fusing the first feature sub-vector with the same other dimensions with the attention weight to obtain a second feature sub-vector; fusing the plurality of second eigenvectors to obtain a second eigenvector corresponding to the first eigenvector; The first output tensor is determined according to the plurality of second eigenvectors.

4. The method according to claim 2, characterized in that The step of predicting each text character in the text image and the reading direction corresponding to each text character according to the first fused feature map to obtain a prediction result includes: Determine a first weight according to the position information of each of the text characters in the first fused feature map; Fusing the first fused feature map with the first weight to obtain a second fused feature map; Using the trained language classification model, classify each text character in the second fused feature graph, and determine a language attribution probability for each text character in the second fused feature graph, where the language attribution probability represents a probability that the text character belongs to the first language text information or the second language text information; determining a second weight according to the language attribution probability of each text character in the second fused feature map; Fusing the second fused feature map with the second weight to obtain a third feature fusion map; Outputting a confidence score according to the third feature fusion graph, the confidence score being used to represent a proportion of the first language text information and a proportion of the second language text information in the text image; Each text character in the text image and a reading direction corresponding to each text character are determined according to the confidence score and the third feature fusion map.

5. The method according to claim 2, characterized in that The method of using the training set to perform multiple rounds of model parameter updates on the basic model to obtain the text recognition model includes: Based on a preset learning rate decay strategy, the training set is used to perform multiple rounds of model parameter updates on the basic model to obtain the text recognition model, wherein the learning rate decay strategy is as follows: During the i+1 round of model parameter update, if i is an integer multiple of the preset decay period, the learning rate of the i+1 round of model parameter update is adjusted to 0.5 times the learning rate of the i-th round of model parameter update, where i is a positive integer.

6. The method according to claim 1, characterized in that After the basic model is trained using the training set to obtain a text recognition model, the method further includes: Get the image to be identified; Outputting target text in the image to be recognized based on the image of text to be recognized using the text recognition model; The trained semantic correction model is used to correct the target text to obtain a corrected text.

7. The method according to claim 1, characterized in that The multiple text images include multiple text backgrounds and multiple text fonts.

8. A training device for a text recognition model, characterized in that: The device comprises: A first generating module is configured to generate a plurality of text messages, each of the text messages including a first language text message and a second language text message, wherein the first language text message is read from left to right, and the second language text message is read from right to left; A second generating module is configured to generate a corresponding text image for each piece of text information, wherein the corresponding text image includes the text information, a label for each text character in the text information, and a label for a reading direction corresponding to each text character; A determination module, configured to determine the plurality of text images as a training set; The training module is used to train the basic model using the training set to obtain a text recognition model, and the text recognition model is used to output the target text in the image to be recognized based on the acquired image to be recognized.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the method for training a text recognition model according to any one of claims 1 to 7 is implemented.

10. A machine-readable storage medium, characterized in that The machine-readable storage medium stores instructions, which, when executed by a processor, enable the processor to implement the text recognition model training method according to any one of claims 1 to 7.

Images