Text recognition model training method, text recognition method and device

Abstract

The present application relates to the field of natural language processing, and particularly relates to a text recognition model training method, a text recognition method and device, wherein the text recognition model training method comprises: obtaining a data set; constructing an initial text recognition model; using a first subnetwork of the initial text recognition model to perform feature extraction on a text sample image to obtain a first feature mapping; using a second subnetwork to perform feature extraction on the first feature mapping to obtain a second feature mapping; using a third subnetwork to perform feature extraction on the first feature mapping to obtain a third feature mapping; using a fourth subnetwork to obtain a fusion feature mapping based on the second feature mapping and the third feature mapping; using a decoding subnetwork to decode the fusion feature mapping to obtain a prediction result; and training the initial text recognition model according to the prediction result and the data set to obtain a trained text recognition model. The present application helps to train the recognition model and improves the recognition performance.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and more particularly to a training method for a text recognition model, a text recognition method, and an apparatus. Background Technology

[0002] Natural scene text recognition refers to the process of identifying character sequences in images containing text. Generally speaking, for Chinese, one character is a Chinese character, and for English, one character is a letter. It is an extremely challenging subject. In addition to factors such as complex image backgrounds and varying lighting, the complexity of the recognition output space is also a major difficulty. Since text is composed of a variable number of letters, natural scene text recognition needs to identify sequences of varying lengths from images.

[0003] A common method for recognizing text in natural scenes is through deep learning models. However, deep learning models in this field often have poor feature extraction capabilities, resulting in low recognition accuracy. Summary of the Invention

[0004] According to one aspect of this disclosure, a method for training a text recognition model is provided, comprising:

[0005] Obtain a dataset, which includes at least easily identifiable text sample images with character annotation information;

[0006] An initial text recognition model is constructed, comprising a first sub-network, a second sub-network, a third sub-network, a fourth sub-network, and a decoding sub-network. The first sub-network is used to extract features from the text sample image, resulting in a first feature map with a height greater than 1. The second sub-network is used to extract features from the first feature map, resulting in a second feature map with both width and height equal to 1. The third sub-network is used to extract features from the first feature map, resulting in a third feature map with 1 channel. The fourth sub-network, based on the second and third feature maps, yields a fused feature map. The decoding sub-network decodes the fused feature map to obtain a prediction result.

[0007] Based on the prediction results and the dataset, the initial text recognition model is trained to obtain a trained text recognition model.

[0008] According to another aspect of this disclosure, a text recognition method is provided, comprising:

[0009] Acquire the image to be recognized;

[0010] Based on the image to be recognized and the trained text recognition model, the text recognition result of the image to be recognized is obtained; wherein, the trained text recognition model is obtained based on the training method of the above-mentioned text recognition model.

[0011] According to another aspect of this disclosure, a training apparatus for a text recognition model is provided, comprising:

[0012] An acquisition unit is used to acquire a dataset, which includes at least easily identifiable text sample images with character annotation information;

[0013] A construction unit is used to construct an initial text recognition model, which includes a first sub-network, a second sub-network, a third sub-network, a fourth sub-network, and a decoding sub-network. The first sub-network is used to extract features from the text sample image to obtain a first feature map with a height greater than 1. The second sub-network is used to extract features from the first feature map to obtain a second feature map with both width and height equal to 1. The third sub-network is used to extract features from the first feature map to obtain a third feature map with 1 channel. The fourth sub-network, based on the second and third feature maps, obtains a fused feature map. The decoding sub-network decodes the fused feature map to obtain a prediction result.

[0014] The training unit is used to train the initial text recognition model based on the prediction results and the dataset to obtain a trained text recognition model.

[0015] According to another aspect of this disclosure, a text recognition device is provided, comprising:

[0016] The acquisition unit is used to acquire the image to be recognized;

[0017] The recognition unit is used to obtain the text recognition result of the image to be recognized based on the image to be recognized and the trained text recognition model; wherein the trained text recognition model is obtained based on the training method of the above-mentioned text recognition model.

[0018] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0019] Processor; and

[0020] Stored program memory,

[0021] The program includes instructions that, when executed by the processor, cause the processor to perform the methods of this disclosure.

[0022] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the methods of this disclosure.

[0023] The embodiments of this application provide one or more technical solutions. The text recognition model includes a first sub-network, a second sub-network, a third sub-network, a fourth sub-network, and a decoding sub-network. The first sub-network extracts features from the text image to obtain a first feature map with a height greater than 1. Because the height is greater than 1, more basic shallow features can be retained. The second sub-network extracts features from the first feature map to obtain a second feature map with both width and height equal to 1. This compresses the width and height while retaining the channel dimension, implementing a channel attention mechanism. The third sub-network extracts features from the first feature map to obtain a third feature map with 1 channel. Because the channel becomes 1, a spatial attention mechanism is implemented. The fourth sub-network, based on the second and third feature maps, obtains a fused feature map, thereby enhancing the feature map by combining channel attention and spatial attention. The decoding sub-network decodes the fused feature map to obtain the prediction result. Because the feature extraction considers channel attention and spatial attention, and the feature map is enhanced by combining channel attention and spatial attention, the recognition accuracy of the text recognition model can be improved. Attached Figure Description

[0024] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0025] Figure 1 A flowchart illustrating a method for training a text recognition model according to an exemplary embodiment of the present disclosure is shown;

[0026] Figure 2 A flowchart of a text recognition method according to an exemplary embodiment of the present disclosure is shown;

[0027] Figure 3 A schematic block diagram of a training apparatus for a text recognition model according to an exemplary embodiment of the present disclosure is shown;

[0028] Figure 4 A schematic block diagram of a text recognition device according to an exemplary embodiment of the present disclosure is shown;

[0029] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0030] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0031] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0032] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0033] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0034] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0035] The relevant technologies of the embodiments disclosed herein are described below.

[0036] ResNet is a type of image classification network for natural scenes. It effectively solves the problem of performance degradation when the number of network layers increases, allowing the network to increase the number of layers (deeper) to extract more complex feature patterns. Its core is a structure called residual block. The main feature of the residual structure is cross-layer skip connections. That is, a residual block includes multiple convolutional layers. The output of the input after passing through the residual block is added to the input one channel by one point by one. It is equivalent to the input having two branches, one passing through the residual block and the other directly bypassing the residual block. Finally, the two branches are merged. ResNet has multiple structures with different numbers of convolutional layers, such as 18, 34, 50, 101 and 152. This disclosure adopts a ResNet with 18 convolutional layers, which can be called the ResNet18 network.

[0037] The common paradigm for deep learning-based text recognition technology is a CNN-based feature extractor, an RNN-based feature enhancement, modeling contextual information, and finally decoding using either Connectionist Temporal Classification (CTC) or Attention-based methods. CTC-based methods can decode in parallel, resulting in high speed, while Attention-based methods rely on the previous character for each subsequent character, essentially performing serial decoding, which, despite high accuracy, is slow. Additionally, there are schemes that directly apply Transformer structures and their variants to text recognition, but these are currently less common and relatively slow. For long texts, none of these methods have shown good results. CTC-based methods can forcibly align the output character with its corresponding vector, utilizing character positional information to some extent, while Attention-based methods primarily utilize textual content information, making it difficult to leverage character positional information.

[0038] The Transformer consists of two parts: an encoder and a decoder. The decoder comprises multiple stacked basic modules, which mainly consist of multi-head self-attention layers, skip connections, layer normalization, and feedforward neural networks. The decoder also comprises multiple basic modules, which differ from the first module in that they include two multi-head self-attention layers. The Transformer design not only greatly accelerates network training and inference time but also effectively improves the accuracy of various tasks. Originally designed for natural language understanding tasks, it is now widely used in computer vision tasks due to its excellent performance, achieving remarkable results in multiple tasks.

[0039] The solutions disclosed herein are described below with reference to the accompanying drawings. These solutions can be applied to electronic devices such as servers and / or clients, but the embodiments disclosed herein are not limited thereto.

[0040] This exemplary embodiment provides a text recognition method based on data augmentation, which can realize the recognition of mixed Chinese and English text images. The detection and recognition are implemented using a single neural network, which can save computing resources and improve recognition efficiency. In addition, using a single neural network can avoid the cumulative error between neural networks when using multiple neural networks, thereby improving recognition accuracy.

[0041] Figure 1 A flowchart illustrating a training method for a text recognition model according to an exemplary embodiment of the present disclosure is shown, such as... Figure 1 As shown, the text recognition method includes steps 11 to 18.

[0042] Step 11: Obtain the dataset, which should include at least easily identifiable text sample images with character annotation information.

[0043] In this embodiment, the text sample image can be a text image containing a single line of text.

[0044] Step 12: Construct the initial text recognition model, which includes the first sub-network, the second sub-network, the third sub-network, the fourth sub-network, and the decoding sub-network.

[0045] Step 13: Use the first sub-network to extract features from the text sample image to obtain a first feature map with a height greater than 1.

[0046] In step 13, since the height of the first feature map is greater than 1, more basic shallow features can be retained.

[0047] In this embodiment, the first sub-network may include various types of neural networks that extract features from text sample images to obtain feature maps with specified height, width, and channels. For example, the first sub-network may include a convolutional neural network.

[0048] Step 14: Use the second sub-network to extract features from the first feature map to obtain a second feature map with a width and height of 1.

[0049] In step 14, the feature extraction of the second sub-network is used to compress the width and height while preserving the channel dimension, thereby implementing the channel attention mechanism.

[0050] Step 15: Use the third sub-network to extract features from the first feature map to obtain the third feature map with channel 1.

[0051] In step 15, the third sub-network is used to extract features from the first feature map. Since the channel becomes 1, the spatial attention mechanism is implemented.

[0052] Step 16: Using the fourth sub-network based on the second and third feature maps, obtain the fused feature map.

[0053] In step 16, the fused feature map combines channel attention and spatial attention, thus enhancing the feature map.

[0054] Step 17: Use the decoding subnetwork to decode the fused feature map to obtain the prediction result.

[0055] Step 18: Based on the prediction results and the dataset, train the initial text recognition model to obtain the trained text recognition model.

[0056] The text recognition model in this embodiment includes a first sub-network, a second sub-network, a third sub-network, a fourth sub-network, and a decoding sub-network. The first sub-network extracts features from the text image, obtaining a first feature map with a height greater than 1. Because the height is greater than 1, more basic shallow features are retained. The second sub-network extracts features from the first feature map, obtaining a second feature map with both width and height equal to 1. This compresses the width and height while retaining the channel dimension, implementing a channel attention mechanism. The third sub-network extracts features from the first feature map, obtaining a third feature map with 1 channel. Since the channel becomes 1, a spatial attention mechanism is implemented. The fourth sub-network, based on the second and third feature maps, obtains a fused feature map, thus combining channel attention and spatial attention to enhance the feature map. The decoding sub-network decodes the fused feature map to obtain the prediction result. Because the feature extraction considers both channel attention and spatial attention, and combines channel attention and spatial attention to enhance the feature map, the recognition accuracy of the text recognition model can be improved.

[0057] As one implementation, the first sub-network is ResNet (also known as the ResNet network). ResNet is a type of natural scene image classification network that effectively solves the problem of model performance degradation as the number of network layers increases, allowing the network to increase the number of layers (deeper) to extract more complex feature patterns. ResNet includes structures called residual blocks, which are cross-layer skip connections. That is, a residual block contains multiple convolutional layers, and its input is added to the output of the residual block channel by channel and point by point. This is equivalent to the input having two branches: one passing through the residual block and the other directly and quickly bypassing the residual block. Finally, the two branches are merged. ResNet has multiple structures with different numbers of convolutional layers, such as 18, 34, 50, 101, and 152. It should be understood that the term "ResNet" includes both the usual ResNet and variations thereof.

[0058] In step 13, the first sub-network is used to extract features from the text sample image to obtain a first feature map with a height greater than 1. This includes: inputting the text sample image into ResNet to obtain the first feature map (b,c,h,w), where b represents the batch size of the text sample image, c represents the number of color channels of the first feature map, h represents the height of the first feature map (h>1), and w represents the width of the first feature map.

[0059] As an example, the first sub-network can be a ResNet18 network, which can include four Stage blocks. The output of each Stage is the input of the next Stage block. The features output by the four Stages are called the first feature, the second feature, the third feature, and the fourth feature, respectively. The first feature, the second feature, and the third feature are scaled to the same size as the fourth feature. Then, the scaled first feature, the scaled second feature, the scaled third feature, and the fourth feature are concatenated and superimposed to obtain the first feature map (b, c, h, w).

[0060] It should be noted that the size of h can be determined by adjusting the convolution stride. In order to retain more basic shallow features, h can be adjusted to a value greater than 1.

[0061] In one implementation, the second sub-network includes a 3x3 equal-width convolutional layer, a max-pooling layer, and a 1x1 convolutional layer. The process of using the second sub-network to extract features from the first feature map to obtain a second feature map with a width and height of 1 includes: performing a convolution operation on the first feature map using the 3x3 equal-width convolutional layer, and outputting the convolution result to the max-pooling layer; performing a pooling operation on the shutdown operation result using the max-pooling layer, and inputting the pooling result to the 1x1 convolutional layer; and processing the pooling result using the 1x1 convolutional layer to obtain the second feature map with a width and height of 1. The second feature map is (b, c, 1, 1), where b represents the batch size of the text sample images, and c represents the number of color channels in the first feature map. The second feature map has the same number of color channels as the first feature map, but its height and width are both 1.

[0062] In one implementation, in step 15, the first feature map is dimensionally transformed through the fully connected layer of the third sub-network to obtain a third feature map with 1 channel. This includes: performing fully connected processing on the dimensionally transformed first feature map through the fully connected layer, and then performing dimensional transformation on the result of the fully connected processing to obtain a third feature map with 1 channel.

[0063] As an example, the third sub-network uses a fully connected layer with c nodes to transform the first feature map group (b, c, h, w) output by the first sub-network into (b, h*w, c), while keeping the number of channels c unchanged. Then, another fully connected layer performs a full connection process, further transforming the result back to (b, 1, h, w) to facilitate subsequent feature map fusion. (b, 1, h, w) is then defined as the third feature map.

[0064] In one implementation, in step 16, the fourth sub-network obtains a fused feature map based on the second feature map and the third feature map, which may include the following steps 161-164.

[0065] Step 161: Expand the second feature map based on the scale of the first feature map.

[0066] Step 162: Multiply the expanded second feature map with the first feature map channel by channel and point by point to obtain the intermediate feature map.

[0067] It should be noted that step 162 actually fuses the expanded second feature map with the first feature map, and the resulting intermediate feature map group is equivalent to adding a channel attention mechanism, which can enhance the features extracted by the first sub-network.

[0068] Step 163: Expand the third feature map based on the channel dimension of the first feature map.

[0069] Step 164: Multiply the expanded third feature map with the intermediate feature map channel by channel and point by point to obtain the fused feature map.

[0070] It should be noted that step 164 actually fuses the expanded third feature map with the intermediate feature map. The resulting fused feature map is equivalent to adding a spatial attention mechanism, which can further enhance the features extracted by the first sub-network.

[0071] In one implementation, the decoding subnetwork includes a parallel decoding submodule. In another implementation, the decoding subnetwork includes a serial decoding submodule. In yet another implementation, the decoding subnetwork includes both a parallel decoding submodule and a serial decoding submodule.

[0072] As an example, the parallel decoding submodule may include an average pooling layer and a fully connected layer. The average pooling layer in the parallel decoding submodule compresses the height direction of the fused feature map, which is then processed by the fully connected layer. This step is equivalent to training using the CTC (Connectionist Temporal Classification) training method, and then performing recognition in parallel decoding to obtain the prediction result.

[0073] Furthermore, in step 17 above, the height of the fused feature map is compressed by the average pooling layer in the parallel decoding submodule, and the compressed fused feature map is processed by the fully connected layer to obtain the prediction result.

[0074] As an example, the serial decoding submodule may include an attention layer and a Transformer decoder.

[0075] Furthermore, in step 17 above, contextual information is obtained through the attention layer in the serial decoder submodule, and the softmax function is used as the target function to serially decode the fused feature map through the Transformer decoder.

[0076] Furthermore, contextual information is obtained through the attention layer in the serial decoder submodule. Here, absolute position information is used as the query when calculating the attention score, and then the softmax function is used as the target function. The Transformer decoder performs serial decoding to obtain the prediction result.

[0077] As an example, in step 17 above, the height of the fused feature map is compressed by the average pooling layer in the parallel decoding submodule, and the compressed fused feature map is processed by the fully connected layer to obtain the prediction result; and the context information is obtained by the attention layer in the serial decoder submodule, the softmax function is used as the objective function, and the fused feature map is serially decoded by the Transformer decoder.

[0078] In one implementation, the parallel decoding submodule and the serial decoding submodule can be connected in parallel after the fourth subnetwork. The inputs to both the parallel and serial decoding submodules are fused feature maps, and both are trained separately. In practice, the parallel and serial decoding submodules are concatenated.

[0079] The following section explains how to obtain and train the dataset.

[0080] Optionally, step 18 may include the following various options.

[0081] Option 1: The dataset is a commonly used dataset that has been manually labeled, and a training method without stages is adopted.

[0082] As one implementation method, text sample images are acquired and manually labeled. Based on the prediction results of the text sample images and the labeled ground truth, the parameters in the initial text recognition model are adjusted. After multiple iterations, when the number of iterations reaches a preset threshold or the initial text recognition model converges, training is stopped, and a trained text recognition model is obtained.

[0083] Option 2: The dataset is an augmented dataset, and a training method without stages is adopted.

[0084] The accuracy of a recognition model is inextricably linked to the quality of its training data. Data augmentation can effectively increase the diversity of training data, thereby improving the accuracy of the recognition model to some extent. It also reduces the labeling cost of training data. Therefore, data augmentation can be performed on training data.

[0085] In one implementation, existing text images can be collected as initial sample images, which may include straight text, slanted text, and curved text images, as well as regular blurred text images, photocopied text images, and other text images. These text images are single-line text images, and may include text of different lengths.

[0086] Furthermore, the received initial sample image undergoes a predetermined number of data augmentation operations. Multiple image data points obtained from one initial sample image through these augmentation operations are grouped together as a batch, and these multiple batches of image data are combined into an augmented dataset. This results in a dataset containing both the initial sample image and the sample images obtained from the data augmentation operations.

[0087] The preset number of times can be determined based on the batch size when training the model. For example, it can be 30 times or 50 times.

[0088] Alternatively, data augmentation operations may include, but are not limited to, the following:

[0089] Rigid transformation, elastic transformation, noise addition, color transformation, and random smearing, scratching, and masking;

[0090] Rigid transformations can include flipping, rotation, translation, and vertical truncation; elastic transformations can include various degrees of stretching and twisting.

[0091] In one feasible implementation, in this step, data augmentation is performed on the labeled sample images to obtain multiple labeled sample images with the same annotation information. In this way, the number of samples is increased without increasing the number of annotation operations, thus maximizing data performance.

[0092] In addition, for each image, all augmentation methods are applied once, and then the images obtained from the augmentation operations are arranged in order. This order is equivalent to temporal information. The images arranged in order can be considered as constituting a slowly changing video stream. Therefore, this method can be understood as a data augmentation method based on video stream.

[0093] When training a model using an augmented dataset, a batch of augmented image data can be input into the initial text recognition model. This batch shares the same ground truth value, but the prediction results obtained by the initial text recognition model will differ. The parameters of the initial text recognition model are then adjusted based on the prediction results and the ground truth values. Training the initial text recognition model batch by batch yields a trained text recognition model.

[0094] Option 3 uses an augmented dataset and a phased training approach.

[0095] In one feasible implementation, the augmented dataset can be obtained in accordance with the above-mentioned scheme two, and the training method can be as follows: steps 181-184.

[0096] Step 181: Train the initial text recognition model based on multiple batches of image data in the augmented dataset to obtain the first intermediate model.

[0097] Step 182: Train the first intermediate model based on the first dataset to obtain the second intermediate model.

[0098] Step 183: Train the model based on the unlabeled text sample images and the second intermediate model until the model converges, and obtain the trained initial text recognition model.

[0099] It should be noted that the unlabeled text sample images mentioned in this step can be the sample images collected and classified into the second part in step 11 above, or other sample images collected by the user. This disclosure does not limit them.

[0100] Optionally, step 183 above may include steps 1831-1833.

[0101] Step 1831: Input the unlabeled text sample image into the second intermediate model to obtain the recognition result corresponding to the unlabeled text sample image.

[0102] Step 1832: Adjust the recognition results corresponding to the unlabeled text sample images to obtain the sample ground truth values ​​corresponding to the unlabeled text sample images, and combine the unlabeled text sample images and the corresponding sample ground truth values ​​to form a third dataset.

[0103] Step 1833: Train the second intermediate model based on the third dataset.

[0104] Step 184: Remove the attention layer from the serial decoding submodule of the decoding subnetwork of the initial text recognition model, and connect the Transformer decoder in the serial decoding submodule after the parallel decoding submodule to obtain the trained text recognition model.

[0105] In one feasible implementation, the serial coding subnetwork and the parallel coding subnetwork can be trained simultaneously, and then the two can be used in series when needed, which can improve the recognition performance of the recognition model.

[0106] In this embodiment, various forms of data augmentation are applied to the same labeled text image. Multiple text images obtained after augmentation are then used as a batch to train the initial text recognition model. This approach obtains more comprehensive sample images without increasing the annotation workload. The model is then trained via video streaming, effectively expanding data diversity and improving recognition model accuracy to some extent. Simultaneously, it reduces the annotation cost of training data, effectively aiding model training and improving recognition performance. This achieves improved text recognition accuracy with limited labeled data and further reduces the time required for long text recognition. Furthermore, training the parallel decoding sub-network and the serial decoding sub-network separately and then using them in series further enhances the recognition model's performance.

[0107] Figure 2 A flowchart of a text recognition method according to an exemplary embodiment of the present disclosure is shown, such as... Figure 2 As shown, the text recognition method includes steps 21 to 22.

[0108] Step 21: Obtain the image to be recognized.

[0109] Step 22: Based on the image to be recognized and the trained text recognition model, obtain the text recognition result of the image to be recognized.

[0110] Optionally, the trained text recognition model includes a first sub-network, a second sub-network, a third sub-network, a fourth sub-network, and a decoding sub-network. The decoding sub-network includes a parallel decoding sub-module and a Transformer decoder sub-module, with the Transformer decoder sub-module connected in series after the parallel decoding sub-module.

[0111] Based on this, step 22 above, which obtains the text recognition result of the image to be recognized based on the image to be recognized and the trained text recognition model, can further include the following steps 221-226.

[0112] Step 221: Use the first sub-network to extract features from the text sample image to obtain a first feature map with a height greater than 1.

[0113] It should be noted that the specific processing method for this step can be referred to step 13 above, and will not be repeated here.

[0114] Step 222: Use the second sub-network to extract features from the first feature map to obtain a second feature map with a width and height of 1.

[0115] It should be noted that the specific processing method for this step can be referred to step 14 above, and will not be repeated here.

[0116] Step 223: Use the third sub-network to extract features from the first feature map to obtain the third feature map with channel 1.

[0117] It should be noted that the specific processing method for this step can be referred to step 15 above, and will not be repeated here.

[0118] Step 224: Using the fourth sub-network based on the second and third feature maps, obtain the fused feature map.

[0119] It should be noted that the specific processing method for this step can be referred to step 16 above, and will not be repeated here.

[0120] Step 225: Use the decoding submodule to decode the fused feature map to obtain the initial prediction result.

[0121] The initial prediction results may include the predicted characters and the confidence score for each character.

[0122] Step 226: Based on the Transformer decoder submodule and the initial prediction results, obtain the text recognition results of the image to be recognized.

[0123] Optionally, step 226 may include steps 2261-2262 below.

[0124] Step 2261: Compare the confidence score of each character in the initial prediction result with a preset threshold, and mask the characters corresponding to confidence scores below the preset threshold.

[0125] Step 2262: Input the masked characters into the Transformer decoder submodule for prediction, and determine the result as the text recognition result of the image to be recognized.

[0126] In this embodiment, various forms of data augmentation are applied to the same labeled text image. Multiple text images obtained after augmentation are then used as a batch to train the initial text recognition model. This approach obtains more comprehensive sample images without increasing the annotation workload. The model is then trained via video streaming, effectively expanding data diversity and improving recognition model accuracy to some extent. Simultaneously, it reduces the annotation cost of training data, effectively aiding model training and improving recognition performance. This achieves improved text recognition accuracy with limited labeled data and further reduces the time required for long text recognition. Furthermore, training the parallel decoding sub-network and the serial decoding sub-network separately and then using them in series further enhances the recognition model's performance.

[0127] This exemplary embodiment also provides a training apparatus for a text recognition model.

[0128] Figure 3 A schematic block diagram of a training apparatus for a text recognition model according to an exemplary embodiment of the present disclosure is shown, such as Figure 3 As shown, it includes an acquisition unit 310, a construction unit 320, and a training unit 330, wherein:

[0129] The acquisition unit 310 is used to acquire a dataset, which includes at least easily identifiable text sample images with character annotation information;

[0130] The construction unit 320 is used to construct an initial text recognition model, which includes a first sub-network, a second sub-network, a third sub-network, a fourth sub-network, and a decoding sub-network. The first sub-network is used to extract features from the text sample image to obtain a first feature map with a height greater than 1. The second sub-network is used to extract features from the first feature map to obtain a second feature map with both width and height equal to 1. The third sub-network is used to extract features from the first feature map to obtain a third feature map with 1 channel. The fourth sub-network, based on the second and third feature maps, obtains a fused feature map. The decoding sub-network decodes the fused feature map to obtain a prediction result.

[0131] Training unit 330 is used to train the initial text recognition model based on the prediction results and the dataset to obtain a trained text recognition model.

[0132] Optionally, the building unit 320 is further configured to:

[0133] The text sample image is input into the ResNet network to obtain the first feature map (b,c,h,w), where b represents the batch size of the text sample image, c represents the number of color channels of the first feature map, h represents the height of the first feature map (h>1), and w represents the width of the text sample image.

[0134] Optionally, the building unit 320 is further configured to:

[0135] The first feature map is extracted by using a 3*3 equal-width convolutional layer, a max-pooling layer, and a 1*1 convolutional layer of the second sub-network, respectively, to obtain a second feature map with a width and height of 1.

[0136] Optionally, the building unit 320 is further configured to:

[0137] The first feature map is dimensionally transformed through the fully connected layer of the third sub-network;

[0138] The first feature map, which undergoes dimensional transformation, is processed by a fully connected layer. The result of the fully connected processing is then subjected to dimensional transformation again to obtain the third feature map with 1 channel.

[0139] Optionally, the building unit 320 is further configured to:

[0140] The second feature map is expanded based on the scale of the first feature map;

[0141] The expanded second feature map is multiplied with the first feature map channel by channel and point by point to obtain the intermediate feature map;

[0142] The third feature map is expanded based on the channel dimension of the first feature map;

[0143] The expanded third feature map is multiplied with the intermediate feature map channel by channel and point by point to obtain the fused feature map.

[0144] Optionally, the building unit 320 is further configured to:

[0145] The height of the fused feature map is compressed by the average pooling layer in the parallel decoding submodule, and the compressed fused feature map is processed by the fully connected layer to obtain the prediction result.

[0146] And / or,

[0147] The decoding subnetwork includes a serial decoding submodule. The decoding of the fused feature map using the decoding subnetwork to obtain the prediction result includes:

[0148] Contextual information is obtained through the attention layer in the serial decoder submodule, and the softmax function is used as the target function. The fused feature map is then serially decoded through the Transformer decoder.

[0149] Optionally, the dataset includes a first dataset and a second dataset. The first dataset includes text sample images with character annotation information, and the second dataset includes multiple batches of image augmentation data obtained by performing data augmentation operations on the text sample images with character annotation information in the first dataset. The character annotation information of the image augmentation data belonging to the same batch is the same.

[0150] Optionally, training unit 330 is further used for:

[0151] Based on the prediction results and the second dataset, the initial text recognition model is trained to obtain a first intermediate model;

[0152] The first intermediate model is trained based on the first dataset to obtain the second intermediate model;

[0153] The model is trained using unlabeled text sample images and the second intermediate model until the model converges, resulting in a trained initial text recognition model.

[0154] Remove the attention layer from the serial decoding submodule of the decoding subnetwork of the initial text recognition model, and connect the Transformer decoder in the serial decoding submodule after the parallel decoding submodule to obtain the trained text recognition model.

[0155] Optionally, training unit 330 is further used for:

[0156] Input the unlabeled text sample image into the second intermediate model to obtain the recognition result corresponding to the unlabeled text sample image;

[0157] The recognition results corresponding to the unlabeled text sample images are adjusted to obtain the sample ground truth values ​​corresponding to the unlabeled text sample images. The unlabeled text sample images and their corresponding sample ground truth values ​​are then combined to form a third dataset.

[0158] The second intermediate model is trained based on the third dataset.

[0159] Optionally, the data augmentation operation includes, but is not limited to:

[0160] Rigid transformation, elastic transformation, noise addition, color transformation, and random smearing, scratching, and masking;

[0161] The rigid transformation includes flipping, rotation, translation, and vertical truncation; the elastic transformation includes various degrees of stretching and twisting.

[0162] In this embodiment, various forms of data augmentation are applied to the same labeled text image. Multiple text images obtained after augmentation are then used as a batch to train the initial text recognition model. This approach obtains more comprehensive sample images without increasing the annotation workload. The model is then trained via video streaming, effectively expanding data diversity and improving recognition model accuracy to some extent. Simultaneously, it reduces the annotation cost of training data, effectively aiding model training and improving recognition performance. This achieves improved text recognition accuracy with limited labeled data and further reduces the time required for long text recognition. Furthermore, training the parallel decoding sub-network and the serial decoding sub-network separately and then using them in series further enhances the recognition model's performance.

[0163] This disclosure also provides a text recognition device through exemplary embodiments.

[0164] Figure 4 A schematic block diagram of a text recognition device according to an exemplary embodiment of the present disclosure is shown, such as Figure 4 As shown, it includes: an acquisition unit 410 and an identification unit 420, wherein:

[0165] Acquisition unit 410 is used to acquire the image to be recognized;

[0166] The recognition unit 420 is used to obtain the text recognition result of the image to be recognized based on the image to be recognized and the trained text recognition model; wherein the trained text recognition model is obtained based on the method described in claims 1-10.

[0167] In this embodiment, various forms of data augmentation are applied to the same labeled text image. Multiple text images obtained after augmentation are then used as a batch to train the initial text recognition model. This approach obtains more comprehensive sample images without increasing the annotation workload. The model is then trained via video streaming, effectively expanding data diversity and improving recognition model accuracy to some extent. Simultaneously, it reduces the annotation cost of training data, effectively aiding model training and improving recognition performance. This achieves improved text recognition accuracy with limited labeled data and further reduces the time required for long text recognition. Furthermore, training the parallel decoding sub-network and the serial decoding sub-network separately and then using them in series further enhances the recognition model's performance.

[0168] Exemplary embodiments of this disclosure also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this disclosure.

[0169] Exemplary embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to embodiments of this disclosure.

[0170] Exemplary embodiments of this disclosure also provide a computer program product, including a computer program, wherein, when executed by a processor of a computer, the computer program is used to cause the computer to perform a method according to an embodiment of this disclosure.

[0171] refer to Figure 5The present invention describes a structural block diagram of an electronic device 500 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0172] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0173] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information to electronic device 500. Input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 507 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, disk and optical disk. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0174] The computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the text recognition model training method and the text recognition method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. In some embodiments, the computing unit 501 can be configured by any other suitable means (e.g., by means of firmware) to perform the text recognition model training method and the text recognition method.

[0175] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0176] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0177] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0178] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0179] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0180] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

Claims

1. A method for training a text recognition model, characterized in that, include: Acquire a dataset, which includes a first dataset and a second dataset. The first dataset includes text sample images with character annotation information, and the second dataset includes multiple batches of image augmentation data obtained by performing data augmentation operations on the text sample images with character annotation information in the first dataset. An initial text recognition model is constructed, comprising a first sub-network, a second sub-network, a third sub-network, a fourth sub-network, and a decoding sub-network. The first sub-network is used to extract features from the text sample image to obtain a first feature map with a height greater than 1. The second sub-network is used to extract features from the first feature map to obtain a second feature map with both width and height equal to 1. The third sub-network is used to extract features from the first feature map to obtain a third feature map with 1 channel. The fourth sub-network, based on the second and third feature maps, obtains a fused feature map. The fused feature map is decoded using the decoding sub-network to obtain the prediction result; Based on the prediction results and the second dataset, the initial text recognition model is trained to obtain a first intermediate model; The first intermediate model is trained based on the first dataset to obtain the second intermediate model; The model is trained using unlabeled text sample images and the second intermediate model until the model converges, resulting in a trained initial text recognition model. Remove the attention layer from the serial decoding submodule of the decoding subnetwork of the trained initial text recognition model, and connect the Transformer decoder in the serial decoding submodule after the parallel decoding submodule to obtain the trained text recognition model.

2. The training method for the text recognition model as described in claim 1, characterized in that, The step of extracting features from the text sample image using the first sub-network to obtain a first feature map with a height greater than 1 includes: The text sample images are input into ResNet to obtain the first feature map (b,c,h,w), where b represents the batch size of the text sample images, c represents the number of color channels of the first feature map, h represents the height of the first feature map (h>1), and w represents the width of the first feature map.

3. The training method for the text recognition model as described in claim 1, characterized in that, The step of extracting features from the first feature map using the second sub-network to obtain a second feature map with a width and height both equal to 1 includes: The first feature map is extracted by using a 3*3 equal-width convolutional layer, a max-pooling layer, and a 1*1 convolutional layer of the second sub-network, respectively, to obtain a second feature map with a width and height of 1.

4. The training method for the text recognition model as described in claim 1, characterized in that, The step of using the third sub-network to extract features from the first feature map to obtain a third feature map with channel 1 includes: The first feature map is dimensionally transformed through the fully connected layer of the third sub-network; The first feature map, which undergoes dimensional transformation, is processed by a fully connected layer. The result of the fully connected processing is then subjected to dimensional transformation again to obtain the third feature map with 1 channel.

5. The training method for the text recognition model as described in claim 1, characterized in that, The process of obtaining the fused feature map using the fourth sub-network based on the second and third feature maps includes: The second feature map is expanded based on the scale of the first feature map; The expanded second feature map is multiplied with the first feature map channel by channel and point by point to obtain the intermediate feature map; The third feature map is expanded based on the channel dimension of the first feature map; The expanded third feature map is multiplied with the intermediate feature map channel by channel and point by point to obtain the fused feature map.

6. The training method for the text recognition model as described in claim 1, characterized in that, The decoding subnetwork includes a parallel decoding submodule. The decoding subnetwork is used to decode the fused feature map to obtain the prediction result, which includes: The height of the fused feature map is compressed by the average pooling layer in the parallel decoding submodule, and the compressed fused feature map is processed by the fully connected layer to obtain the prediction result. The decoding subnetwork includes a serial decoding submodule. The decoding of the fused feature map using the decoding subnetwork to obtain the prediction result includes: Contextual information is obtained through the attention layer in the serial decoder submodule, and the softmax function is used as the target function. The fused feature map is then serially decoded through the Transformer decoder.

7. The training method for the text recognition model as described in claim 1, characterized in that, Image augmentation data belonging to the same batch have the same character annotation information.

8. The training method for the text recognition model as described in claim 1, characterized in that, The step of training the model based on unlabeled text sample images and the second intermediate model includes: Input the unlabeled text sample image into the second intermediate model to obtain the recognition result corresponding to the unlabeled text sample image; The recognition results corresponding to the unlabeled text sample images are adjusted to obtain the sample ground truth values ​​corresponding to the unlabeled text sample images. The unlabeled text sample images and their corresponding sample ground truth values ​​are then combined to form a third dataset. The second intermediate model is trained based on the third dataset.

9. The training method for the text recognition model as described in claim 1, characterized in that, The data augmentation operation includes: Rigid transformation, elastic transformation, noise addition, color transformation, and random smearing, scratching, and masking; The rigid transformation includes flipping, rotation, translation, and vertical truncation; the elastic transformation includes various degrees of stretching and twisting.

10. A text recognition method, characterized in that, include: Acquire the image to be recognized; Based on the image to be recognized and the trained text recognition model, the text recognition result of the image to be recognized is obtained; wherein the trained text recognition model is obtained based on the method described in claims 1-9.

11. A training device for a text recognition model, characterized in that, include: An acquisition unit is used to acquire a dataset, which includes a first dataset and a second dataset. The first dataset includes text sample images with character annotation information, and the second dataset includes multiple batches of image augmentation data obtained by performing data augmentation operations on the text sample images with character annotation information in the first dataset. A construction unit is used to construct an initial text recognition model, which includes a first sub-network, a second sub-network, a third sub-network, a fourth sub-network, and a decoding sub-network. The first sub-network is used to extract features from the text sample image to obtain a first feature map with a height greater than 1. The second sub-network is used to extract features from the first feature map to obtain a second feature map with both width and height equal to 1. The third sub-network is used to extract features from the first feature map to obtain a third feature map with 1 channel. The fourth sub-network, based on the second and third feature maps, obtains a fused feature map. The fused feature map is decoded using the decoding sub-network to obtain the prediction result; The training unit is used to train the initial text recognition model based on the prediction results and the second dataset to obtain a first intermediate model; to train the first intermediate model based on the first dataset to obtain a second intermediate model; to train the model based on unlabeled text sample images and the second intermediate model until the model converges to obtain a trained initial text recognition model; to remove the attention layer in the serial decoding submodule of the decoding subnetwork of the trained initial text recognition model, and to connect the Transformer decoder in the serial decoding submodule after the parallel decoding submodule to obtain a trained text recognition model.

12. A text recognition device, characterized in that, include: The acquisition unit is used to acquire the image to be recognized; The recognition unit is used to obtain the text recognition result of the image to be recognized based on the image to be recognized and the trained text recognition model; wherein the trained text recognition model is obtained based on the method described in claims 1-9.

13. An electronic device, comprising: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-9 or to perform the method according to claim 10.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9 or the method according to claim 10.

Images