A character recognition method and device based on deep learning and a storage medium

Abstract

The application discloses a kind of based on deep learning's character recognition method, device and storage medium, the present application is trained by training set to obtain detection classification model, character detection model and character recognition model, detection classification model includes large category classification model and the subdivision classification model corresponding to each large category, can be decoupled and used to improve applicability, the combination of both is conducive to improving accuracy rate;Character detection network takes deep convolutional neural network as main network and combines pixel aggregation network, so that high and low layer features have strong spatial and semantic information, and the character recognition network is trained according to CTC loss combined with Center Loss to improve training speed and recognition effect;The target classification is determined by classifying the object to be analyzed by the detection classification model, and the target character region is determined by detecting the object to be analyzed by the character detection network, and the target character region is recognized by the character recognition model, which is conducive to improving the accuracy of the character recognition result.

Description

Technical Field

[0001] This invention relates to the field of computers, and in particular to a method, apparatus and storage medium for character recognition based on deep learning. Background Technology

[0002] Traditional OCR solutions are based on a strong-layout methodology, mainly consisting of five steps: locating specific text elements, correcting skewed text, segmenting individual characters, recognizing individual characters, and finally performing semantic error correction. Traditional character segmentation often employs algorithms such as connected component analysis, projection analysis, character clustering, and template matching. However, issues like character blurring, breaks, adhesion, and corruption increase the difficulty of character segmentation in text recognition, resulting in poor robustness of existing solutions. Furthermore, since character recognition lacks contextual information, its accuracy is low. Summary of the Invention

[0003] In view of this, in order to solve at least one of the above-mentioned technical problems, the object of the present invention is to provide a deep learning-based text recognition method, apparatus and storage medium to improve accuracy.

[0004] This invention provides a deep learning-based text recognition method, comprising:

[0005] Obtain a training set; the training set includes documents and / or images;

[0006] The detection classification network, text detection network, and text recognition network are trained using the training set to obtain the detection classification model, text detection model, and text recognition model, respectively. The detection classification model includes a decoupled major category classification model and a sub-classification model corresponding to each major category. The text detection network uses a deep convolutional neural network as the backbone network and combines it with a pixel aggregation network. The text recognition network is trained using CTC loss combined with CenterLoss.

[0007] The process involves acquiring an object to be analyzed, classifying it using the detection and classification model to determine the target category, detecting the object using the text detection network to determine the target text region, and then recognizing the target text region using the text recognition model to obtain the text recognition result.

[0008] Furthermore, classifying the object to be analyzed using the detection and classification model to determine the target classification includes:

[0009] When the major category of the object to be analyzed is known, the target subdivision classification model corresponding to the known major category is determined, and the object to be analyzed is classified by the target subdivision classification model to obtain the target classification;

[0010] Otherwise, the object to be analyzed is classified into a first category using the major category classification model, and a target sub-classification model corresponding to the first category is determined. The object to be analyzed is then classified into a second category using the target sub-classification model to obtain the target category. The first category includes one of the following: contract, resume, form, card, and invoice.

[0011] Furthermore, the training of the major category classification model includes:

[0012] A major category classification network is constructed, consisting of a ResNet network and a Transformer network; the detection and classification network includes the major category classification network.

[0013] The class classification network is trained using the training set for an image-to-text pre-training task. During the training process, first difficult examples are added to the training set. The training set also includes samples added through manual intervention or augmentation. The first difficult examples include at least one of blurred, occluded, misaligned, and abnormal data.

[0014] When the training reaches the first termination condition, a major classification model is obtained.

[0015] Furthermore, the training of the subdivision classification model includes:

[0016] A subdivision classification network with a YOLO network structure is constructed; the regression of rotation angle in the subdivision classification network is added to the output of the YOLO network, and the detection classification network includes the subdivision classification network;

[0017] The subdivision classification network is trained with mixed precision using the training set and the PyTorch framework. During the training process, Warmup and CosineDecay are used as the learning rate, and a second difficult example is added to the training set. The second difficult example includes at least one of the following: blurred, occluded, misaligned, and abnormal data.

[0018] When the training reaches the second termination condition, the subdivision classification model is obtained.

[0019] Furthermore, the training set includes training data for major categories and subcategories, and the training of the text detection model includes:

[0020] A text detection network is constructed by using a deep convolutional neural network as the backbone network and combining it with a pixel aggregation network.

[0021] The text detection network is trained using the training set and the PyTorch framework. During training, a boundary distance loss with an outward expansion distance is added, and third-difficulty examples are added to the training set. The training set also includes samples added through manual intervention or augmentation operations. The third-difficulty examples include at least one of blurred, occluded, misaligned, and recognition-abnormal data. The outward expansion distance is the average boundary distance calculated based on the boundary pixels of the text regions identified by the text detection network during inference.

[0022] When the training reaches the third termination condition, the text detection model is obtained.

[0023] Furthermore, the training of the character recognition model includes:

[0024] A text recognition network with a network structure combining convolutional neural networks and recurrent neural networks was constructed.

[0025] The text recognition network is trained using the training set and the PyTorch framework; the training set includes samples added through manual intervention or augmentation operations.

[0026] During training, the CTC loss and the Center Loss are calculated. When the CTC loss and the Center Loss satisfy the fourth termination condition, the character recognition model is obtained.

[0027] Furthermore, the method also includes:

[0028] Obtain the error data corresponding to the text recognition result;

[0029] Add the erroneous data to the training set, and return to the step of training the detection classification network, the text detection network, and the text recognition network respectively using the training set, to obtain at least one of the new detection classification model, the new text detection model, and the new text recognition model.

[0030] This invention also provides a character recognition device, comprising:

[0031] An acquisition module is used to acquire a training set; the training set includes documents and / or images.

[0032] The training module is used to train the detection classification network, the text detection network, and the text recognition network respectively using the training set to obtain the detection classification model, the text detection model, and the text recognition model; wherein, the detection classification model includes a decoupled major category classification model and a sub-classification model corresponding to each major category, the text detection network uses a deep convolutional neural network as the backbone network and combines it with a pixel aggregation network, and the text recognition network is trained according to CTCloss combined with Center Loss.

[0033] The processing module is used to acquire the object to be analyzed, classify the object to be analyzed using the detection and classification model to determine the target category, detect the object to be analyzed using the text detection network to determine the target text region, and recognize the target text region using the text recognition model to obtain the text recognition result.

[0034] This invention also provides a character recognition device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set are loaded and executed by the processor to implement the method.

[0035] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method.

[0036] The beneficial effects of this invention are:

[0037] By acquiring a training set, the detection classification network, text detection network, and text recognition network are trained using the training set to obtain detection classification models, text detection models, and text recognition models. The detection classification model includes a major category classification model and sub-classification models corresponding to each major category. The major category classification model and the sub-classification models can be used decoupled to improve applicability. Each sub-classification model only needs to focus on a certain sub-category to reduce training difficulty, and the combination of the major category classification model and the sub-classification model helps to improve accuracy. The text detection network uses a deep convolutional neural network as the backbone network combined with a pixel aggregation network, which helps to ensure that both high-level and low-level features have strong spatial and semantic information. The text recognition network is trained using CTC loss combined with Center Loss, which can improve training speed and recognition effect. The object to be analyzed is acquired, classified by the detection classification model to determine the target category, detected by the text detection network to determine the target text region, and recognized by the text recognition model to obtain the text recognition result, which helps to improve the accuracy of the text recognition result.

[0038] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the steps of the deep learning-based text recognition method of the present invention.

[0040] Figure 2 This is a schematic diagram of text detection using a text detection network, which is a specific implementation of the present invention.

[0041] Figure 3 This is a schematic diagram illustrating the application framework of the deep learning-based character recognition method of this invention.

[0042] Figure 4 This is a schematic diagram illustrating the text recognition effect of financial documents according to a specific embodiment of the present invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0044] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0045] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0046] like Figure 1 As shown, this embodiment of the invention provides a deep learning-based text recognition method, including steps S100-S300:

[0047] S100, Obtain the training set.

[0048] Optionally, the training set includes documents and / or images. For example, images include, but are not limited to, images of contracts, resumes, forms, cards, receipts, and other content. Document content includes, but is not limited to, images containing contracts, resumes, forms, cards, receipts, and other content. It should be noted that the training set can be a single, unified set used for training different models, or it can include a first subset, a second subset, a third subset, and a fourth subset, used for training a broad category classification model, a sub-category classification model, a text detection model, and a text recognition model, respectively.

[0049] S200. The detection classification network, text detection network, and text recognition network are trained using the training set to obtain the detection classification model, text detection model, and text recognition model, respectively.

[0050] In this embodiment of the invention, the detection and classification model includes a decoupled major category classification model and a sub-classification model corresponding to each major category. The text detection network uses a deep convolutional neural network as the backbone network combined with a pixel aggregation network, and the text recognition network is trained using CTC loss combined with Center Loss. The detection and classification network includes a major category classification network and a sub-classification network. The major category classification results of the major category classification network include, but are not limited to, one of six types: contract, resume, form, card, invoice, and others.

[0051] Optionally, the training of the major category classification model in step S200 includes steps S201-S203:

[0052] S201. Construct a class classification network with a network structure that combines a ResNet network with a Transformer network.

[0053] Optionally, the ResNet network includes, but is not limited to, ResNet18, and the network structure is constructed as a class classification network combining a ResNet18 network and a Transformer network.

[0054] S202. Train the large-class classification network for image-to-text pre-training tasks using the training set, adding first-difficulty examples to the training set during the training process; the training set also includes samples added through manual intervention or enhancement operations, and the first-difficulty examples include at least one of the following: blurred, occluded, misaligned, and data with recognition anomalies. Optionally, enhancement operations include, but are not limited to, rotation, erosion, bending, tilting, erosion, superimposition, perspective transformation, stamping, and noise addition.

[0055] It's important to note that, compared to using a ResNet18+Linear network structure, in practice, document image classification differs from general image classification. If only image features are considered for differentiation, it can be difficult to distinguish between forms and tickets. Therefore, using a ResNet18 network combined with a Transformer network, and designing an image-to-text pre-training task (where text refers to all text within the image), specifically: the large-class classification network is trained on a training set for the image-to-text pre-training task. This allows the large-class classification network to extract both image features and text features. During training, the network parameters are fine-tuned for the document image classification task. For example, the training set (e.g., the first subset) can include the VL-CDIP dataset and a labeled dataset. The labeled dataset contains pre-annotated real-world information. The network is trained on the first subset for the pre-training task, and then the network parameters are fine-tuned on the labeled dataset for the image task.

[0056] It should be noted that the training set (e.g., the first subset) can be artificially intervened or augmented beforehand to increase the number of samples in the original collected data. Furthermore, during training, the first set of difficult examples is continuously and gradually added to the training set (e.g., the first subset), enriching the samples and improving the training effect of the large-class classification network, ensuring data validity and improving training efficiency. Optionally, augmentation operations include, but are not limited to, rotation, erosion, bending, tilting, erosion, superimposition, perspective transformation, stamping, and noise addition. The first set of difficult examples includes at least one of the following: blurred, occluded, misaligned, and data with recognition anomalies. Data with recognition anomalies can refer to data with poor recognition performance, such as data whose similarity to real information is lower than a similarity threshold, without specific limitations.

[0057] S203. When the training reaches the first termination condition, the major classification model is obtained.

[0058] Optionally, the large-class classification network can be continuously trained and iterated using the training set (such as the first subset), and the network parameters can be continuously adjusted until the first termination condition is met. At this point, the large-class classification network is the large-class classification model. It should be noted that the first termination condition can be that the number of training iterations reaches the first threshold, or that the first loss calculated by the first loss function is less than the first threshold, without specific limitations.

[0059] Optionally, the training of the subdivision classification model in step S200 includes steps S211-S213:

[0060] S211. Construct a subdivision classification network with the YOLO network structure; add the regression of rotation angle in the subdivision classification network to the output of the YOLO network.

[0061] It should be noted that images often contain multiple certificates, forms, or tickets, and often exhibit a certain rotation angle. Furthermore, the position of the target within the image is not fixed. Therefore, a model capable of detecting different types of targets and correcting the perspective is needed, such as a subdivision classification network.

[0062] It should be noted that, besides the main category of "others," each of the five main categories—contracts, resumes, forms, cards / certificates, and invoices—has corresponding subcategories. For example, the "invoices" category includes financial invoice types such as train tickets, taxi receipts, VAT invoices, general VAT invoices, special VAT invoices, flight itineraries, and general machine-printed invoices (subcategories); the "cards / certificates" category includes subcategories such as ID cards, driver's licenses, bank cards, passports, and Hong Kong and Macau travel permits. These are identified through corresponding subcategorization networks, and the structures of each subcategorization network are identical. In this embodiment of the invention, the combination of main category classification and subcategorization detection results in higher accuracy, as each subcategorization network only needs to focus on a specific subcategory. Furthermore, the main category classification and subcategorization models are decoupled and can be used independently depending on the scenario. If the main category is known, only the subcategorization detection model is needed, including but not limited to scenarios such as financial reimbursement.

[0063] Optionally, a subdivision classification network with a YOLO network structure, such as YOLOv5, is constructed, and the regression of rotation angle in the subdivision classification network is added to the output of the YOLO network. It should be noted that during experiments, YOLOv5 was used to detect the target's position, and a ResNet18+Linear network structure was used to regress the target's rotation angle. However, it was found that separating the rotation angle regression from the target detection increased memory usage and inference time. To solve this problem, in this embodiment of the invention, the rotation angle regression is added to the output of YOLOv5, and experiments have verified that its effect is comparable to separating the two methods, reducing memory usage and inference time while maintaining a certain level of performance.

[0064] S212. The subdivision classification network is trained with mixed precision using the training set and the PyTorch framework. The learning rate during the training process is Warmup and CosineDecay. Second difficult examples are added to the training set. The second difficult examples include at least one of the following: blurred, occluded, misaligned, and abnormal data.

[0065] In this embodiment of the invention, the subdivision classification network is trained with mixed precision using a training set (such as a second subset) and the PyTorch framework. The learning rate during training employs Warmup and CosineDecay. Simultaneously, during training, second-difficulty examples are continuously added to the training set (such as the second subset), enriching the samples in the training set (such as the second subset), improving the training effect of the subdivision classification network, ensuring data validity, and improving training efficiency. Similarly, the second-difficulty examples include at least one of the following: blurred, occluded, misaligned, and abnormally identified data.

[0066] S213. When the training reaches the second termination condition, the subdivision classification model is obtained.

[0067] Optionally, the subdivision classification network can be continuously trained and iterated using the training set (such as the second subset), and its network parameters can be continuously adjusted until the second termination condition is met. The subdivision classification network at this point is the subdivision classification model. It should be noted that the second termination condition can be that the number of training iterations reaches the second threshold, or that the second loss calculated by the second loss function is less than the second threshold; there is no specific limitation.

[0068] Optionally, the training set used for training the text detection model includes training data for major categories and subcategories, and the training of the text detection network in step S200 includes steps S221-S223:

[0069] S221. A text detection network is constructed by using a deep convolutional neural network as the backbone network and combining it with a pixel aggregation network.

[0070] Optionally, deep convolutional neural networks include, but are not limited to, ResNet such as ResNet18. A deep convolutional neural network is used as the backbone network and combined with a pixel aggregation network (PAN) to construct a text detection network. It should be noted that in text detection algorithms, PANet's post-processing requires pixel clustering, which is inefficient. Therefore, DBNet is chosen for text detection. DBNet is a segmentation-based text detection algorithm that proposes a differentiable binarization network. This network can obtain adaptive pixel thresholds through supervised learning, making the obtained thresholds robust. In this embodiment of the invention, a new DBNet is invented, using the pixel aggregation network PAN instead of the existing DBNet's FPN for feature fusion. Compared to FPN, the pixel aggregation network PAN has both top-down and bottom-up feature fusion, making both high- and low-level features possess strong spatial and semantic information.

[0071] S222. The text detection network is trained using the training set and the PyTorch framework. During training, the boundary distance loss of the outward expansion distance is added, and third-difficulty examples are added to the training set. The training set also includes samples added through manual intervention or augmentation operations. The third-difficulty examples include at least one of the following: blurred, occluded, misaligned, and recognition anomalies. The outward expansion distance is the average boundary distance calculated based on the boundary pixels of the text region when the text detection network recognizes the text region.

[0072] Optionally, the training set (such as the third subset) includes data labeled with major categories and real information on subcategories, as well as information such as labeled real detection values. The text detection network is trained using the training set (such as the third subset) and the PyTorch framework. For example, it can be pre-trained using an open-source dataset without labeled real detection values ​​and fine-tuned using data with real detection values ​​and real information.

[0073] It should be noted that the training set (e.g., the third subset) can be artificially modified or augmented beforehand by the original collected data (e.g., open-source datasets) to increase the number of samples in the training set (e.g., the third subset). Furthermore, during training, the third set of difficult examples is continuously and gradually added to the training set (e.g., the third subset), enriching the samples and improving the training effect of the text detection network, ensuring data validity, and increasing training efficiency. Similarly, the third set of difficult examples includes at least one of blurred, occluded, misaligned, and anomaly-identified data. Augmentation operations include, but are not limited to, rotation, erosion, bending, tilting, superimposition, perspective changes, stamping, and noise addition. It should be noted that the contribution of non-difficult example data to the model decreases over time; adding too much can actually reduce model training efficiency. This embodiment of the invention only adds the third set of difficult example data, which is beneficial for improving training effect and efficiency.

[0074] In this embodiment of the invention, during the training of the text detection network, it was found that the boundaries of the text regions were not precise enough. Short text regions often had excessively large boundaries, while long text regions had boundaries that were too small or even failed to contain the text. This is because during training, the text regions are shrunk inwards, and during inference, they are expanded outwards. The distance between the shrunk and expanded regions is calculated by multiplying the area-to-perimeter ratio of the text region by a fixed coefficient. Using the same coefficient for shrunk regions has little impact, but the shrunk text region used to calculate the expansion distance during inference is a prediction and not entirely accurate. Using the same coefficient again leads to the aforementioned problems. To solve this problem, the expansion distance is used as a learnable parameter of the text detection network. The network learns the distance from each pixel within the text region to the boundary. During training, a boundary distance loss is added to the expansion distance. During inference, the average boundary distance is calculated based on the pixels at the boundary of the text region as the expansion distance. For example, consider a text region with four coordinate points (2,3), (5,3), (5,5), and (2,5). This quadrilateral contains 10 pixels: besides the four text region coordinate points, there are six others: (3,3), (4,3), (5,4), (4,5), (3,5), and (2,4). In the inference result, the distances from these 10 pixels to the boundary are extracted, summed, and averaged as the extrapolation distance. Experiments have shown that the extrapolation distance obtained through learning is much more accurate than that calculated directly from a formula. Therefore, it can be understood that the extrapolation distance is the average boundary distance calculated by the text detection network based on the boundary pixels of the text region during inference.

[0075] S223. When the training reaches the third termination condition, the text detection model is obtained.

[0076] Optionally, the text detection network can be continuously trained and iterated using the training set (such as the third subset), and its network parameters can be continuously adjusted. With each iteration, the text detection performance will continuously improve. Figure 2 (a) shows the text detection results before iteration, and Figure 2 In Figure (b), the text detection results after iteration are shown. The invoice number and total amount are poorly detected due to occlusion. After optimization and iteration, the results are significantly improved until the third termination condition is met. At this point, the text detection network is the text detection model. It should be noted that the third termination condition can be that the number of training iterations reaches the third threshold, or that the third loss calculated by the third loss function considering the boundary distance loss of the outward expansion distance is less than the third threshold. No specific limitation is imposed.

[0077] It should be noted that the text detection model is trained using training data that includes major categories and subcategories. Therefore, the text detection model is a general-purpose text detection model, applicable to text detection in images / documents of all major categories and all subcategories. It is highly versatile and suitable for a wide range of application scenarios.

[0078] Optionally, the training of the character recognition network in step S200 includes steps S231-S233:

[0079] S231. Construct a text recognition network with a network structure that combines convolutional neural networks and recurrent neural networks.

[0080] Optionally, a text recognition network structure combining convolutional neural networks (CNNs) and recurrent neural networks (RNNs) can be constructed to form a sequence recognition algorithm. Compared to single-character recognition, this algorithm can associate contextual information, resulting in higher accuracy and efficiency, while also avoiding single-character segmentation. It should be noted that the CNNs include, but are not limited to, ResNet34. ResNet34 is used for feature extraction, and then an LSTM recurrent neural network is used for contextual feature extraction.

[0081] S232. Train the text recognition network using a training set and the PyTorch framework; the training set includes samples that have been added through manual intervention or augmentation.

[0082] Optionally, the training set (e.g., the fourth subset) may contain pre-labeled real labels for subcategories. The original collected data can be manually intervened or augmented beforehand to increase the number of samples in the training set (e.g., the fourth subset), making the samples richer, improving the training effect of the text recognition network, ensuring data validity, and improving training efficiency. Similarly, augmentation operations include, but are not limited to, rotation, erosion, and noise addition. In this embodiment of the invention, the text recognition network is trained using the training set (e.g., the fourth subset) and the PyTorch framework.

[0083] S233. Calculate CTC loss and Center Loss during training. When CTC loss and Center Loss satisfy the fourth termination condition, the character recognition model is obtained.

[0084] In this embodiment of the invention, CTC loss and Center Loss are calculated during training. CTC loss is flexible and has a fast decoding speed, which is beneficial for improving training efficiency. However, through training iterations, two problems were found to occur in certain situations. To achieve better recognition results, the following problems need to be further addressed:

[0085] 1) During training, the same character result can correspond to multiple paths, causing the convolutional neural network (CNN) to be unaware of the specific position of each character when extracting features, making it difficult to extract effective features and affecting the final accuracy. In contrast, the CNN+Seq2Seq+Attention approach, due to its attention mechanism, allows the CNN to accurately know the position of each character. Therefore, we attempted to add an Attention branch during training to guide the CNN's feature extraction learning, while removing the Attention branch during inference. This combines the advantages of both approaches without reducing the efficiency of the inference process. The overall architecture of the model during training, including the loss function, can be simply represented as follows: The model structure during inference, plus the CTC decoder, can be simply represented as CNN+RNN+CTC (decoder). The Attention branch is not needed during inference, so it is removed.

[0086] 2) The problem of easily misidentifying similar-looking characters: In this embodiment of the invention, center loss is added to reduce intra-class distance and increase inter-class distance. Experiments have shown that adding center loss can improve the recognition effect of similar-looking characters. Specifically, Center Loss uses the distance between features and feature centers, along with softmax loss, as the loss function, making the intra-class distance smaller.

[0087] Specifically, through training iterations and continuous adjustment of the network parameters of the text recognition network, the accuracy of the text recognition network will increase. When the fourth termination condition is met, the text recognition network at this point is considered the obtained text recognition model. The final text recognition model, after testing, achieves an accuracy of up to 99.8%, which is high. Optionally, the fourth termination condition may include a fourth threshold and a fifth threshold. The fourth termination condition is considered met when the CTC loss is less than the fourth threshold and the Center Loss is less than the fifth threshold.

[0088] S300. Obtain the object to be analyzed, classify the object to be analyzed using a detection and classification model to determine the target classification, detect the object to be analyzed using a text detection network to determine the target text region, and recognize the target text region using a text recognition model to obtain the text recognition result.

[0089] like Figure 3As shown, in this embodiment of the invention, the trained detection and classification model (including the decoupled major category classification model and the sub-classification model corresponding to each major category), text detection model, and text recognition model are used as the underlying general model capabilities to expand the application types and different industry demand scenarios. It supports industry demand scenarios including but not limited to risk control, financial reimbursement, medical care, transportation, etc., as well as recognition under various industry demand scenarios including but not limited to document and image classification, document and image detection, fixed document recognition, invoice / document recognition, etc. It supports a wide range of scenarios, with strong recognition applicability and recognition capabilities.

[0090] Optionally, the object to be analyzed can be an image or a document. A detection and classification model is used to classify the object to determine the target category. A text detection network is then used to detect the object to determine the target text regions. Finally, a text recognition model is used to recognize the target text regions, yielding the text recognition result. It should be noted that each sub-category has a corresponding template. After obtaining the text recognition result, it can be displayed using the template corresponding to the target category, resulting in a better display of the text recognition result. For example... Figure 4 The image shown is a schematic diagram illustrating the text recognition effect of financial documents, demonstrating excellent text recognition performance.

[0091] Optionally, step S300 involves classifying the object to be analyzed using a detection classification model to determine the target classification, including step S310:

[0092] Once the major category of the object to be analyzed is known, determine the target subdivision classification model corresponding to the known major category, and classify the object to be analyzed using the target subdivision classification model to obtain the target category.

[0093] Otherwise, the object to be analyzed is classified into a first category using the broad category classification model, and the target sub-classification model corresponding to the first category is determined. The object to be analyzed is then classified into a second category using the target sub-classification model to obtain the target category. The first category includes one of the following: contract, resume, form, card, and invoice.

[0094] Optionally, when the major category of the object to be analyzed is known, the user can input a specific sub-category. The system responds to the input sub-category by determining the target sub-category model corresponding to the known major category, and then classifies the object to be analyzed using the target sub-category model to obtain the target category. Conversely, when the major category of the object to be analyzed is unknown, the major category classification model is used to perform a first classification of the object, determining the target sub-category model corresponding to the first classification. The target sub-category model is then used to perform a second classification of the object to obtain the target category. In this embodiment of the invention, combining the major category classification model with the sub-category classification model improves the accuracy of the target classification.

[0095] Optionally, the deep learning-based character recognition method of this embodiment further includes steps S410-S420:

[0096] S410. Obtain the error data corresponding to the text recognition result.

[0097] Optionally, errors can be detected in the text recognition results through manual inspection or feedback from relevant users. In this case, the error data corresponding to the text recognition results can be obtained, such as the object to be analyzed corresponding to the erroneous text recognition result.

[0098] S420. Add the erroneous data to the training set, and return to the steps of training the detection classification network, the text detection network, and the text recognition network respectively using the training set, to obtain at least one of the new detection classification model, the new text detection model, and the new text recognition model.

[0099] Specifically, erroneous data is added to the training set, and the steps of training the detection classification network, text detection network, and text recognition network using the training set are repeated to obtain at least one of the new detection classification model, new text detection model, and new text recognition model. In this embodiment of the invention, by reverting erroneous data, the underlying models are automatically updated and iterated to effectively and continuously improve the recognition accuracy.

[0100] Compared to existing technologies, the deep learning-based text recognition method of this invention realizes a general deep learning text recognition solution based on detection and classification models, text detection models, and text recognition models. By extracting common detection and classification models, text detection models, and text recognition models, a foundational algorithm is constructed. Various application types and different industry demand scenarios are then extended on this foundation to form a general solution. The extraction of the capabilities of the underlying general algorithm model can reduce the cost of incubating new scenarios and implementing customized requirements, facilitate expansion, cater to more long-tail scenario needs, improve the utilization rate of hardware resources, shorten the overall process, improve accuracy, save manpower, effectively reduce manual workload, and achieve automation of the work.

[0101] This invention also provides a character recognition device, comprising:

[0102] The acquisition module is used to acquire the training set; the training set includes documents and / or images.

[0103] The training module is used to train the detection classification network, text detection network, and text recognition network respectively using the training set to obtain the detection classification model, text detection model, and text recognition model. Among them, the detection classification model includes a decoupled major category classification model and a sub-classification model corresponding to each major category. The text detection network uses a deep convolutional neural network as the backbone network and combines it with a pixel aggregation network. The text recognition network is trained according to CTC loss combined with CenterLoss.

[0104] The processing module is used to acquire the object to be analyzed, classify the object to be analyzed using a detection and classification model to determine the target category, detect the object to be analyzed using a text detection network to determine the target text region, and recognize the target text region using a text recognition model to obtain the text recognition result.

[0105] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments, so they will not be repeated here.

[0106] This invention also provides another character recognition device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the deep learning-based character recognition method of the aforementioned embodiments. Optionally, the character recognition device includes, but is not limited to, mobile phones, tablets, computers, and in-vehicle computers.

[0107] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments, so they will not be repeated here.

[0108] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the deep learning-based character recognition method of the aforementioned embodiments.

[0109] This invention also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the deep learning-based character recognition method of the aforementioned embodiments.

[0110] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0111] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Additionally, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0113] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A deep learning-based character recognition method, characterized in that, include: Obtain the training set; The training set includes documents and / or images; The detection classification network, text detection network, and text recognition network are trained using the training set to obtain detection classification models, text detection models, and text recognition models, respectively. The detection classification model includes a decoupled major category classification model and a sub-classification model corresponding to each major category. The text detection network uses a deep convolutional neural network as its backbone network combined with a pixel aggregation network. The text recognition network is trained using CTC loss combined with Center Loss. The detection classification network includes a sub-classification network of the YOLO network, and during training, the regression of rotation angle is added to the output of the YOLO network. The process involves acquiring an object to be analyzed, classifying the object using the detection and classification model to determine the target category, detecting the object using the text detection network to determine the target text region, and recognizing the target text region using the text recognition model to obtain the text recognition result. The step of classifying the object to be analyzed using the detection and classification model to determine the target classification includes: When the major category of the object to be analyzed is known, the target subdivision classification model corresponding to the known major category is determined, and the object to be analyzed is classified by the target subdivision classification model to obtain the target classification; Otherwise, the object to be analyzed is classified into a first category using the major category classification model, and a target sub-classification model corresponding to the first category is determined. The object to be analyzed is then classified into a second category using the target sub-classification model to obtain the target category. The first category includes one of the following: contract, resume, form, card, and invoice.

2. The deep learning-based character recognition method according to claim 1, characterized in that: The training of the major category classification model includes: A major category classification network is constructed, consisting of a ResNet network and a Transformer network; the detection and classification network includes the major category classification network. The class classification network is trained using the training set for an image-to-text pre-training task. During the training process, first difficult examples are added to the training set. The training set also includes samples added through manual intervention or augmentation. The first difficult examples include at least one of blurred, occluded, misaligned, and abnormal data. When the training reaches the first termination condition, a major classification model is obtained.

3. The deep learning-based character recognition method according to claim 1, characterized in that: The training of the subdivision classification model includes: Construct a subdivision classification network with a YOLO network structure; The subdivision classification network is trained with mixed precision using the training set and the PyTorch framework. During the training process, Warmup and CosineDecay are used as the learning rate, and a second difficult example is added to the training set. The second difficult example includes at least one of the following: blurred, occluded, misaligned, and abnormal data. When the training reaches the second termination condition, the subdivision classification model is obtained.

4. The deep learning-based character recognition method according to claim 1, characterized in that: The training set includes training data for major categories and subcategories. The training of the text detection model includes: A text detection network is constructed by using a deep convolutional neural network as the backbone network and combining it with a pixel aggregation network. The text detection network is trained using the training set and the PyTorch framework. During training, a boundary distance loss with an outward expansion distance is added, and third-difficulty examples are added to the training set. The training set also includes samples added through manual intervention or augmentation operations. The third-difficulty examples include at least one of blurred, occluded, misaligned, and recognition-abnormal data. The outward expansion distance is the average boundary distance calculated based on the boundary pixels of the text regions identified by the text detection network during inference. When the training reaches the third termination condition, the text detection model is obtained.

5. The deep learning-based character recognition method according to claim 1, characterized in that: The training of the character recognition model includes: A text recognition network with a network structure combining convolutional neural networks and recurrent neural networks was constructed. The text recognition network is trained using the training set and the PyTorch framework; the training set includes samples added through manual intervention or augmentation operations. During training, the CTC loss and the Center Loss are calculated. When the CTC loss and the Center Loss satisfy the fourth termination condition, the character recognition model is obtained.

6. The deep learning-based character recognition method according to claim 1, characterized in that: The method further includes: Obtain the error data corresponding to the text recognition result; Add the erroneous data to the training set, and return to the step of training the detection classification network, the text detection network, and the text recognition network respectively using the training set, to obtain at least one of the new detection classification model, the new text detection model, and the new text recognition model.

7. A character recognition device, characterized in that, include: An acquisition module is used to acquire a training set; the training set includes documents and / or images. The training module is used to train the detection classification network, the text detection network, and the text recognition network respectively using the training set to obtain the detection classification model, the text detection model, and the text recognition model. The detection classification model includes a decoupled major category classification model and a sub-classification model corresponding to each major category. The text detection network uses a deep convolutional neural network as its backbone network combined with a pixel aggregation network. The text recognition network is trained using CTC loss combined with Center Loss. The detection classification network includes a sub-classification network of the YOLO network, and during training, the regression of rotation angle is added to the output of the YOLO network. The processing module is used to acquire the object to be analyzed, classify the object to be analyzed using the detection and classification model to determine the target classification, detect the object to be analyzed using the text detection network to determine the target text region, and recognize the target text region using the text recognition model to obtain the text recognition result. The step of classifying the object to be analyzed using the detection and classification model to determine the target classification includes: When the major category of the object to be analyzed is known, the target subdivision classification model corresponding to the known major category is determined, and the object to be analyzed is classified by the target subdivision classification model to obtain the target classification; Otherwise, the object to be analyzed is classified into a first category using the major category classification model, and a target sub-classification model corresponding to the first category is determined. The object to be analyzed is then classified into a second category using the target sub-classification model to obtain the target category. The first category includes one of the following: contract, resume, form, card, and invoice.

8. A character recognition device, characterized in that: The character recognition device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method as described in any one of claims 1-6.

Images