Character recognition method and device based on deep learning and storage medium

Abstract

The invention discloses a character recognition method and device based on deep learning and a storage medium, training is carried out through a training set to obtain a detection classification model, a character detection model and a character recognition model, and the detection classification model comprises a large class classification model and a subdivision classification model corresponding to each large class. The applicability can be improved through decoupling use, and the accuracy can be improved through combination of the two; the character detection network takes a deep convolutional neural network as a trunk network and is combined with a pixel aggregation network, so that high-layer and low-layer features have strong spatial and semantic information, and the character recognition network performs training according to CTC loss and Center Loss to improve the training speed and the recognition effect; the to-be-analyzed object is classified through the detection classification model to determine the target classification, the to-be-analyzed object is detected through the character detection network to determine the target character area, and the target character area is recognized through the character recognition model, so that the accuracy of a character recognition result is improved.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a text recognition method, device and storage medium based on deep learning. Background Art

[0002] The traditional OCR solution is based on the strong layout methodology, which mainly includes five steps: specific element text positioning, followed by tilted text correction, then segmenting and recognizing single words, and finally semantic error correction. Traditional single word segmentation mostly uses algorithms such as connected domain analysis, projection analysis, character clustering and template matching. Due to the ambiguity, breakage, adhesion, and contamination of characters, the difficulty of single word segmentation in text recognition increases and the existing solutions have poor robustness. In addition, since there is no contextual information in single word recognition in text recognition, the accuracy is low. Summary of the invention

[0003] In view of this, in order to solve at least one of the above-mentioned technical problems, an object of the present invention is to provide a text recognition method, device and storage medium based on deep learning to improve accuracy.

[0004] The embodiment of the present invention provides a text recognition method based on deep learning, comprising:

[0005] Acquire a training set; the training set includes documents and / or images;

[0006] The detection classification network, the text detection network and the text recognition network are trained respectively by the training set to obtain the detection classification model, the text detection model and the text recognition model; wherein the detection classification model includes a decoupled large-category classification model and a subdivision classification model corresponding to each large category, the text detection network uses a deep convolutional neural network as the backbone network and is combined with a pixel aggregation network, and the text recognition network is trained according to CTC loss combined with CenterLoss;

[0007] Acquire the object to be analyzed, classify the object to be analyzed by the detection classification model, determine the target classification, detect the object to be analyzed by the text detection network, determine the target text area, recognize the target text area by the text recognition model, and obtain the text recognition result.

[0008] Further, the object to be analyzed is classified by the detection classification model to determine the target classification, including:

[0009] When the major category of the object to be analyzed is known, a target subdivision classification model corresponding to the known major category is determined, and the object to be analyzed is classified by the target subdivision classification model to obtain a target classification;

[0010] Otherwise, the object to be analyzed is first classified by the major classification model, the target segmentation classification model corresponding to the first classification is determined, and the object to be analyzed is second classified by the target segmentation classification model to obtain the target classification; the first classification includes one of contracts, resumes, forms, cards and bills.

[0011] Furthermore, the training of the major classification model includes:

[0012] Constructing a large-category classification network with a network structure of Resnet network combined with transformer network; the detection classification network includes the large-category classification network;

[0013] The large-category classification network is trained for an image-to-text pre-training task by using the training set, and first difficult example data is added to the training set during the training process; the training set further has samples added by manual intervention or enhancement operation, and the first difficult example data includes at least one of blurred, occluded, misaligned, and recognized abnormal data;

[0014] When the training reaches the first termination condition, a large-category classification model is obtained.

[0015] Furthermore, the training of the segmentation classification model includes:

[0016] Constructing a segmentation classification network whose network structure is a Yolo network; adding the regression of the rotation angle in the segmentation classification network to the output of the Yolo network, and the detection classification network includes the segmentation classification network;

[0017] The segmentation classification network is trained with mixed precision by using the training set and the pytorch framework, Warmup and CosineDecay are used as the learning rate in the training process, and second difficult example data is added to the training set; the second difficult example data includes at least one of blurred, occluded, misaligned and recognized abnormal data;

[0018] When the training reaches the second termination condition, a segmentation classification model is obtained.

[0019] Furthermore, the training set includes training data of major categories and subcategories, and the training of the text detection model includes:

[0020] A text detection network is constructed using a deep convolutional neural network as the backbone network and combined with a pixel aggregation network;

[0021] The text detection network is trained by using the training set and the pytorch framework, and the boundary distance loss of the outward expansion distance is added during the training process, and third difficult example data is added to the training set; the training set also has samples added by manual intervention or enhancement operation, and the third difficult example data includes at least one of blurred, occluded, misaligned and recognized abnormal data; the outward expansion distance is the average boundary distance calculated according to the boundary pixel points of the text area when the text detection network recognizes the text area during reasoning;

[0022] When the training reaches the third termination condition, the text detection model is obtained.

[0023] Furthermore, the training of the text recognition model includes:

[0024] Construct a text recognition network with a network structure of convolutional neural network combined with recurrent neural network;

[0025] The text recognition network is trained by using the training set and the pytorch framework; the training set has samples added by manual intervention or enhancement operation;

[0026] The CTC loss and the Center Loss are calculated during the training process, and when the CTC loss and the Center Loss meet the fourth termination condition, a text recognition model is obtained.

[0027] Furthermore, the method further comprises:

[0028] Obtaining error data corresponding to the text recognition result;

[0029] The erroneous data is added to the training set, and the step of training the detection classification network, the text detection network and the text recognition network respectively through the training set is returned to obtain at least one of a new detection classification model, a new text detection model and a new text recognition model.

[0030] The embodiment of the present invention further provides a text recognition device, comprising:

[0031] An acquisition module, used to acquire a training set; the training set includes documents and / or images;

[0032] A training module, used to train the detection classification network, the text detection network and the text recognition network respectively through the training set to obtain a detection classification model, a text detection model and a text recognition model; wherein the detection classification model includes a decoupled large-category classification model and a subdivision classification model corresponding to each large category, the text detection network uses a deep convolutional neural network as a backbone network and is combined with a pixel aggregation network, and the text recognition network is trained according to CTCloss combined with Center Loss;

[0033] The processing module is used to obtain the object to be analyzed, classify the object to be analyzed through the detection classification model, determine the target classification, detect the object to be analyzed through the text detection network, determine the target text area, and recognize the target text area through the text recognition model to obtain the text recognition result.

[0034] An embodiment of the present invention also provides a text recognition device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the method.

[0035] An embodiment of the present invention also provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the method.

[0036] The beneficial effects of the present invention are:

[0037] By obtaining a training set, the detection classification network, the text detection network and the text recognition network are trained respectively by the training set to obtain a detection classification model, a text detection model and a text recognition model. The detection classification model includes a large category classification model and a subdivision classification model corresponding to each large category. The large category classification model and the subdivision classification model can be decoupled for use to improve applicability. Each subdivision classification model only needs to focus on a certain subdivision category to reduce the difficulty of training, and the combination of the large category classification model and the subdivision classification model is conducive to improving the accuracy rate; the text detection network uses a deep convolutional neural network as a backbone network and combines it with a pixel aggregation network, which is conducive to making both high and low layer features have strong spatial and semantic information. The text recognition network is trained according to CTC loss combined with Center Loss, which can improve the training speed and recognition effect; the object to be analyzed is obtained, the object to be analyzed is classified by the detection classification model, the target classification is determined, and the object to be analyzed is detected by the text detection network to determine the target text area. The target text area is recognized by the text recognition model to obtain a text recognition result, which is conducive to improving the accuracy of the text recognition result.

[0038] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the steps of the text recognition method based on deep learning of the present invention;

[0040] Figure 2 A text detection schematic diagram of a text detection network specifically implementing the present invention;

[0041] Figure 3 A schematic diagram of an application framework for implementing a text recognition method based on deep learning in the present invention;

[0042] Figure 4 It is a schematic diagram of the text recognition effect of a financial document according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0044] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0045] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0046] like Figure 1 As shown, an embodiment of the present invention provides a text recognition method based on deep learning, including steps S100-S300:

[0047] S100: Obtain a training set.

[0048] Optionally, the training set includes documents and / or images, for example, images include but are not limited to images of contracts, resumes, forms, cards, tickets and other contents, and the contents in the documents include but are not limited to pictures of contracts, resumes, forms, cards, tickets and other contents. It should be noted that the training set can be a whole set for training different models, or the training set can include a first subset, a second subset, a third subset and a fourth subset, which are respectively used for training a large-category classification model, a detailed classification model, a text detection model and a text recognition model.

[0049] S200, respectively train the detection classification network, the text detection network and the text recognition network through the training set to obtain the detection classification model, the text detection model and the text recognition model.

[0050] In the embodiment of the present invention, the detection classification model includes a decoupled large-category classification model and a segmented classification model corresponding to each large category. The text detection network uses a deep convolutional neural network as the backbone network and combines it with a pixel aggregation network. The text recognition network is trained based on CTC loss combined with Center Loss. Among them, the detection classification network includes a large-category classification network and a segmented classification network. The large-category classification results of the large-category classification network include, but are not limited to, contracts, resumes, forms, cards, bills, and other six types.

[0051] Optionally, the training of the major category classification model in step S200 includes steps S201-S203:

[0052] S201. Construct a large-category classification network with a network structure of Resnet network combined with transformer network.

[0053] Optionally, the Resnet network includes but is not limited to Resnet18, and the network structure is constructed as a large-category classification network that combines the Resnet18 network with the transformer network.

[0054] S202, training the large-category classification network for the pre-training task of image to text through the training set, and adding the first difficult example data to the training set during the training process; the training set also has samples added through manual intervention or enhancement operation, and the first difficult example data includes at least one of blur, occlusion, dislocation and recognition abnormal data. Optionally, the enhancement operation includes but is not limited to rotation, corrosion, bending, tilting, corrosion, overlapping, perspective change, seal covering and noise addition.

[0055] It should be noted that, compared with the network structure of Resnet18+Linear, it is found that document images are different from general image classification in the actual use of the network structure of Resnet18+Linear. If only the image features are used to distinguish, sometimes some forms and bills are difficult to distinguish. Resnet18 network is combined with transformer network, and image-to-text pre-training tasks are designed. Text refers to all the text in the image. Specifically: the large-category classification network is trained for the image-to-text pre-training task through the training set, so that the large-category classification network can extract both image features and text features in the image, and then fine-tune the network parameters of the large-category classification network in the task of document image classification during the training process. For example, the training set (such as the first subset) for training the large-category classification network may include the VL-CDIP data set and the annotated data set. The annotated data set contains real information annotated in advance. The above pre-training task is trained through the first subset, and then the network parameters of the image task are fine-tuned on the annotated data set.

[0056] It should be noted that the original collected data in the training set (such as the first subset) can be artificially intervened or enhanced in advance to increase the samples in the training set (such as the first subset), and the first difficult example data can be gradually added to the training set (such as the first subset) during the training process, so that the samples of the training set (such as the first subset) are richer, the training effect of the large-category classification network is improved, the validity of the data is guaranteed, and the training efficiency is improved. Optionally, the enhancement operation includes but is not limited to rotation, corrosion, bending, tilting, corrosion, overlapping, perspective change, seal covering and noise addition, and the first difficult example data includes at least one of blurring, occlusion, misalignment and recognition abnormal data. Among them, the recognition abnormal data can refer to data with poor recognition effect, for example, the data obtained by recognition is compared with the real information, and the calculated similarity is lower than the similarity threshold, etc., without specific limitation.

[0057] S203: When the training reaches the first termination condition, a large-category classification model is obtained.

[0058] Optionally, the training set (such as the first subset) is continuously used to train the large-category classification network, and the network parameters are continuously adjusted until the first termination condition is reached. At this time, the large-category classification network is the large-category classification model. It should be noted that the first termination condition can be that the number of training iterations reaches a first number threshold, or the first loss calculated by the first loss function is less than the first threshold, without specific limitation.

[0059] Optionally, the training of the segmentation classification model in step S200 includes steps S211-S213:

[0060] S211. Construct a segmentation classification network whose network structure is a Yolo network; and add the regression of the rotation angle in the segmentation classification network to the output of the Yolo network.

[0061] It should be noted that images often contain multiple certificates, forms or receipts, and often have a certain rotation angle. In addition, the position of the target in the image is not fixed. Therefore, a model that can detect different types of targets and correct the perspective is needed, such as a segmentation classification network.

[0062] It should be noted that, in addition to the other categories, each of the five categories of contracts, resumes, forms, cards and certificates, and bills has a corresponding sub-classification (for example, the bill category includes financial bill types such as train tickets, taxi tickets, value-added tax coupons, general value-added tax invoices, special value-added tax invoices, flight itineraries, general machine-printed invoices, etc. (sub-classifications); the card category includes identity cards, driver's licenses, bank cards, passports, Hong Kong and Macao passes, etc., which are identified by corresponding sub-classification networks, and the structures of each sub-classification network are the same. In the embodiment of the present invention, the combination of major category classification and sub-classification detection makes the final accuracy higher, and each sub-classification network only needs to focus on a certain sub-category. At the same time, the two models of major category classification and sub-category classification are decoupled and can be used separately according to the scenario. If the major category is known, only the sub-category detection model is required, including but not limited to the financial reimbursement scenario.

[0063] Optionally, a subdivision classification network whose network structure is a Yolo network, such as Yolov5, is constructed, and the regression of the rotation angle in the subdivision classification network is added to the output of the Yolo network. It should be noted that during the experiment, Yolov5 was used to detect the position of the target and the network structure of Resnet18+Linear was used to regress the rotation angle of the target. However, it was found that the rotation angle regression was separated from the target detection, which would increase the video memory usage and reasoning time. In order to solve this problem, in an embodiment of the present invention, the regression of the rotation angle is added to the output of Yolov5, and it has been verified through experiments that its effect is equivalent to that of the separation, which reduces the video memory usage and reasoning time while ensuring a certain effect.

[0064] S212. Perform mixed precision training on the segmentation classification network through the training set and the pytorch framework. Warmup and CosineDecay are used as the learning rate during the training process, and the second difficult example data is added to the training set; the second difficult example data includes at least one of blurred, occluded, misaligned and abnormal recognition data.

[0065] In an embodiment of the present invention, the segmentation classification network is trained with mixed precision by using a training set (such as the second subset) and a pytorch framework, and the learning rate during the training process uses Warmup and CosineDecay. At the same time, during the training process, the second difficult example data is continuously added to the training set (such as the second subset), so that the samples of the training set (such as the second subset) are richer, the training effect of the segmentation classification network is improved, the validity of the data is ensured, and the training efficiency is improved. Similarly, the second difficult example data includes at least one of blurred, occluded, misaligned, and abnormally recognized data.

[0066] S213: When the training reaches the second termination condition, a segmentation classification model is obtained.

[0067] Optionally, the training set (such as the second subset) is continuously used to train the segmentation classification network, and its network parameters are continuously adjusted until the second termination condition is reached. At this time, the segmentation classification network is the segmentation classification model. It should be noted that the second termination condition can be that the number of training iterations reaches a second number threshold, or the second loss calculated by the second loss function is less than the second threshold, without specific limitation.

[0068] Optionally, the training set used for training the text detection model includes training data of major categories and subcategories. The training of the text detection network in step S200 includes steps S221-S223:

[0069] S221. Use deep convolutional neural network as the backbone network and combine it with pixel aggregation network to build a text detection network.

[0070] Optionally, the deep convolutional neural network includes but is not limited to resnet such as resnet18. The deep convolutional neural network is used as the backbone network and combined with the pixel aggregation network PAN to construct a text detection network. It should be noted that in the text detection algorithm, since the post-processing of PANet requires pixel clustering, which is inefficient, DBNet is selected for text detection. DBNet is a segmentation-based text detection algorithm that proposes a differentiable binary network. The network can obtain an adaptive pixel threshold through supervised learning, so that the obtained threshold is robust. In the embodiment of the present invention, a new DBNet is invented, which uses a pixel aggregation network PAN instead of the existing DBNet FPN for feature fusion. Compared with FPN, the pixel aggregation network PAN has both top-to-bottom feature fusion and bottom-to-top feature fusion, so that both high- and low-level features have strong spatial and semantic information.

[0071] S222. Train the text detection network using a training set and a pytorch framework, add a boundary distance loss of the outward expansion distance during the training process, and add third-difficulty data to the training set; the training set also has samples added through manual intervention or enhancement operations, and the third-difficulty data includes at least one of blurred, occluded, misaligned, and recognition abnormal data; the outward expansion distance is the average boundary distance calculated based on the boundary pixels of the text area when the text detection network recognizes the text area during reasoning.

[0072] Optionally, the training set (such as the third subset) includes data labeled with major categories and real information of detailed categories, as well as information labeled with real detection values, and the text detection network is trained through the training set (such as the third subset) and the pytorch framework, for example, pre-training is performed through an open source data set that is not labeled with real detection values ​​and other information, and network parameters are fine-tuned through data with real detection values ​​and real information.

[0073] It should be noted that the training set (such as the third subset) can be manually intervened or enhanced in advance on the original collected data (such as the open source data set) to increase the samples in the training set (such as the third subset), and the third difficult example data can be gradually added to the training set (such as the third subset) during the training process, so that the samples of the training set (such as the third subset) are richer, the training effect of the text detection network is improved, the validity of the data is guaranteed, and the training efficiency is improved; similarly, the third difficult example data includes at least one of blurred, blocked, dislocated and abnormal recognition data, and the enhancement operation includes but is not limited to rotation, corrosion, bending, tilting, corrosion, overlapping, perspective change, seal covering and noise addition. It should be noted that the increase of non-difficult example data contributes less and less to the model, and adding too much will reduce the model training efficiency. The embodiment of the present invention only adds the third difficult example data, which is conducive to improving the training effect and training efficiency.

[0074] In an embodiment of the present invention, during the training of the text detection network, it is found that the boundary of the text area (text area) is not accurate enough, the boundary of the short text area is often too large, and the boundary of the long text area is too small or even cannot cover the text. The reason is that the text area will be shrunk during training and expanded during reasoning. The distance between the shrinkage and expansion is calculated based on the area-to-perimeter ratio of the text area multiplied by a fixed coefficient. Using the same coefficient for shrinkage has little effect, but the shrunken text area used to calculate the expansion distance during reasoning is predicted and not completely accurate. At this time, the fixed coefficient is still used, which will lead to the above problems. In order to solve the above problems, the expansion distance is used as a learnable parameter of the text detection network, and the distance from each pixel in the text area (text area) to the boundary is learned. The boundary distance loss of the expansion distance is increased during training, and the average boundary distance is calculated as the expansion distance based on the pixel points at the boundary of the text area during reasoning. For example, there is a text area with 4 coordinate points (2,3), (5,3), (5,5), (2,5). At this time, there are 10 pixels on this quadrilateral. In addition to the 4 coordinate points of the text area, there are 6 more pixels (3,3), (4,3), (5,4), (4,5), (3,5), (2,4). In the inference result, the distances from these 10 pixels to the boundary are taken and the average value is added as the expansion distance. Experiments have shown that the expansion distance learned is much more accurate than the one directly calculated by the formula. Therefore, it can be understood that the expansion distance is the average boundary distance calculated based on the boundary pixels of the text area when the text detection network recognizes the text area during inference.

[0075] S223. When the training reaches the third termination condition, a text detection model is obtained.

[0076] Optionally, the text detection network is trained iteratively using the training set (such as the third subset), and its network parameters are continuously adjusted. With the iteration, the text detection effect is continuously improved, such as Figure 2 (a) is the text detection result before iteration, and Figure 2 (b) is the text detection result after iteration. The invoice number and the total amount have poor detection effect due to occlusion. After optimization iteration, there is a significant improvement until the third termination condition is reached. At this time, the text detection network is the text detection model. It should be noted that the third termination condition can be that the number of training iterations reaches the third number threshold, or the third loss calculated by the third loss function considering the boundary distance loss of the outer expansion distance is less than the third threshold, without specific limitation.

[0077] It should be noted that the text detection model is trained by using training data including major categories and subcategories. Therefore, the text detection model is a universal text detection model, which is suitable for detecting text in images / documents of all major categories and all subcategories. It is highly versatile and suitable for a wide range of application scenarios.

[0078] Optionally, the training of the character recognition network in step S200 includes steps S231-S233:

[0079] S231. Construct a text recognition network whose network structure is a convolutional neural network combined with a recurrent neural network.

[0080] Optionally, a network structure is constructed as a text recognition network of a convolutional neural network combined with a recurrent neural network to form an algorithm for sequence recognition. Compared with single-word recognition, it can associate context information, and the recognition accuracy and efficiency are higher, while also avoiding the operation of single-word segmentation. It should be noted that the convolutional neural network includes but is not limited to resnet34, which is used for feature extraction by resnet34, and then the recurrent neural network LSTM is used for context feature extraction.

[0081] S232. Train the text recognition network using a training set and a pytorch framework; the training set has samples added through manual intervention or enhancement operations.

[0082] Optionally, the training set (such as the fourth subset) may have real labels of pre-annotated subdivision classifications, and the original collected data may be artificially intervened or enhanced in advance to increase the samples in the training set (such as the fourth subset), so that the samples in the training set (such as the fourth subset) are richer, the training effect of the text recognition network is improved, the validity of the data is ensured, and the training efficiency is improved. Similarly, the enhancement operation includes but is not limited to rotation, corrosion, noise addition and other operations. In an embodiment of the present invention, the text recognition network is trained using a training set (such as the fourth subset) and a pytorch framework.

[0083] S233. Calculate CTC loss and Center Loss during the training process. When CTC loss and Center Loss meet the fourth termination condition, obtain a text recognition model.

[0084] In the embodiment of the present invention, CTC loss and Center Loss are calculated during the training process. CTC loss is very flexible and has a fast decoding speed, which is conducive to improving training efficiency. However, through training iterations, it is found that two problems may occur in some cases. In order to achieve better recognition results, the following problems need to be further solved:

[0085] 1) During the training process, the same text result can correspond to multiple paths, which causes the convolutional neural network to not know the specific position of each word when extracting features, making it difficult to extract effective features, affecting the final accuracy. However, due to the attention mechanism, the convolutional neural network can accurately know the position of each word in the CNN+Seq2Seq+Attention method. Therefore, we try to add the Attention branch during the training process to guide the learning of the convolutional neural network to extract features, and remove the Attention branch during the reasoning process. This combines the advantages of the two methods without reducing the efficiency of the reasoning process. The model structure during training plus the loss function, the overall architecture can be simply expressed as The model structure during inference plus the CTC decoder can be simply expressed as CNN+RNN+CTC(decoder). The Attention branch is not needed during inference, so it is removed.

[0086] 2) Problem of misrecognition of similar characters: In the embodiment of the present invention, center loss is added to reduce the intra-class distance and increase the inter-class distance. Experiments have shown that adding center loss can improve the recognition effect of similar characters. Center Loss uses the distance between the feature and the feature center and softmax loss as the loss function to make the intra-class distance smaller.

[0087] Specifically, through training iterations and continuously adjusting the network parameters of the text recognition network, the accuracy of the text recognition network will become higher and higher. When the fourth termination condition is met, the text recognition network at this time is the text recognition model. Among them, the accuracy of the text recognition model finally obtained after testing can reach 99.8%, which is high. Optionally, the fourth termination condition may include a fourth threshold and a fifth threshold. When CTC loss is less than the fourth threshold and Center Loss is less than the fifth threshold, the fourth termination condition is considered to be met.

[0088] S300, obtaining an object to be analyzed, classifying the object to be analyzed through a detection and classification model to determine a target classification, detecting the object to be analyzed through a text detection network to determine a target text area, recognizing the target text area through a text recognition model to obtain a text recognition result.

[0089] like Figure 3As shown, in an embodiment of the present invention, the detection classification model obtained after training (including the decoupled major category classification model and the subdivision classification model corresponding to each major category), the text detection model and the text recognition model are used as the underlying general model capabilities to expand the application types and different industry demand scenarios, supporting industry demand scenarios including but not limited to risk control, financial reimbursement, medical care, transportation, etc., and recognition under application types including but not limited to document image classification, document image detection, fixed document recognition, bill / receipt recognition, etc. in various industry demand scenarios, with rich supported scenarios, strong recognition applicability and recognition capabilities.

[0090] Optionally, the object to be analyzed can be an image or a document. The object to be analyzed is classified by the detection classification model to determine the target classification, and the object to be analyzed is detected by the text detection network to determine the target text area. The target text area is recognized by the text recognition model to obtain the text recognition result. It should be noted that each sub-classification has a corresponding template. After the text recognition result is obtained, the text recognition result can be displayed with the template corresponding to the target classification, so that the display effect of the text recognition result is better. Figure 4 As shown, it is a schematic diagram of the text recognition effect of financial documents, and it can be seen that it has a very good text recognition effect.

[0091] Optionally, in step S300, the object to be analyzed is classified by detecting the classification model to determine the target classification, including step S310:

[0092] When the major categories of the objects to be analyzed are known, the target segmentation classification model corresponding to the known major categories is determined, and the objects to be analyzed are classified by the target segmentation classification model to obtain the target classification.

[0093] Otherwise, the object to be analyzed is first classified through the major classification model, the target segmentation classification model corresponding to the first classification is determined, and the object to be analyzed is second classified through the target segmentation classification model to obtain the target classification; the first classification includes one of contracts, resumes, forms, cards and bills.

[0094] Optionally, when the major category of the object to be analyzed is known, the user can input a specific subdivision classification, and the system determines the target subdivision classification model corresponding to the known major category in response to the input subdivision classification, and then classifies the object to be analyzed through the target subdivision classification model to obtain the target classification. When the major category of the object to be analyzed is unknown, the major category classification model is used to perform a first classification on the object to be analyzed, and the target subdivision classification model corresponding to the first classification is determined, and the target subdivision classification model is used to perform a second classification on the object to be analyzed to obtain the target classification. In the embodiment of the present invention, the accuracy of the target classification is improved by combining the major category classification model with the subdivision classification model.

[0095] Optionally, the text recognition method based on deep learning in the embodiment of the present invention further includes steps S410-S420:

[0096] S410: Obtain error data corresponding to the text recognition result.

[0097] Optionally, errors may be found in the text recognition results through manual inspection or relevant user feedback, and error data corresponding to the text recognition results may be obtained, for example, the object to be analyzed corresponding to the erroneous text recognition result may be obtained as error data.

[0098] S420, adding the erroneous data to the training set, returning to the step of respectively training the detection classification network, the text detection network and the text recognition network through the training set, and obtaining at least one of a new detection classification model, a new text detection model and a new text recognition model.

[0099] Specifically, the error data is added to the training set, and the steps of training the detection classification network, the text detection network, and the text recognition network by the training set are returned to obtain at least one of a new detection classification model, a new text detection model, and a new text recognition model. In the embodiment of the present invention, the error data is reflowed to automatically update and iterate the underlying models, so as to effectively and continuously improve the recognition accuracy.

[0100] Compared with the prior art, the deep learning-based text recognition method of the embodiment of the present invention realizes a deep learning general text recognition solution based on detection and classification models, text detection models and text recognition models. By extracting common detection and classification models, text detection models and text recognition models, an underlying algorithm base is constructed, and various application types and different industry demand scenarios are expanded on the algorithm base to form a general solution. The extraction of the capabilities of the underlying general algorithm model can reduce the cost of incubating new scenarios and realizing customized needs, facilitate expansion, face more long-tail scenario needs, improve the utilization rate of hardware resources, shorten the overall process, improve accuracy, save manpower, effectively reduce manual workload, and realize work automation.

[0101] The embodiment of the present invention further provides a text recognition device, comprising:

[0102] An acquisition module, used for acquiring a training set; the training set includes documents and / or images;

[0103] A training module is used to train the detection classification network, the text detection network and the text recognition network respectively through the training set to obtain the detection classification model, the text detection model and the text recognition model; wherein the detection classification model includes a decoupled large-category classification model and a subdivision classification model corresponding to each large category, the text detection network uses a deep convolutional neural network as the backbone network and is combined with a pixel aggregation network, and the text recognition network is trained according to CTC loss combined with CenterLoss;

[0104] The processing module is used to obtain the object to be analyzed, classify the object to be analyzed through the detection and classification model, determine the target classification, detect the object to be analyzed through the text detection network, determine the target text area, and recognize the target text area through the text recognition model to obtain the text recognition result.

[0105] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments, and will not be repeated herein.

[0106] The embodiment of the present invention also provides another text recognition device, which includes a processor and a memory, wherein at least one instruction, at least one program, code set or instruction set is stored in the memory, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the text recognition method based on deep learning in the above embodiment. Optionally, the text recognition device includes but is not limited to a mobile phone, a tablet computer, a computer, and a car computer.

[0107] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments, and will not be repeated herein.

[0108] An embodiment of the present invention also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the deep learning-based text recognition method of the aforementioned embodiment.

[0109] The embodiment of the present invention further provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the text recognition method based on deep learning of the aforementioned embodiment.

[0110] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0111] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0112] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present application can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0113] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0114] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A text recognition method based on deep learning, characterized in that: include: Get the training set; The training set includes documents and / or images; The detection classification network, the text detection network and the text recognition network are trained respectively by the training set to obtain a detection classification model, a text detection model and a text recognition model; wherein the detection classification model includes a decoupled large-category classification model and a subdivision classification model corresponding to each large category, the text detection network uses a deep convolutional neural network as a main network and is combined with a pixel aggregation network, and the text recognition network is trained according to CTC loss combined with Center Loss; Acquire the object to be analyzed, classify the object to be analyzed by the detection classification model, determine the target classification, detect the object to be analyzed by the text detection network, determine the target text area, recognize the target text area by the text recognition model, and obtain the text recognition result.

2. The method for character recognition based on deep learning according to claim 1, characterized in that: The step of classifying the object to be analyzed by using the detection classification model to determine the target classification includes: When the major category of the object to be analyzed is known, a target subdivision classification model corresponding to the known major category is determined, and the object to be analyzed is classified by the target subdivision classification model to obtain a target classification; Otherwise, the object to be analyzed is first classified by the major classification model, the target segmentation classification model corresponding to the first classification is determined, and the object to be analyzed is second classified by the target segmentation classification model to obtain the target classification; the first classification includes one of contracts, resumes, forms, cards and bills.

3. The method for character recognition based on deep learning according to any one of claims 1 to 2, characterized in that: The training of the major classification model includes: Constructing a large-category classification network with a network structure of Resnet network combined with transformer network; the detection classification network includes the large-category classification network; The large-category classification network is trained for an image-to-text pre-training task by using the training set, and first difficult example data is added to the training set during the training process; the training set further has samples added by manual intervention or enhancement operation, and the first difficult example data includes at least one of blurred, occluded, misaligned, and recognized abnormal data; When the training reaches the first termination condition, a large-category classification model is obtained.

4. The method for character recognition based on deep learning according to any one of claims 1 to 2, characterized in that: The training of the segmentation classification model includes: Constructing a segmentation classification network whose network structure is a Yolo network; adding the regression of the rotation angle in the segmentation classification network to the output of the Yolo network, and the detection classification network includes the segmentation classification network; The segmentation classification network is trained with mixed precision by using the training set and the pytorch framework, Warmup and CosineDecay are used as the learning rate in the training process, and second difficult example data is added to the training set; the second difficult example data includes at least one of blurred, occluded, misaligned and recognized abnormal data; When the training reaches the second termination condition, a segmentation classification model is obtained.

5. The method for character recognition based on deep learning according to any one of claims 1 to 2, characterized in that: The training set includes training data of major categories and subcategories, and the training of the text detection model includes: A text detection network is constructed using a deep convolutional neural network as the backbone network and combined with a pixel aggregation network; The text detection network is trained by using the training set and the pytorch framework, and the boundary distance loss of the outward expansion distance is added during the training process, and third difficult example data is added to the training set; the training set also has samples added by manual intervention or enhancement operation, and the third difficult example data includes at least one of blurred, occluded, misaligned and recognized abnormal data; the outward expansion distance is the average boundary distance calculated according to the boundary pixel points of the text area when the text detection network recognizes the text area during reasoning; When the training reaches the third termination condition, the text detection model is obtained.

6. The method for character recognition based on deep learning according to any one of claims 1 to 2, characterized in that: The training of the text recognition model includes: Construct a text recognition network with a network structure of convolutional neural network combined with recurrent neural network; The text recognition network is trained by using the training set and the pytorch framework; the training set has samples added by manual intervention or enhancement operation; The CTC loss and the Center Loss are calculated during the training process, and when the CTC loss and the Center Loss meet the fourth termination condition, a text recognition model is obtained.

7. The method for character recognition based on deep learning according to any one of claims 1 to 2, characterized in that: The method further comprises: Obtaining error data corresponding to the text recognition result; The erroneous data is added to the training set, and the step of training the detection classification network, the text detection network and the text recognition network respectively through the training set is returned to obtain at least one of a new detection classification model, a new text detection model and a new text recognition model.

8. A text recognition device, characterized in that: include: An acquisition module, used to acquire a training set; the training set includes documents and / or images; A training module, used to train the detection classification network, the text detection network and the text recognition network respectively through the training set to obtain a detection classification model, a text detection model and a text recognition model; wherein the detection classification model includes a decoupled large-category classification model and a subdivision classification model corresponding to each large category, the text detection network uses a deep convolutional neural network as a backbone network and is combined with a pixel aggregation network, and the text recognition network is trained according to CTC loss combined with Center Loss; The processing module is used to obtain the object to be analyzed, classify the object to be analyzed through the detection classification model, determine the target classification, detect the object to be analyzed through the text detection network, determine the target text area, recognize the target text area through the text recognition model, and obtain the text recognition result.

9. A text recognition device, characterized in that: The text recognition device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1-7.

Images