A general method and system for structured document recognition based on deep learning

Abstract

This invention relates to the field of image recognition technology, and more particularly to a general-purpose structured document recognition method and system based on deep learning. The method includes: acquiring document image information and preprocessing the acquired images; inputting a standardized document image into a text detection network to locate text instances in the image and saving the location information to a file; extracting text images from the document image based on the detected text instance location coordinates and inputting them into a text recognition network for recognition, saving the recognition result after the location information; using a key text extraction network to classify the text entities in the recognition result, removing non-key types, and then saving the classification result after the recognition result; and performing structured processing on the key text extraction result and displaying it. This invention can extract key text information from complex document images, enabling intelligent document reading, and is applicable to various types of documents.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a general method and system for structured identification of documents based on deep learning. Background Technology

[0002] Certificates and licenses, as valid proof of citizenship, are frequently used in various situations requiring identity verification, such as banks, airports, customs, and train stations. Accurate verification of certificates and licenses is crucial for safeguarding citizen and public security. Manual verification is inefficient, wastes significant human resources, and is ill-suited to the rapidly evolving globalized world. Therefore, researching a structured identification scheme for certificates and licenses to automate verification can not only improve verification efficiency but also protect citizens' privacy and personal information security, making it a valuable research endeavor.

[0003] Certificates and licenses consist of multiple areas, including machine-readable, visually readable, and portrait areas. They are printed with important information such as name, gender, date of birth, document number, and expiration date, making them a special type of structured document. Structured recognition of certificates and licenses refers to the process of converting certificate and license images into text content using text detection and recognition technologies, extracting key information, and storing it as a structured output in key-value pair format. In practical applications, certificates and licenses are diverse in type and layout, often featuring complex background patterns and anti-counterfeiting features, and frequently suffer from character stains and damage, posing significant challenges to existing algorithms. Therefore, how to provide a structured recognition method for multiple certificates and licenses, extracting key text information from complex certificate and license images, and achieving intelligent document reading is a problem that this invention urgently needs to solve.

[0004] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0005] One of the objectives of this invention is to overcome the deficiencies in the prior art and provide a general document structure recognition method and system based on deep learning. First, text detection technology is used to locate the position of text instances in the document image. Then, text recognition technology is used to recognize the text image to obtain the recognition result. Finally, key text is extracted from the recognition result through key text extraction technology, thus solving the shortcomings of the existing technical solutions.

[0006] This invention is implemented as follows:

[0007] In a first aspect, the present invention provides a general method for structured identification of identification documents based on deep learning, comprising:

[0008] Collect document image information and preprocess the collected images to obtain standardized document images;

[0009] Standardized document images are input into a text detection network to locate text instances in the image and save the location information to a file.

[0010] Based on the detected text instance location coordinates, the text image is extracted from the document image and input into the text recognition network for recognition. After obtaining the recognition result, it is saved after the location information.

[0011] The key text extraction network is used to classify the text entities in the recognition results, remove non-key types, and then save the classification results after the recognition results.

[0012] The extracted key text results are structured and displayed, and then uploaded to the business system for further processing.

[0013] Furthermore, the structure of the text detection network includes:

[0014] Based on DBNet, ResNet18 is used as the feature extraction network, and the regular 3×3 convolutional structure in ResNet18 is replaced by a grouped multi-path selectable convolutional structure; during training, a discriminative loss function is used to supervise the feature maps of the feature extraction output.

[0015] Furthermore, the grouped multi-path selectable convolutional structure includes three convolutional branches with kernel sizes of 1×3, 1×5 and 3×3, respectively. Then, a channel attention mechanism is used to assign weights to the branches, and finally, the weighted sum of each branch is used as the final convolutional feature.

[0016] The discriminative loss function uses Euclidean distance to measure the distance between feature vectors, and its calculation method includes mapping each pixel in the text image to a... The vector is dimensional, and a text instance is considered as a cluster. Let the number of clusters be . , Indicates the first The number of elements in a cluster. The first in the cluster The feature vector of each element Indicates the first The average eigenvector of each cluster is used as the cluster center. express Distance or distance, express , and These represent the boundaries of variance loss and distance loss, respectively. Based on experience, in this use case... and The selected values ​​are 0.5 and 1.5. Therefore, the feature vector distance loss within the same cluster can be obtained. and distance loss between different clusters :

[0017]

[0018]

[0019] Determine the loss ,equal and The sum is:

[0020]

[0021] Among them, by minimizing The loss function can minimize the distance between feature vectors within a cluster, reducing the differences in feature vector representations across the cluster. This is achieved by minimizing... Loss can maximize the distance between different clusters, increasing the distinguishability between them.

[0022] Furthermore, the structure of the text recognition network includes:

[0023] Based on the CRNN network, the Transformer encoder is used as the sequence modeling module to replace the BiLSTM module in the CRNN network. The semantic modeling ability of the Transformer encoder is trained by masking the input features. The CVSM module is added to the text recognition network to assign different weights to the context features and visual features obtained after sequence modeling, so as to obtain joint features and decode them for output.

[0024] Furthermore, the CVSM module uses an attention mechanism to assign weights to the visual features and the contextual features obtained in the sequence modeling stage, resulting in joint features. Specifically, the text image obtains visual features after the feature extraction stage, denoted as... , The context features containing sequence semantic information obtained through the sequence modeling stage are denoted as follows: , and With the same feature dimension, joint features The calculation process includes:

[0025]

[0026]

[0027]

[0028] In the formula, Indicates a stacking operation. For activation function, This represents matrix multiplication.

[0029] Furthermore, the key text extraction network uses text images and detection and recognition results as input, and includes three parts: feature encoding, relation modeling, and feature decoding, wherein:

[0030] The feature encoding specifically includes: encoding the visual image, text characters, and position coordinates of the text entity respectively, and then fusing them into a feature vector by addition as the feature representation of the text entity;

[0031] The relationship modeling specifically includes: modeling the relationship between text entities using BiLSTM. The input of BiLSTM is the output of the feature encoding stage. Before inputting the feature encoding vector into BiLSTM, the text is first sorted according to the relative order in which it appears in the document, that is, sorted from top to bottom and from left to right. The feature sequence of the text is obtained after sorting and then input into BiLSTM to learn the mutual relationship and order information between the elements of the feature sequence.

[0032] The feature decoding specifically includes: in the decoding prediction stage, the text extraction task is regarded as a text entity classification task, and the probability of each category of different text entities is directly predicted using a fully connected layer. Then, the softmax function is used to find the category with the highest probability, which represents the classification result of the corresponding text.

[0033] Furthermore, the encoding of the visual image, text characters, and position coordinates of the text entity includes visual feature encoding, text character feature encoding, and position feature encoding, wherein:

[0034] The visual feature encoding method specifically includes: based on the detected text location information, converting it into the corresponding position in the extracted feature map using ROIAlign, and then extracting the feature vector of that position as a visual feature; or based on the detected text location information, extracting the text image from the original image, and then reusing the feature extraction network to perform feature extraction operations as a visual feature.

[0035] The text character feature encoding method specifically includes: firstly, adding an extra character to the beginning and end of the string result obtained from text recognition. <cls>Characters are used to obtain the overall information of the entire text. Then, each character in the string is encoded into a 512-dimensional word vector using Embedding. Finally, the text encoding result is obtained through a Transformer encoder, and the first dimension of the result is taken. <cls>The output at the given position represents the text feature encoding result of the entire string;

[0036] The location feature encoding method specifically includes: first, normalizing the coordinates using the width and height of the image; then, encoding the normalized coordinates through a linear layer to obtain a location feature vector; specifically, assuming the coordinates of the text box are... The image has a width of w and a height of h. First, the coordinates of the text box are normalized to obtain normalized coordinate information: The normalized coordinates are passed through a linear layer to obtain the location features. ,in These are learnable parameters, and max represents the ReLU activation function.

[0037] Furthermore, the process of collecting document image information and preprocessing the collected images to obtain standardized document images specifically includes:

[0038] The system collects document image information and preprocesses the collected images, including brightness correction, background cropping, and rotation correction, to obtain a standardized document image. Brightness correction is used to obtain a document image with uniform brightness; background cropping is used to remove the document background to obtain a background-removed document image; and rotation correction is used to correct the orientation of the document image.

[0039] Furthermore, the structuring processing of the extracted key text results specifically includes:

[0040] Read the saved text location coordinates, text recognition results, and text classification results line by line, and convert them into a structured form.

[0041] Secondly, the present invention provides a general-purpose document structure recognition system based on deep learning, used to implement the method described in the first aspect. The system includes an image acquisition module, an image preprocessing module, a text detection module, a text recognition module, a key text extraction module, and an information processing and display module; wherein:

[0042] The image acquisition module is used to acquire images of the documents and transmit them to a computer for processing;

[0043] The image preprocessing module includes brightness correction, background cropping, and rotation correction operations, used to extract a document image with uniform brightness, no background, and correct orientation from the original image;

[0044] The text detection module is used to detect text regions on document images;

[0045] The text recognition module is used to recognize the text content in the detected text area;

[0046] The key text extraction module is used to classify the identified text and remove non-key text.

[0047] The information processing and display module is used to structure and display key texts, and simultaneously upload them to the business system for further processing.

[0048] In summary, the beneficial effects of the present invention are as follows:

[0049] (1) In the document text detection task, firstly, a grouped multi-path selectable convolutional structure is proposed to address the problem of detecting long and thin text in documents, thereby improving the receptive field of the feature extraction network and increasing the detection accuracy of the text detection network for long and thin text. Secondly, to address the problem of text sticking or breaking, a text detection method based on discriminative loss is proposed to enhance the network's ability to express and distinguish features of different text instances, thus solving the problem of text sticking or breaking.

[0050] (2) In the document text recognition task, to address the problems of background interference, character wear and stains, a sequence modeling method based on a Transformer encoder is proposed. Furthermore, by using mask training, the semantic correlation between text characters is fully explored, improving the prediction accuracy of the text recognition network for characters with interference and stains. Further, to ensure the recognition effect on semantically nuanced text in documents, a context feature and visual feature selection module is proposed, combining contextual and visual information for decoding output, thereby improving the network's text recognition accuracy in documents.

[0051] (3) In the key text extraction task, considering the complex and diverse layout of the certificate and the difficulty of information extraction, the text extraction is regarded as a classification task of text entities in the certificate. A feature encoding method that integrates visual, text and location information is proposed to represent text entities using multi-dimensional information. At the same time, a text entity context relationship modeling method based on BiLSTM is proposed to make full use of the relationship between text entities in the certificate and layout information. Finally, a key text extraction network consisting of feature encoding, relationship modeling and decoding prediction is established to realize the structured recognition of multiple certificates. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A flowchart of a general document structure recognition method based on deep learning provided in Embodiment 1 of the present invention;

[0054] Figure 2 This is a schematic diagram of the GMSK convolutional structure provided in Embodiment 1 of the present invention;

[0055] Figure 3 This is a schematic diagram of the CVSM module structure provided in Embodiment 1 of the present invention;

[0056] Figure 4 This is a schematic diagram of the key text extraction network structure provided in Embodiment 1 of the present invention;

[0057] Figure 5 This is a schematic diagram of the feature encoding part in the key text extraction network provided in Embodiment 1 of the present invention;

[0058] Figure 6 This is a schematic diagram of the relationship modeling part in the key text extraction network provided in Embodiment 1 of the present invention;

[0059] Figure 7 This is a schematic diagram of the decoding and prediction part of the key text extraction network provided in Embodiment 1 of the present invention;

[0060] Figure 8 This is a schematic diagram of a general document structure recognition system based on deep learning, provided in Embodiment 2 of the present invention.

[0061] Figure 9 This is a schematic diagram of a general-purpose document structure recognition device based on deep learning, provided in Embodiment 3 of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0063] This invention is an architecture of a specific functional system. Therefore, the specific embodiments mainly describe the functional logic relationship of each structural module, and do not limit the specific software and hardware implementation methods.

[0064] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other, and the order of the steps can be changed if they are logical and do not conflict.

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Example 1:

[0067] like Figure 1 As shown, Embodiment 1 of the present invention provides a general document structure recognition method based on deep learning, comprising the following steps:

[0068] Step 100: Collect document image information and preprocess the collected images to obtain standardized document images.

[0069] Step 200: Input the standardized document image into the text detection network, locate the text instances in the image, and save the location information to a file.

[0070] Step 300: Based on the detected text instance location coordinates, extract the text image from the document image and input it into the text recognition network for recognition. After obtaining the recognition result, save it after the location information.

[0071] Step 400: Use a key text extraction network to classify the text entities in the recognition results, remove non-key types, and then save the classification results after the recognition results.

[0072] Step 500: The key text extraction results are structured and displayed, and then uploaded to the business system for further processing.

[0073] This preferred embodiment can solve the problems existing in the prior art through the above steps, and can extract key text information from complex certificate images to realize intelligent recognition of certificates, and is suitable for various types of certificates.

[0074] In this preferred embodiment, step 100 (collecting document image information and preprocessing the collected images to obtain standardized document images) specifically includes: collecting document image information and preprocessing the collected images, including brightness correction, background cropping, and rotation correction operations, to obtain standardized document images; wherein, the brightness correction is used to obtain document images with uniform brightness; the background cropping is used to remove the document background to obtain document images after background removal; and the rotation correction is used to correct the orientation of the document images.

[0075] In the above process, the acquisition of document image information begins with capturing an image of the document using a dedicated document-taking device, such as a camera or a CIS scanner. Alternatively, other devices with image acquisition capabilities can be used to capture the image. The image is then transferred to a computer via USB cable, Bluetooth, or Wi-Fi. For brightness correction, background cropping, and rotation correction operations, due to the instability of lighting conditions on different devices, the brightness of the captured images may vary, and different areas within the same image may also have different brightness levels. To avoid affecting subsequent operations, brightness correction is necessary to obtain a uniformly bright document image. Then, to process the document information, the background needs to be cropped from the document image to extract the document itself. Since the orientation of the document placed by the user is uncertain, rotation correction is also required.

[0076] The brightness correction operation specifically involves: first, placing a standard white sheet of paper into the device for image acquisition. Then, comparing each pixel of the acquired image with a preset brightness, recording the quotient between the preset brightness and each pixel as a correction coefficient. When the original image is acquired, multiplying the pixel value of each pixel in the original image by the corresponding correction coefficient yields the brightness-corrected image.

[0077] The background operation is as follows: First, the image is binarized using Otsu's method. Then, the `findContours` function provided by OpenCV is used to find the contour with the largest area. The contour is represented by a set of points. Considering that there may be raised shapes such as hands in the image, it is necessary to find the concave parts adjacent to the contour. The `convexityDefects` function provided by OpenCV is used to find the concave parts. The contour point set between two adjacent concave parts is removed, which is equivalent to cropping the raised part. After the raised part is cropped, the contour of the document is obtained. Finally, the `minAreaRect` function is used to find the smallest bounding rectangle of the contour. The document can be extracted from the image by performing the corresponding affine transformations according to the horizontal and vertical directions of the rectangle, which is to achieve the background cropping operation.

[0078] The rotation operation is as follows: For documents with machine-readable codes, the position of the machine-readable code is used to determine the up or down position. For documents with a portrait, the position of the portrait can be detected and the position can be used to determine the up or down position. For other documents, the position can be determined by calculating the variance of the up, down, left and right areas of the document, or by finding and locating a specific pattern.

[0079] In this preferred embodiment, step 200 specifically includes: inputting the image obtained in step 100 into the improved text detection network of the present invention, locating text instances in the image, and saving the location information to a file. Optionally, the location information of the text instance in the image in this embodiment can be represented using the four coordinates of the smallest bounding rectangle of the text instance, or using the center point, length, width, and rotation angle of the rectangle. For example, using... To represent the location information of a text instance, where , represents the vertex of the rectangle. After obtaining all the detection results, some erroneous detection results are first filtered out according to preset rules, such as excluding all detection results with an area less than 25. Finally, the position information of all text instances is saved line by line in the Result.txt file.

[0080] In this embodiment, the text detection network in step 200 is based on DBNet and uses ResNet18 as the feature extraction network. Furthermore, to address the issue of long and thin text, this invention proposes a Grouped Multiple-Path Selectable Convolutional Structure (GMSK convolution) to replace the conventional 3×3 convolutional structure in ResNet18. When detecting long and thin text, the conventional square convolutional structure in ResNet18 struggles to adequately cover text instances and easily introduces excessive noise in the vertical direction, leading to a decrease in the accuracy of the text detection network for long and thin text.

[0081] In this embodiment, the grouped multiplex selectable convolutional structure (GMSK convolutional structure) is as follows: Figure 2 As shown, the network includes three convolutional branches with kernel sizes of 1×3, 1×5, and 3×3. A channel attention mechanism is then used to assign weights to these branches, and finally, a weighted sum of the branch features is obtained as the final convolutional feature. GMSK convolution combines multi-sized convolutional kernels with a channel attention mechanism, enabling it to adaptively select different convolutional structures based on the target shape and stack them in multi-layer networks, resulting in a highly flexible receptive field. For example, stacking two 1×5 convolutional kernels yields a 1×25 receptive field; stacking 1×3 and 1×5 kernels yields a 1×15 receptive field; and stacking 3×3 and 1×5 kernels yields a 3×15 receptive field. This flexible receptive field improves the accuracy of text detection networks, especially for long and thin texts, allowing for better acquisition of complete information about the text instance while reducing noise introduction, thus achieving higher detection accuracy.

[0082] In this embodiment, during training, a discriminative loss function is used to supervise the feature map output from feature extraction. The loss function used for training the detection network consists of two parts: one is the discriminative loss function proposed in this invention (denoted as...). One is used to supervise the generation of feature vectors for different text instances, solving the problem of text instances sticking together or breaking apart; the other is the loss of the network prediction part output (denoted as ). Based on experience, hyperparameters The value is 0.2. Then the total network loss (denoted as...) It can be written as:

[0083]

[0084] Discriminant loss function The distance between feature vectors is measured using Euclidean distance. The specific calculation method is as follows: each pixel in the text image is mapped to a... The vector is dimensional, and a text instance is considered as a cluster. Let the number of clusters be . , Indicates the first The number of elements in a cluster. The first in the cluster The feature vector of each element Indicates the first The average eigenvector of each cluster is used as the cluster center. express Distance or distance, express , and These represent the boundaries of variance loss and distance loss, respectively. Based on experience, in this use case... and The selected values ​​are 0.5 and 1.5. Therefore, the feature vector distance loss within the same cluster can be obtained. and distance loss between different clusters :

[0085]

[0086]

[0087] Determine the loss ,equal and The sum is:

[0088]

[0089] Among them, by minimizing The loss function can minimize the distance between feature vectors within a cluster, reducing the differences in feature vector representations across the cluster. This is achieved by minimizing... Loss can maximize the distance between different clusters, increasing the distinguishability between them.

[0090] Loss of network output Including text score graph loss (denoted as...) Threshold map loss (denoted as) ) and binarization map loss (denoted as ):

[0091]

[0092] Based on experience, the hyperparameters μ and ν are typically set to 1.0 and 5.0, respectively. and Calculated using BCELoss The result is obtained using L1Loss:

[0093]

[0094] Text score graph (denoted as) Threshold map (denoted as) The binarized image (denoted as ) is obtained from network prediction. )Depend on and The calculation yielded:

[0095]

[0096] During training, the network is first pre-trained on a public dataset, and then trained on the entire network using labeled real certificate data.

[0097] In this preferred embodiment, step 300 specifically includes: extracting a text image from the document image based on the detected text instance location coordinates, inputting it into the improved text recognition network of this invention for recognition, and storing the recognition result after the coordinate information. Specifically, during recognition, the coordinates in Result.txt are read line by line, and the text instance image is extracted from the preprocessed document image based on the coordinates. The image is then input into the text recognition network to obtain the recognition result string, which is stored in the corresponding line of Result.txt after the coordinate information. The structure of the text recognition network in this embodiment includes: using a CRNN network as a foundation, a Transformer encoder is used as the sequence modeling module to replace the BiLSTM module in the CRNN network. The semantic modeling capability of the Transformer encoder is trained by masking the input features; and a CVSM module is added to the text recognition network to assign different weights to the context features and visual features obtained after sequence modeling, obtaining joint features, which are then decoded and output.

[0098] The text recognition network described in this embodiment is an improvement upon the CRNN network. To address the issues of background interference and character corruption, this invention uses a Transformer encoder as the sequence modeling module instead of the BiLSTM module in the CRNN network, and employs a masking training method to model the semantic information between characters, thereby assisting the network in recognizing interfered and corrupted characters. To improve the lexical dependency problem of the text recognition network, this invention proposes a contextual and visual feature selection module (CVSM) based on an attention mechanism and incorporates it into the text recognition network. Different weights are assigned to contextual features and visual features to obtain joint features for decoding and output.

[0099] In this embodiment, the CVSM module uses an attention mechanism to assign weights to visual features and contextual features obtained in the sequence modeling stage, resulting in joint features. The structure of the CVSM module is as follows: Figure 3 As shown, after the feature extraction stage, the text image obtains visual features (denoted as ). ), The sequence modeling stage yields contextual features containing sequence semantic information (denoted as...). ), and They have the same feature dimensions. Joint features The calculation process is as follows:

[0100]

[0101]

[0102]

[0103] In the formula Indicates a stacking operation. For activation function, This represents matrix multiplication.

[0104] The training of a text recognition network is divided into two stages: pre-training of the Transformer encoder and training of the entire text recognition network.

[0105] During the pre-training of the Transformer encoder, the visual features of the input are randomly masked, and CTCLoss and CELoss are used for supervision at the same time. This enables the sequence modeling module to fully capture the semantic association information between characters and improve the network's recognition accuracy for both distorted and lossless characters.

[0106] When training the text recognition network as a whole, the parameters of the Transformer encoder are fixed, the CVSM module is added to the network, and finally CTCLOss is used for supervision.

[0107] In this preferred embodiment, step 400 specifically includes: classifying text entities using the key text extraction network proposed in this invention, removing non-key types, and then saving the classification results after the recognition results.

[0108] First, define the key text entity types. For example, for passports, define 14 text entity types: passport code, issuing country or issuing authority code, passport number, primary name identifier, secondary name identifier, nationality, date of birth, personal identification number, gender, place of birth, place of issuance, date of issuance, issuing authority, and expiry date. For domestic travel documents, define 12 text entity types: document number, Chinese name, pinyin name, date of birth, validity period, gender, issuing authority, place of issuance, number of issuances, Hong Kong and Macau document number, date of issuance, and expiry date.

[0109] Those other than critical text entity types are defined as non-critical text entity types.

[0110] During classification, the text location coordinates and text recognition results in Result.txt are read line by line. Based on the coordinates, text instance images are extracted from the preprocessed document image. Then, the text location, text characters, and text image are input into the key text extraction network to obtain the classified results, which are saved after the text recognition results in the corresponding line of Result.txt. Lines containing non-key text entity types are then deleted.

[0111] The structure of the key text extraction network is as follows: Figure 4 As shown, the text image and detection and recognition results are used as input, and the process includes three parts: feature encoding, relationship modeling, and feature decoding.

[0112] Feature encoding methods such as Figure 5 As shown, the visual image, text characters, and position coordinates of the text entity are encoded separately, and then directly fused into a feature vector by addition as the feature representation of the text entity.

[0113] Specifically, the visual feature encoding method preferably involves converting the text location information detected in step 200 into the corresponding location in the feature map extracted in step 200 using ROIAlign, and then extracting the feature vector of that location as a visual feature. Optionally, based on the text location information detected in step 200, the text image is extracted from the original image, and then the feature extraction network is used again to perform feature extraction operations as a visual feature.

[0114] The text character feature encoding method specifically involves adding an extra character to the beginning and end of the string result obtained from text recognition. <cls>Characters are used to obtain the overall information of the entire text. Then, each character in the string is encoded into a 512-dimensional word vector using Embedding. Finally, the text encoding result is obtained through a Transformer encoder, and the first dimension of the result (i.e., ...) is taken. <cls>The output at the current position represents the text feature encoding result of the entire string.

[0115] The location feature encoding method specifically involves first normalizing the coordinates using the width and height of the image, and then encoding the normalized coordinates through a linear layer to obtain a location feature vector. Assume the coordinates of the text box are... The image has a width of w and a height of h. First, the coordinates of the text box are normalized to obtain normalized coordinate information. The normalized coordinates are passed through a linear layer to obtain the location features. ,in These are learnable parameters, and max represents the ReLU activation function.

[0116] Relationship modeling methods such as Figure 6 As shown, BiLSTM is used to model the relationships between text entities. The relationship modeling module used in this embodiment consists of two layers of BiLSTM with a hidden state feature dimension of 512. The input to the BiLSTM is the output of the feature encoding stage. Before inputting the feature encoding vector into the BiLSTM, the text is first sorted according to its relative order of appearance in the document, i.e., from top to bottom and from left to right. This sorting yields the text's feature sequence, which is then input into the BiLSTM to learn the relationships and order information between the elements of the feature sequence.

[0117] In certificates, a text entity is highly correlated with its neighboring text. After the sorting process described above, these associated text entities will be closely distributed on its left and right sides. BiLSTM can effectively model the contextual information in both left and right directions of the sequence. Secondly, although the distribution of text entities may change in different certificates, the relative order in which they appear is relatively stable, and BiLSTM can effectively model this order information. Finally, since the feature vectors representing text entities contain positional information, the above modeling method can also learn the layout information between text entities to a certain extent. In summary, modeling using bidirectional long short-term memory units can effectively learn the contextual and layout information between text entities, thus achieving good results.

[0118] Decoding process as follows Figure 7 As shown, in the decoding and prediction stage, this embodiment of the invention treats the text extraction task as a text entity classification task. It uses fully connected layers to directly predict the probability of each category for different text entities, and then uses a softmax function to find the category with the highest probability, representing the classification result of the corresponding text. Currently, mainstream text extraction methods use BiLSTM-CRF layers for character-level prediction, then use the Viterbi algorithm to find the optimal path, and finally extract the text entities; this algorithm is time-consuming. In the text detection task, this embodiment of the invention solves the problem of text instance concatenation or disconnection by introducing discriminative loss supervision. Therefore, the detected text instance corresponds to a complete entity category, eliminating the need for character-level prediction. Based on these conditions, this paper removes the time-consuming BiLSTM-CRF layer from the text extraction task, directly predicting the category of the text entity, significantly improving prediction efficiency without sacrificing network accuracy.

[0119] During network training, the Adam optimizer was used for optimization. To increase the diversity of training samples and improve the robustness of the model, online augmentation was employed to enhance the training data. Online augmentation techniques included blurring, color and brightness transformations, random cropping, etc. During training, to address the problem of imbalanced samples, the FocalLoss loss function was used to focus on learning from difficult examples.

[0120] In this preferred embodiment, step 500 specifically includes: performing structured processing on the key text extraction results of step 400 and displaying them on the user interface, while simultaneously uploading them to the business system for processing, such as opening a gate at an airport or opening an account at a bank.

[0121] The structured processing specifically involves reading the text position coordinates, text recognition results, and text classification results stored in Result.txt line by line, and converting them into a structured format. Optionally, for example, JSON format can be used.

[0122] {

[0123] "ID Number": {"Coordinates": [[545, 54], [762, 54], [762, 86], [545, 86]], "Result": "C12345678"},

[0124] "Chinese Name": {"Coordinates": [[280, 121], [392, 121], [392, 154], [280,154]], "Result": "Zhang San"},

[0125] "Pinyin Name": {"Coordinates": [[280, 141], [412, 141], [412, 174], [280,174]], "Result": "ZHANG,SAN"},

[0126] "Date of Birth": {"Coordinates": [[280, 242], [428, 242], [428, 269], [280,269]], "Result": "2000.01.01"}

[0127] }

[0128] In summary, this invention addresses the issue of detecting long and thin text in certificates and documents by first proposing a grouped multi-path selectable convolutional structure to improve the receptive field of the feature extraction network and enhance the detection accuracy of the text detection network for long and thin text. Secondly, to address the problem of text concatenation or breakage, a text detection method guided by discriminative loss is proposed to enhance the network's ability to express and distinguish features of different text instances, thus resolving the issue of text concatenation or breakage.

[0129] In the document text recognition task, this invention proposes a sequence modeling method based on a Transformer encoder to address the problems of background interference, character wear and stains, and other issues. It also utilizes masking training to fully exploit the semantic correlations between text characters, improving the prediction accuracy of the text recognition network for characters with interference and stains. Furthermore, to ensure effective recognition of semantically insensitive text within documents, a contextual and visual feature selection module is proposed. This module combines contextual and visual information for decoding output, further enhancing the network's text recognition accuracy in documents.

[0130] In the key text extraction task, this invention addresses the challenges of complex and diverse document layouts and the difficulty of information extraction. Treating text extraction as a classification task of text entities within the documents, it proposes a feature encoding method that integrates visual, textual, and locational information. This method utilizes multidimensional information to represent text entities. Furthermore, it proposes a BiLSTM-based text entity context relationship modeling approach, fully leveraging the relationships and layout information between text entities within the documents. Finally, a key text extraction network consisting of feature encoding, relationship modeling, and decoding prediction is established, enabling structured recognition of multiple documents.

[0131] Example 2:

[0132] Based on the deep learning-based general document structure recognition method provided in Embodiment 1 above, Embodiment 2 of the present invention provides a deep learning-based general document structure recognition system to implement the method provided in Embodiment 1, such as... Figure 8 As shown, the system includes an image acquisition module, an image preprocessing module, a text detection module, a text recognition module, a key text extraction module, and an information processing and display module.

[0133] The image acquisition module described in this embodiment is used to operate a dedicated device to acquire images of documents and transmit them to a computer for processing.

[0134] The image preprocessing module in this embodiment includes brightness correction, background cropping, and rotation correction operations, used to extract a document image with uniform brightness, no background, and correct orientation from the original image.

[0135] The text detection module described in this embodiment is used to detect text regions on document images; specifically, the image output by the image preprocessing module is input into the improved text network of this invention to detect text regions on document images, and the position information is saved line by line to Result.txt.

[0136] The text recognition module described in this embodiment is used to recognize the text content in the detected text region; specifically, it reads the position information in Result.txt line by line, extracts the text image from the document image, inputs it into the improved text recognition network of this invention, obtains the text character recognition result, and saves it in the corresponding line of Result.txt after the position information.

[0137] The key text extraction module in this embodiment is used to classify the identified text and remove non-key text; the text image, text characters and text position information are input into the key text extraction network proposed in this invention. Specifically, the text is actually classified and the classification results are saved in Result.txt, and then the lines containing non-key text are removed.

[0138] The information processing and display module described in this embodiment is used to perform structured processing on key text and display it, while simultaneously uploading it to the business system for further processing. Specifically, it reads the text position coordinates, text recognition results, and text classification results stored in Result.txt line by line, converts them into a structured form, and then displays them on the user interface for easy viewing and interaction, while simultaneously uploading them to the business system for business processing.

[0139] Example 3:

[0140] Based on the deep learning-based general document structure recognition method provided in Embodiment 1 above, this invention also provides a deep learning-based general document structure recognition device that can be used to implement the above method and system, such as... Figure 9 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. This embodiment of the deep learning-based general document structure recognition device includes one or more processors 21 and a memory 22. Figure 9 Take a processor 21 as an example.

[0141] Processor 21 and memory 22 can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0142] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the deep learning-based universal document structure recognition method in Embodiment 1. The processor 21 executes various functional applications and data processing of the deep learning-based universal document structure recognition device by running the non-volatile software programs, instructions, and modules stored in the memory 22, thereby realizing the deep learning-based universal document structure recognition method of Embodiment 1.

[0143] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0144] The program instructions / modules are stored in memory 22. When executed by one or more processors 21, they perform the deep learning-based general document structure recognition method described in Embodiment 1 above, for example, the method described above. Figure 1 The steps shown.

[0145] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0146] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention. Contents not described in detail in this specification are prior art known to those skilled in the art.

[0147] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention. Contents not described in detail in this specification are prior art known to those skilled in the art.< / cls> < / cls> < / cls> < / cls>

Claims

1. A general method for structured identification of identity documents based on deep learning, characterized in that, include: Collect document image information and preprocess the collected images to obtain standardized document images; Standardized document images are input into a text detection network to locate text instances in the image and save the location information to a file. Based on the detected text instance location coordinates, the text image is extracted from the document image and input into the text recognition network for recognition. After obtaining the recognition result, it is saved after the location information. The key text extraction network is used to classify the text entities in the recognition results, remove non-key types, and then save the classification results after the recognition results. The key text extraction results are structured and displayed, and then uploaded to the business system for further processing. The structure of the text detection network includes: based on DBNet, using ResNet18 as the feature extraction network, and replacing the conventional 3×3 convolutional structure in ResNet18 with a grouped multi-path selectable convolutional structure; during training, a discriminative loss function is used to supervise the feature maps output by the feature extraction. The structure of the text recognition network includes: based on the CRNN network, a Transformer encoder is used as the sequence modeling module to replace the BiLSTM module in the CRNN network, and the semantic modeling ability of the Transformer encoder is trained by masking the input features; and a CVSM module is added to the text recognition network to assign different weights to the context features and visual features obtained after sequence modeling, so as to obtain joint features and decode and output them. The key text extraction network uses text images and detection and recognition results as input, and includes three parts: feature encoding, relationship modeling, and feature decoding. The feature encoding specifically includes: encoding the visual image, text characters, and position coordinates of the text entity respectively, and then fusing them into a feature vector by addition as the feature representation of the text entity; The relationship modeling specifically includes: modeling the relationship between text entities using BiLSTM. The input of BiLSTM is the output of the feature encoding stage. Before inputting the feature encoding vector into BiLSTM, the text is first sorted according to the relative order in which it appears in the document, that is, sorted from top to bottom and from left to right. The feature sequence of the text is obtained after sorting and then input into BiLSTM to learn the mutual relationship and order information between the elements of the feature sequence. The feature decoding specifically includes: in the decoding prediction stage, the text extraction task is regarded as a text entity classification task, and the probability of each category of different text entities is directly predicted using a fully connected layer. Then, the softmax function is used to find the category with the highest probability, which represents the classification result of the corresponding text.

2. The general document structure recognition method based on deep learning according to claim 1, characterized in that, The grouped multi-path selectable convolutional structure includes three convolutional branches with kernel sizes of 1×3, 1×5 and 3×3, respectively. Then, a channel attention mechanism is used to assign weights to the branches, and finally, the weighted sum of each branch is used as the final convolutional feature. The discriminative loss function uses Euclidean distance to measure the distance between feature vectors, and its calculation method includes mapping each pixel in the text image to a... A vector of dimension, and consider a text instance as a cluster, let the number of clusters be . , Represents a cluster The number of elements in the middle. The first in the cluster The feature vector of each element Represents a cluster The average eigenvector, which is the cluster center, express Distance or distance, express , and Let represent the boundaries of variance loss and distance loss, respectively. From this, we can obtain: in, This represents the distance loss of feature vectors within the same cluster. Minimizing this loss can minimize the distance between feature vectors within the cluster, thereby reducing the differences in feature vector representations within the cluster. This represents the distance loss between different clusters. Minimizing this loss can maximize the distance between different clusters and increase the distinguishability of different clusters. To determine the loss, equal to and sum.

3. The general document structure recognition method based on deep learning according to claim 1, characterized in that, The CVSM module uses an attention mechanism to assign weights to visual features and contextual features obtained in the sequence modeling stage, resulting in joint features. Specifically, after the feature extraction stage, the text image obtains visual features, denoted as... , The context features containing sequence semantic information obtained through the sequence modeling stage are denoted as follows: , and With the same feature dimension, joint features The calculation process includes: In the formula, Indicates a stacking operation. For activation function, This represents matrix multiplication.

4. The general document structure recognition method based on deep learning according to claim 1, characterized in that, The encoding of the visual image, text characters, and position coordinates of the text entity includes visual feature encoding, text character feature encoding, and position feature encoding, wherein: The visual feature encoding method specifically includes: based on the detected text location information, converting it into the corresponding position in the extracted feature map using ROIAlign, and then extracting the feature vector of that position as a visual feature; or based on the detected text location information, extracting the text image from the original image, and then reusing the feature extraction network to perform feature extraction operations as a visual feature. The text character feature encoding method specifically includes: firstly, adding an extra character to the beginning and end of the string result obtained from text recognition. <cls>Characters are used to obtain the overall information of the entire text. Then, each character in the string is encoded into a 512-dimensional word vector using Embedding. Finally, the text encoding result is obtained through a Transformer encoder, and the first dimension of the result is taken. <cls> The output at the given position represents the text feature encoding result of the entire string;< / cls> < / cls> The location feature encoding method specifically includes: first, normalizing the coordinates using the width and height of the image; then, encoding the normalized coordinates through a linear layer to obtain a location feature vector; specifically, assuming the coordinates of the text box are... The image has a width of w and a height of h. First, the coordinates of the text box are normalized to obtain normalized coordinate information: The normalized coordinates are passed through a linear layer to obtain the positional features. ,in These are learnable parameters, and max represents the ReLU activation function.

5. The general document structure recognition method based on deep learning according to any one of claims 1-4, characterized in that, The process of collecting document image information and preprocessing the collected images to obtain standardized document images specifically includes: The system collects document image information and preprocesses the collected images, including brightness correction, background cropping, and rotation correction, to obtain a standardized document image. Brightness correction is used to obtain a document image with uniform brightness; background cropping is used to remove the document background to obtain a background-removed document image; and rotation correction is used to correct the orientation of the document image.

6. The general document structure recognition method based on deep learning according to any one of claims 1-4, characterized in that, The specific steps of structuring the extracted key text results include: Read the saved text location coordinates, text recognition results, and text classification results line by line, and convert them into a structured form.

7. A general-purpose document structure recognition system based on deep learning, used to implement the general-purpose document structure recognition method based on deep learning as described in any one of claims 1-6, characterized in that, It includes an image acquisition module, an image preprocessing module, a text detection module, a text recognition module, a key text extraction module, and an information processing and display module; among which: The image acquisition module is used to acquire images of the documents and transmit them to a computer for processing; The image preprocessing module includes brightness correction, background cropping, and rotation correction operations, used to extract a document image with uniform brightness, no background, and correct orientation from the original image; The text detection module is used to detect text regions on document images; The text recognition module is used to recognize the text content in the detected text area; The key text extraction module is used to classify the identified text and remove non-key text. The information processing and display module is used to structure and display key texts, and simultaneously upload them to the business system for further processing.

Images