ID card image recognition method, device, computer equipment and storage medium

Abstract

This application relates to a document image recognition method, apparatus, computer device, and storage medium. The method comprises: acquiring an image to be recognized; detecting text in the image to be recognized and identifying the text fields of each detected text; performing categorization analysis on the text content and position of each text field to determine the text category of each text field; and obtaining a document image recognition result for the image to be recognized based on the text category of each text field. This method can extract structured information from non-fixed-format images and implement artificial intelligence-based image recognition, thereby improving document image recognition performance.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a document image recognition method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the development of computer and image processing technologies, technologies capable of detecting and recognizing text in images have emerged. For example, OCR (Optical Character Recognition) technology detects and recognizes text, converting unstructured image data into semi-structured data such as text. In actual image recognition technology, structured data is what is truly valuable to users. For example, when recognizing ID images, users are more concerned with and require recognition results for important fields such as names and ID numbers rather than simply text recognition results. However, traditional OCR structured recognition methods have poor recognition performance. Summary of the Invention

[0003] Based on this, it is necessary to provide a document image recognition method, device, computer equipment and storage medium to improve recognition performance in response to the above technical problems.

[0004] A document image recognition method, the method comprising:

[0005] Obtain the image to be recognized;

[0006] Detecting text in the image to be recognized, and identifying the text field of each detected text;

[0007] Performing category analysis on the text content and text position of each text field to determine the text category of each text field;

[0008] Based on the text category of each of the text fields, a document image recognition result of the image to be recognized is obtained.

[0009] In one embodiment, after acquiring the image to be recognized and before detecting text in the image to be recognized, the method further includes the step of performing image preprocessing on the image to be recognized.

[0010] In one embodiment, performing image preprocessing on the image to be recognized includes: correcting the angle of the image to be recognized.

[0011] In one embodiment, the method for training and obtaining the angle recognition model includes:

[0012] Get the initial sample image with a rotation angle of 0 degrees;

[0013] Performing sample expansion on the initial sample image to obtain an expanded sample, where the expanded sample includes the initial sample image and a sample obtained by rotating the initial sample image by a predetermined angle;

[0014] The expanded samples are used to train the angle recognition model to be trained to obtain a trained angle recognition model.

[0015] In one embodiment, when the extended sample is used to train the angle recognition model to be trained, the loss function is the square of the remainder of 360 degrees between the training recognition angle of the extended sample and the rotation angle corresponding to the extended sample.

[0016] In one embodiment, fusing the content code and the position code to obtain the node code includes: concatenating the content code and the position code to obtain the node code.

[0017] In one embodiment, based on the text category of each text field, field correction processing is performed on the text field, including: when the text category is an ID number, checking whether the ID number in the text content of the text field complies with the ID number rules; if it does not comply with the ID number rules, determining that the text field is recognized incorrectly or that the ID is illegal.

[0018] In one embodiment, based on the text category of each text field, field correction processing is performed on the text field, including: when the text category is a gender field, checking whether the text content of the text field belongs to a predetermined enumeration type; if not, correcting the text content of the text field.

[0019] In one embodiment, field correction processing is performed on the text fields based on the text categories of the text fields, including: when the text category is an address category, checking and error correction is performed on the text content of the text field based on an address library.

[0020] In one embodiment, based on the text category of each text field, field correction processing is performed on the text field, including: when the text category is the date of birth category, the text content of the text field is the first date information, and checking whether the first date information is earlier than the text content of the text field whose text category is the issuance date category. If not, it is determined that the document recognition is incorrect.

[0021] A document image recognition device, comprising:

[0022] An image acquisition module, used to acquire an image to be identified;

[0023] A text detection module, configured to detect text in the image to be recognized;

[0024] A text field recognition module, used to recognize the text fields of each text detected by the text detection module;

[0025] The classification module is used to perform classification analysis on the text content and text position of each text field to determine the text category of each text field;

[0026] The result determination module is used to obtain the document image recognition result of the image to be recognized based on the text category of each text field.

[0027] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0028] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0029] The above-mentioned document image recognition method, device, computer equipment and storage medium, after detecting and identifying the text field of the text in the image, perform category analysis processing based on the text content and text position of the identified text field to determine the text category of the text field, and on this basis determine the document image recognition result of the image to be identified. It determines the text category of the text field in combination with the text content and text position, and on this basis obtains the document image recognition result. It does not need to rely on the specific layout structure of the image with identification, so that it can realize the structured information extraction of non-fixed layout images, thereby improving the recognition performance of document images. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A diagram showing an application environment of a document image recognition method in one embodiment;

[0031] Figure 2 1 is a flow chart of a method for document image recognition in one embodiment;

[0032] Figure 3 is a flow chart of a document image recognition method in another embodiment;

[0033] Figure 4 A schematic diagram of the model structure of an angle recognition model in one embodiment;

[0034] Figure 5 Schematic diagram of a process for classification based on a graph neural network in one embodiment;

[0035] Figure 6 A schematic diagram of the principle of obtaining node codes in one embodiment;

[0036] Figure 7 A schematic diagram of the principle of classification performed by a graph neural network in one embodiment;

[0037] Figure 8 is a flowchart of a document image recognition method in a specific example;

[0038] Figure 9 A schematic diagram of an image to be recognized and a corresponding document image recognition result in an application example;

[0039] Figure 10 is a structural block diagram of a document image recognition device in one embodiment;

[0040] Figure 11 is a diagram of the internal structure of a computer device in one embodiment;

[0041] Figure 12 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0043] The document image recognition method provided in this application can be applied to Figure 1 In the application environment shown. Among them, the terminal 10 communicates with the server 20 through the network. The terminal 10 can obtain the image to be identified from the terminal locally or the server 20, and identify the document image recognition result in the image to be identified. During the identification process, when a network model is needed, such as the angle recognition model and graph neural network in the following embodiments, the network model can be obtained by training the terminal 10 itself, or it can be obtained by training the server 20 and then provided to the terminal 10. In some embodiments, the server 20 can also obtain the image to be identified from the terminal 10 and identify the document image recognition result in the image to be identified. During the identification process, when a network model is needed, such as the angle recognition model and graph neural network in the following embodiments, the network model can be obtained by training the server 20 itself, or it can be obtained by training the terminal 10 and then provided to the server 20 if necessary. Among them, the terminal 10 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 20 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0044] In some embodiments, the solutions of the embodiments of the present application may involve artificial intelligence computer vision technology, for example, identifying the category information and specific content of structured fields in document images through artificial intelligence.

[0045] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0046] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0047] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying, tracking, and measuring objects. This involves further processing the images, transforming them into images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0048] In one embodiment, Figure 2 、 Figure 3 As shown, a document image recognition method is provided, which is applied to Figure 1 The terminal 10 or the server 20 in the embodiment is used as an example to illustrate, including the following steps S201 to S204.

[0049] Step S201: Acquire an image to be recognized.

[0050] The image to be identified is the target image for which document image recognition is to be performed, and the image to be identified can be obtained through various possible means. In some embodiments, when the method is executed by a terminal, the terminal may obtain the image to be identified from the terminal device itself, a server, or another device. For example, the image to be identified may be obtained by real-time capture from a camera connected to the terminal device. In some embodiments, when the method is executed by a server, the server may obtain the image to be identified from the terminal device, from its own database, or from another server or database.

[0051] The obtained image to be identified may be in any possible image format, and the embodiment of the present application does not limit the image format of the image to be identified.

[0052] In one embodiment, after obtaining the image to be recognized and before entering into the subsequent detection of text in the image to be recognized, the following steps may also be included:

[0053] Step S2012: performing image preprocessing on the image to be recognized.

[0054] The image preprocessing performed on the image to be recognized may include any possible image preprocessing operation. In one specific example, the image preprocessing performed on the image to be recognized may include correcting the angle of the image to be recognized. The following example uses the image preprocessing process of correcting the angle of the image to be recognized as an example.

[0055] When correcting the angle of the image to be recognized, any possible angle correction method can be used, such as angle processing methods based on text jump characteristics (such as the MSER (Maximally Stable Extremal Regions) method, processing methods based on card edge line detection), and angle estimation methods based on deep learning. In the following specific examples of this application, the angle correction of the image to be recognized using the deep learning-based angle estimation method is used as an example.

[0056] In a specific example, correcting the angle of the image to be recognized may include the following steps 1 and 2.

[0057] Step 1: Using a pre-trained angle recognition model to identify the rotation angle of the image to be identified;

[0058] Step 2: Correct the image to be recognized based on the rotation angle to obtain a corrected image to be recognized.

[0059] When training to obtain an angle recognition model, machine learning techniques can be used to obtain the angle recognition model. Machine Learning (ML) is a multidisciplinary interdisciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory, and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning typically include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0060] refer to Figure 4 As shown, the model structure of the angle recognition model in one embodiment of the present application can be a model structure using CNN (Convolutional Neural Networks), and its typical structure includes an input layer, a convolution layer, a pooling layer, and a fully connected layer, wherein the input layer is used to perform standardization processing on the input image and process the input image into several matrices. The convolution layer is used to perform convolution processing, the purpose of which is to extract image features based on the matrix processed by the input layer. The pooling layer performs pooling processing to reduce the dimension of the image features processed by the convolution layer. The pooling processing can reduce the size of the convolution kernel while retaining the corresponding features to reduce the processing amount. The fully connected layer can perform dimensionality transformation and output the classification results. In actual technical applications, there can be multiple convolution layers and pooling layers. For example, after a convolution layer performs convolution processing, it is input to the next convolution layer for processing. After a pooling layer performs dimensionality reduction processing, it is input to the next convolution layer for convolution processing. The setting positions between each convolution layer and each pooling layer can also be set differently based on different technical needs, and the embodiments of the present application do not make specific limitations. As Figure 4 As shown, in the embodiment of the present application, the angle recognition model uses the above-mentioned image to be recognized as the model input, and after being processed by the angle recognition model, outputs the rotation angle of the image to be recognized identified by the angle recognition model.

[0061] In one embodiment, the method for training and obtaining the angle recognition model includes the following steps.

[0062] First, an initial sample image with a rotation angle of 0 degrees is obtained. The initial sample image with a rotation angle of 0 degrees refers to an ID image that has not been rotated. This initial sample image can be obtained through various possible methods, such as manually adjusting the ID image after obtaining it to obtain an initial sample image with a rotation angle of 0. In other embodiments, these initial sample images can also be obtained through other methods.

[0063] Secondly, the initial sample image is sample expanded to obtain expanded samples, and the expanded samples include the initial sample image and a sample obtained by rotating the initial sample image by a predetermined angle. Specifically, when rotating these initial sample images, the rotation can be performed in various possible ways, such as rotating the initial sample image by a predetermined angle through a data enhancement algorithm. The data enhancement algorithm can generate more training data from existing training samples. It uses a variety of random transformations that can generate credible images to increase samples. Its goal is to not view the same image twice during training, so that more data samples can be obtained, more content of the data can be observed, and better generalization ability can be obtained. Among them, the predetermined angle can specifically include more than one different angle. For example, for a specific initial sample image, it may be rotated by multiple different predetermined angles, thereby obtaining multiple different expanded samples.

[0064] Then, the angle recognition model to be trained is trained using the expanded samples to obtain a trained angle recognition model. When the angle recognition model to be trained is trained using the expanded samples, the expanded samples may be input into the recognition model to be trained, and the recognition model to be trained identifies the training recognition angle corresponding to each expanded sample, i.e., the rotation angle of the expanded sample identified by the recognition model to be trained. Based on the difference between the identified training recognition angle and the predetermined angle corresponding to the corresponding expanded sample, the model parameters of the recognition model to be trained are adjusted until the model training end condition is met. The predetermined angle corresponding to the expanded sample refers to the actual rotation angle of the expanded sample, which can be determined when the expanded sample is obtained. For example, for the initial sample image, the predetermined angle is 0, and for other images in the expanded sample, the corresponding predetermined angle is the corresponding rotation angle when the sample is expanded. The model training end condition may be that the difference between the training recognition angle obtained from the final training and the predetermined angle corresponding to the corresponding expanded sample is within an acceptable angle difference range, or it may be that a predetermined number of training times has been reached. In other embodiments, the model training end condition may also be set differently.

[0065] The angle recognition model, trained using the aforementioned method, expands from a small initial sample of images to a larger number of expanded samples, resulting in strong generalization capabilities. Furthermore, the angle recognition model can be trained using only a small sample size. This allows for accurate prediction of the angle of ID photos for various types of documents, achieving high prediction efficiency. The predicted rotation angle can be used to perform rotation correction on the image, facilitating subsequent detection and recognition.

[0066] In the process of training the angle recognition model to be trained using the expanded sample, the network can be trained using the back propagation algorithm through the angle regression loss function. In some embodiments, the minimum mean square error can be used as the minimum loss function. In a specific example of the present application, since the range of the angle value of the image rotation is 0-360°, its value exists symmetrically about 360°. For example, when the true angle is 0°, the predicted result is 359°, and its angle deviation should be less than 358°. Therefore, in one embodiment, the loss function can be set to the square of the difference between the training recognition angle of the expanded sample and the rotation angle corresponding to the expanded sample (that is, the above-mentioned predetermined angle) and the remainder of 360 degrees. The training recognition angle of the expanded sample to be recognized by the recognition model to be trained is recorded as predict, and the corresponding predetermined angle of the expanded sample is recorded as Target. The loss function of the embodiment of the present application can be expressed by the formula ((predict-Target) mod 360°) 2 , where mod represents the remainder operation.

[0067] Step 202: Detect text in the image to be recognized, and identify the text field of each detected text.

[0068] It is understood that when detecting text in an image to be recognized, in the case where the image is angle-corrected as described above, the text in the image to be recognized after the angle correction is detected. When detecting text in an image to be recognized, various possible methods can be used. In some embodiments, OCR text detection can be used, such as single-stage object detectors such as Yolo (You Only Look Once), SSD (Single Shot MultiBox Detector), dual-stage object detectors such as Faster RCNN (Faster Region-CNN), dedicated text detectors such as CTPN (Connectionist Text Proposal Network), and DB (Differentiable Binarization). During text detection, enhancement processing can also be performed on the document image to ensure that fields are well segmented.

[0069] When identifying the text fields of each detected text, various possible methods can be used to identify the text fields. In some embodiments, OCR text recognition can be used for recognition, such as text recognition methods based on CNN+RNN structure, attention mechanism-based structure, etc.

[0070] Step S203: performing category analysis on the text content and text position of each text field to determine the text category of each text field.

[0071] In some embodiments, the text content and text position of each text field are subjected to category analysis processing to determine the text category of each text field, including: using a graph neural network to process the text content and text position of each text field to obtain the text category of each text field.

[0072] Graph neural networks are a new type of artificial intelligence neural network that takes graph-structured data as input and outputs representation vectors that represent a high-level summary of property characteristics. Graph neural networks combine the text content and location of text fields for classification and recognition, rather than relying on specific image features. This reduces reliance on real-world samples, making image recognition more cost-effective and secure for highly sensitive data images like ID cards.

[0073] Among them, the graph neural network is used to process the text content and text position of each text field, and the text category of each text field is obtained by reference. Figure 5 As shown, it may include step S501 and step S502.

[0074] Step S501: for each identified text field, a node code corresponding to the text field is generated according to the text content and text position of the text field.

[0075] In one embodiment, when forming the node code, reference is made to Figure 6 As shown, the text content and the text position can be encoded separately, and then the two encoding results are merged to obtain the final node code. Specifically, for each identified text field, a node code corresponding to the text field is formed according to the text content and text position of the text field, including the following steps.

[0076] For the text content of the text field, the text content of the text field is encoded to obtain a content code. In a specific example, the text content can be encoded based on a natural language model, for example, a Bert natural language model or a word vector natural language model can be used. When encoding, by adopting the natural language model, each word in the text content is first encoded to obtain a word code of each word in the text content, and then the word codes of each word are fused. For example, based on the word sequence of each word, the word codes of each word are spliced ​​together to obtain a sentence code (i.e., content code) of the text content, that is, based on the word sequence of each word, the word codes of each word are spliced ​​together in a head-to-tail manner to obtain a sentence code of the text content.

[0077] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0078] For the text position of a text field, the text position of the text field can be normalized to a predetermined size, and then the normalized text position can be encoded to obtain a position code of a predetermined dimension. The predetermined size can be set according to actual needs. For example, it can be set to 1000*1000 in the following embodiment. That is, the text position of the text field is first normalized to a predetermined size of 1000*1000, and then encoded to a predetermined fixed dimension. Among them, the normalization method can be performed in any possible normalization method, and the embodiments of the present application do not specifically limit it.

[0079] The content code and the position code are merged to obtain the node code. In one embodiment, the content code and the position code are merged to obtain the node code, including: concatenating the content code and the position code to obtain the node code. For example, the content code and the position code are concatenated end to end to obtain the node code.

[0080] Step S502: Using the node codes of each text field as nodes of the graph neural network, the graph neural network is used to classify each node code to obtain the classification category of each text field.

[0081] Graph Neural Networks (GCNs) are used to classify specific categories of nodes based on their own features and their relationships with other nodes. The core idea is to learn a mapping through which nodes in the graph can aggregate their own and their neighbors' features to form a new representation of the node. Figure 7 As shown, specifically in the embodiment of the present application, the specific text category of each text field can be determined after classification and recognition by the graph neural network, and in the process of outputting the specific text category of each text field, useless text categories can be eliminated and not output, for example Figure 7 In the example shown, after analyzing 9 nodes, useful categories corresponding to four of the nodes can be output: name, gender, ID number, date of birth, etc.

[0082] In the embodiment of the present application, the graph neural network used is in the form of G = (V, E), where V represents a node, each text field corresponds to a node, the specific content of the node can be the node code of the text field, and E represents an edge. Each node is assumed to have an edge with itself, that is, (v, v) ∈ E. Use x ∈ R n*m Represents all vertices, where m is the characteristic dimension of each node. A represents the adjacency matrix of G, which is used to represent the connection relationship between nodes. D is the degree matrix, and the degree of each node refers to the number of nodes it connects.

[0083] When using GCN for classification, in the initial case, the graph G = (V, E) contains the information of each node, and there are edges between any two nodes, including the edges between each node and itself, and the edges between each node and other nodes. The weight of each edge will be assigned an initial value. In the process of GCN processing, the information of the neighbors in the graph can be obtained through a layer of convolution operation. For example, for a layer of GCN, the new k-dimensional vertex feature matrix is Where ρ is the activation function, X is the hidden layer, and W0 is the parameter matrix. In one embodiment, ReLU can be used as the activation function. In the specific process of using graph neural networks for processing, such convolution operations can have multiple operations, that is, the GCN can include multiple convolutional layers.

[0084] In the application embodiment, during the processing of GCN, the weights between nodes can be determined based on the distance between the nodes. Specifically, the farther the distance between the nodes, the smaller the weight. In some embodiments, the distance between nodes can be determined in combination with the position of the nodes in the image to be identified (it can be understood that in the case of angle correction, it refers to the image to be identified after angle correction). For example, the distance between two nodes can be determined in combination with the position of the two nodes in the image to be identified and the height and width of the image to be identified. In a specific example, the distance between two nodes can be determined using the following formula:

[0085]

[0086] Among them, l ij represents the distance between node i and node j, h max Indicates the height of the image to be recognized, w max Indicates the width of the image to be recognized, i x Indicates the position coordinate of node i on the x-axis, i y Indicates the position coordinate of node i on the y-axis, j x Indicates the position coordinate of node j on the x-axis, j y Represents the position coordinate of node j on the y-axis.

[0087] In a specific example of the present application, a two-layer GCN can be used to calculate the probability of each node belonging to a different category in a cross-entropy manner, thereby obtaining the text category of each text field.

[0088] For certificate images, in addition to content features, the OCR field categories of certificate images usually have a certain arrangement relationship. For example, Chinese and English names are usually close to each other, and there is usually a field name "date of birth" near the date of birth. Therefore, classification processing through graph neural networks can well capture this feature and improve the performance of certificate image recognition. Moreover, during the processing, it only relies on the text content and position, and does not use specific image features, reducing dependence on real samples. For highly sensitive data such as certificates, the cost is lower and it has better security. At the same time, during the processing, after identifying the text field of the text in the image, it relies on the text content and text position of the text field for classification, and does not rely on the specific layout structure of the image, so that it can better be compatible with different layouts of the same certificate and different types of certificates. Among them, the above-mentioned graph neural network can be a graph convolutional network or a graph attention network, etc., and the embodiments of this application do not make specific limitations.

[0089] Step S204: obtaining a document image recognition result of the image to be recognized based on the text category of each text field.

[0090] Based on the text categories of each text field identified above, the text categories of each text field and the corresponding text content can be output in a structured manner, for example, through information output such as tables and lists, so as to intuitively obtain various structured information data, which is convenient for subsequent applications of structured data.

[0091] In one embodiment, obtaining the document image recognition result of the image to be recognized based on the text category of each of the text fields includes:

[0092] Based on the text category of each text field, field correction processing is performed on the text field to obtain a final document image recognition result.

[0093] Field correction processing is a type of field post-processing, which is mainly used to correct the obtained recognition results and determine whether the certificate complies with the specifications. In some specific examples, field correction processing can include field-level post-processing and global post-processing. Among them, field-level post-processing refers to the correction processing of a single text field, verifying and correcting whether the text field complies with the corresponding rules, such as whether the certificate number complies with the certificate number rules, and whether the certificate gender complies with the enumeration type rules. Global post-processing refers to the correction processing of whether the text fields comply with the corresponding relationships, such as whether the date of birth is earlier than the date of issuance, when the certificate number contains gender information, whether the information encoding in the certificate number matches the certificate gender, etc.

[0094] In one embodiment, based on the text category of each text field, field correction processing is performed on the text field, including: when the text category is an ID number, verifying whether the ID number in the text content of the text field complies with the ID number rules; if it does not comply with the ID number rules, determining that the text field has been incorrectly recognized or that the ID is invalid. Because ID numbers are generally required to comply with certain verification rules, numbers that do not comply with the verification rules can be identified accordingly. For numbers that do not comply with the verification rules, it is possible that the identification error or the ID is invalid.

[0095] In one embodiment, based on the text category of each text field, field correction processing is performed on the text field, including: when the text category is a gender field, checking whether the text content of the text field belongs to a predetermined enumeration type, and if not, correcting the text content of the text field. For example, the gender field of a certificate is usually an enumeration type. Therefore, common errors of the enumeration type can be corrected through an error correction mechanism, such as candidate set correction. The specific error correction mechanism is not specifically limited in the embodiment of this application.

[0096] In one embodiment, field correction processing is performed on the text fields based on the text categories of the text fields, including: when the text category is an address category, checking and error correction is performed on the text content of the text field based on an address library.

[0097] In one embodiment, based on the text category of each text field, field correction processing is performed on the text field, including: when the text category is the date of birth category, the text content of the text field is the first date information, and checking whether the first date information is earlier than the text content of the text field whose text category is the issuance date category. If not, it is determined that the document recognition is incorrect.

[0098] Therefore, through the above-mentioned field correction processing, the overall recognition accuracy can be improved, and non-compliant documents can also be rejected.

[0099] The document image recognition method described above, after detecting and identifying the text field of the text in the image, performs category analysis processing based on the text content and text position of the identified text field to determine the text category of the text field, and on this basis determines the document image recognition result of the image to be identified. It determines the text category of the text field in combination with the text content and text position, and on this basis obtains the document image recognition result, thereby realizing the extraction of structured information of non-fixed format images and improving the recognition performance of document images.

[0100] The document image recognition method described above can be applied to any technical scenario where document images need to be recognized. Figure 8 As shown, an example of a document image recognition method in a specific application scenario is provided. Before actually recognizing document images, it is necessary to first train relevant models, such as angle recognition models and graph neural networks. The model training process can be performed by any device, such as a terminal or server, as long as the document image recognition device can obtain the model trained by the training device when ultimately performing document image recognition.

[0101] In one embodiment, when training an angle recognition model, an initial sample image with a rotation angle of 0 degrees is first obtained. This initial sample image is then rotated by a predetermined angle using a data augmentation algorithm to obtain an expanded sample. The angle recognition model is then trained on this expanded sample to obtain a trained angle recognition model. By expanding the initial sample image to a larger number of expanded samples based on a small number of samples, the trained angle recognition model has strong generalization capabilities. Furthermore, the angle recognition model can be trained using only a small number of samples, and can accurately predict the angle of ID photos for various types of documents, resulting in high prediction efficiency.

[0102] In one embodiment, when training to obtain a graph neural network, after obtaining a sample ID image, corresponding graph structure data can be obtained based on the sample ID image. Specifically, after obtaining the text content and text position of each text field of the sample ID image, the text content and text position are encoded respectively, and the node encoding of the text field is obtained in combination with the encoding of the text content and text position. Each text field is used as a node, and the node encoding corresponding to the text field is used as its node vector. The graph neural network to be trained is used for training. During each iterative training process, the parameters of the graph neural network are adjusted to update the weights determined based on the distance between each node until the training end condition is reached to obtain the trained graph neural network.

[0103] Assume that the image to be recognized is as follows: Figure 9-1 , 9-3, for Figure 9-1The image to be identified can be identified using the following process:

[0104] The angle recognition model obtained by the above training is used for image preprocessing. Specifically, after the angle recognition model is used to identify the rotation angle of the image to be identified, the angle of the image to be identified is corrected based on the identified rotation angle to obtain the corrected image to be identified.

[0105] Then, the text in the corrected image to be recognized is detected, and the text field of each detected text is identified, and the text content of each text field is encoded to obtain a content code. After the text position of the text field is normalized to a predetermined size, the normalized text position is encoded to obtain a position code of a predetermined dimension, and the content code and the position code are fused to obtain a node code for each text position.

[0106] Then, the node code of each text position is input into the graph neural network obtained by the above training, and classification processing is performed through the graph neural network to obtain the category of each text field.

[0107] Based on the obtained text field category, field correction processing is performed, such as whether the ID number conforms to the ID number rules, etc., to obtain the final ID recognition result. The final ID recognition result can be displayed in a list, table or any other possible way in combination with each field type for user viewing. Figure 9-1 The result of document image recognition is as follows Figure 9-2 As shown, the identified results include: document type (Type): Hong Kong Permanent Resident Identity Card; Chinese name (CnName): Li**; English name (EnName): LEE, **; Chinese telecommunications code (TelexCode): 2621********; gender (Sex): male M; date of birth (Birthday): 11-04-1989; card symbol (Symbol): ***AZ, etc.

[0108] Similarly, for Figure 9-2 The document image shown is processed in a similar manner as above to obtain Figure 9-4 The identification results shown include: document type (Type): Hong Kong Resident Identity Card; Chinese name (CnName): Guo**; English name (EnName): GUO,**; Chinese telecommunications code (TelexCode): 6753********; gender (Sex): female F; date of birth (Birthday): 01-10-1992; identification symbol (Symbol): CX, etc.

[0109] It should be understood that, although each step in each flow chart related in the above embodiment is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless clear instructions are arranged in this article, the execution of these steps does not have strict order restriction, and these steps can be performed in other order. Moreover, at least a portion of the steps in these flow charts can include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0110] In one embodiment, Figure 10 As shown, a document image recognition device is provided. The device can be a software module or a hardware module, or a combination of the two to form a part of a computer device. The device specifically includes:

[0111] An image acquisition module 101 is used to acquire an image to be recognized;

[0112] A text detection module 102 is used to detect text in the image to be recognized;

[0113] A text field identification module 103, configured to identify the text fields of each text detected by the text detection module;

[0114] The classification module 104 is used to perform classification analysis on the text content and text position of each text field to determine the text category of each text field;

[0115] The result determination module 105 is configured to obtain a document image recognition result of the image to be recognized based on the text category of each text field.

[0116] In one embodiment, the system further includes: an image preprocessing module configured to perform image preprocessing on the image to be identified;

[0117] The text detection module 102 detects text in the image to be recognized after the image preprocessing module performs image preprocessing.

[0118] In one embodiment, the image pre-processing module includes: an angle correction module for correcting the angle of the image to be recognized.

[0119] In one embodiment, the angle correction module uses a pre-trained angle recognition model to identify the rotation angle of the image to be recognized; and corrects the image to be recognized based on the rotation angle to obtain a corrected image to be recognized.

[0120] In one embodiment, an angle recognition model training module is further included, for training and obtaining the angle recognition model.

[0121] In one embodiment, the angle recognition model training module includes:

[0122] A sample acquisition module is used to obtain an initial sample image with a rotation angle of 0 degrees;

[0123] a sample expansion module, configured to perform sample expansion on the initial sample image to obtain an expanded sample, wherein the expanded sample includes the initial sample image and a sample obtained by rotating the initial sample image by a predetermined angle;

[0124] The training module is used to train the angle recognition model to be trained using the expanded samples to obtain a trained angle recognition model.

[0125] In the training module, when the expanded sample is used to train the training angle recognition model, the loss function is the square of the remainder of 360 degrees between the training recognition angle of the expanded sample and the rotation angle corresponding to the expanded sample.

[0126] In one embodiment, the classification module 104 is configured to process the text content and text position of each text field using a graph neural network to obtain a text category of each text field.

[0127] In one embodiment, the classification module 104 includes:

[0128] A node coding module is used to generate a node code corresponding to each identified text field according to the text content and text position of the text field;

[0129] The classification processing module is used to use the node codes of each text field as nodes of the graph neural network, adopt the graph neural network to classify each node code, and obtain the classification category of each text field.

[0130] In one embodiment, the node encoding module includes:

[0131] A content encoding module, configured to encode the text content of the text field to obtain a content code;

[0132] a position encoding module, configured to normalize the text position of the text field to a predetermined size, and then encode the normalized text position to obtain a position code of a predetermined dimension;

[0133] The coding fusion module is used to fuse the content coding and the position coding to obtain the node coding.

[0134] In one embodiment, the coding fusion module connects the content code and the position code to obtain the node code.

[0135] In one embodiment, the result determination module 105 performs field correction processing on the text fields based on the text categories of the text fields to obtain the final document image recognition result.

[0136] In one embodiment, when the text category is the ID number, the result determination module 105 checks whether the ID number in the text content of the text field complies with the ID number rules. If it does not comply with the ID number rules, it is determined that the text field is incorrectly recognized or the ID is illegal.

[0137] In one embodiment, when the text category is a gender field, the result determination module 105 checks whether the text content of the text field belongs to a predetermined enumeration type. If not, the result determination module 105 performs error correction on the text content of the text field.

[0138] In one embodiment, when the text category is an address category, the result determination module 105 performs verification and error correction on the text content of the text field based on an address library.

[0139] In one embodiment, when the text category is the date of birth category, the result determination module 105 verifies whether the first date information is earlier than the text content of the text field whose text category is the issuance date category. If not, it is determined that the document recognition is incorrect.

[0140] The specific definitions of the document image recognition device can be found in the definitions of the document image recognition method above and will not be repeated here. Each module in the document image recognition device described above may be implemented in whole or in part through software, hardware, or a combination thereof. Each of these modules may be embedded in or independent of a processor in a computer device in hardware form, or may be stored in a computer device memory in software form, allowing the processor to call and execute the corresponding operations of each module.

[0141] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 11As shown. The computer device includes a processor, memory, communication interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a document image recognition method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0142] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 12 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store relevant data, such as angle recognition models, graph neural networks, initial sample images, expanded samples, etc. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a document image recognition method is implemented.

[0143] Those skilled in the art will understand that Figure 11 、 12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0144] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0145] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0146] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0147] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0148] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A document image recognition method, characterized in that: The method comprises: Obtain the image to be recognized; Detecting text in the image to be recognized, and identifying the text field of each detected text; Encoding the text content of the text field to obtain a content code; encoding the text position of the text field to obtain a position code; fusing the content code and the position code to obtain a node code corresponding to the text field; The node codes corresponding to the text fields are used as nodes of the graph neural network. The two-layer graph neural network is used to classify the node codes, and the probability of each node belonging to a different category is calculated in a cross-entropy manner to obtain the classification category of each text field. Based on the text category of each of the text fields, a document image recognition result of the image to be recognized is obtained.

2. The method according to claim 1, characterized in that After acquiring the image to be recognized and before detecting the text in the image to be recognized, the method further includes the following steps: Correct the angle of the image to be recognized.

3. The method according to claim 2, characterized in that Correcting the angle of the image to be recognized includes: Using a pre-trained angle recognition model to identify the rotation angle of the image to be identified; The image to be recognized is corrected based on the rotation angle to obtain a corrected image to be recognized.

4. The method according to claim 1, wherein After acquiring the image to be recognized and before detecting the text in the image to be recognized, the method further includes the following steps: Perform image preprocessing on the image to be recognized.

5. The method according to claim 3, characterized in that The method for training and obtaining the angle recognition model includes: Get the initial sample image with a rotation angle of 0 degrees; Performing sample expansion on the initial sample image to obtain an expanded sample, where the expanded sample includes the initial sample image and a sample obtained by rotating the initial sample image by a predetermined angle; The expanded samples are used to train the angle recognition model to be trained to obtain a trained angle recognition model.

6. The method according to claim 1, characterized in that Encoding the text position of the text field to obtain a position code includes: After normalizing the text position of the text field to a predetermined size, the normalized text position is encoded to obtain the position code of a predetermined dimension.

7. The method according to claim 1, characterized in that Obtaining a document image recognition result of the image to be recognized based on the text category of each of the text fields includes: Based on the text category of each text field, field correction processing is performed on the text field to obtain a final document image recognition result.

8. The method according to claim 5, characterized in that In the process of training the training angle recognition model using the expanded samples, the loss function is the square of the remainder of 360 degrees between the training recognition angle of the expanded samples and the rotation angle corresponding to the expanded samples.

9. The method according to claim 8, characterized in that The loss function is: ((predict-Target)mod360°) 2 Among them, predict is the training recognition angle of the expanded sample to be recognized by the recognition model to be trained, Target is the corresponding predetermined angle of the expanded sample, and mod represents the remainder operation.

10. The method according to claim 1, characterized in that The content code and the position code are merged to obtain the node code, including: connecting the content code and the position code to obtain the node code.

11. The method according to claim 7, characterized in that Based on the text category of each text field, field correction processing is performed on the text field, including: When the text category is the ID number, the ID number in the text content of the text field is checked to see if it complies with the ID number rules. If it does not comply with the ID number rules, it is determined that the text field is incorrectly recognized or the ID is illegal.

12. The method according to claim 7, characterized in that Based on the text category of each text field, field correction processing is performed on the text field, including: When the text category is a gender field, it is checked whether the text content of the text field belongs to a predetermined enumeration type. If not, error correction is performed on the text content of the text field.

13. The method according to claim 7, characterized in that Based on the text category of each text field, field correction processing is performed on the text field, including: When the text category is an address category, the text content of the text field is verified and error corrected based on the address library.

14. The method according to claim 7, wherein: Based on the text category of each text field, field correction processing is performed on the text field, including: When the text category is the date of birth category, the text content of the text field is the first date information, and it is checked whether the first date information is earlier than the text content of the text field whose text category is the issuance date category. If not, it is determined that the document recognition is incorrect.

15. The method according to any one of claims 1 to 14, characterized in that In the process of classifying the node codes using the two-layer graph neural network, the weights between the nodes are determined according to the distance between the nodes. The distance between two nodes is determined using the following formula: Among them, l ij represents the distance between node i and node j, h max Indicates the height of the image to be recognized, w max Indicates the width of the image to be recognized, i x Indicates the position coordinate of node i on the x-axis, i y Indicates the position coordinate of node i on the y-axis, j x Indicates the position coordinate of node j on the x-axis, j y Represents the position coordinate of node j on the y-axis.

16. The method according to any one of claims 1 to 14, characterized in that In the process of outputting the specific text category of each text field, useless text categories are eliminated and not output.

17. A document image recognition device, characterized in that: The device comprises: An image acquisition module, used to acquire an image to be identified; A text detection module, configured to detect text in the image to be recognized; A text field recognition module, used to recognize the text fields of each text detected by the text detection module; A classification module is configured to encode the text content of the text field to obtain a content code; encode the text position of the text field to obtain a position code; fuse the content code and the position code to obtain a node code corresponding to the text field; use the node code corresponding to each text field as a node of a graph neural network, employ a two-layer graph neural network to classify each node code, calculate the probability of each node belonging to a different category using a cross entropy method, and obtain a classification category for each text field; The result determination module is used to obtain the document image recognition result of the image to be recognized based on the text category of each text field.

18. The device according to claim 17, characterized in that Also includes: An image preprocessing module, configured to perform image preprocessing on the image to be identified; The text detection module detects text in the image to be recognized after the image preprocessing module performs image preprocessing.

19. The device according to claim 18, characterized in that The image pre-processing module includes: an angle correction module, which is used to correct the angle of the image to be recognized.

20. The device according to claim 19, characterized in that The angle correction module uses a pre-trained angle recognition model to identify the rotation angle of the image to be recognized; and corrects the image to be recognized based on the rotation angle to obtain a corrected image to be recognized.

21. The device according to claim 20, characterized in that Also includes: The angle recognition model training module is used to train and obtain the angle recognition model.

22. The device according to claim 21, characterized in that The angle recognition model training module includes: A sample acquisition module is used to obtain an initial sample image with a rotation angle of 0 degrees; a sample expansion module, configured to perform sample expansion on the initial sample image to obtain an expanded sample, wherein the expanded sample includes the initial sample image and a sample obtained by rotating the initial sample image by a predetermined angle; The training module is used to train the angle recognition model to be trained using the expanded samples to obtain a trained angle recognition model.

23. The device according to claim 22, characterized in that In the training module, when the expanded sample is used to train the training angle recognition model, the loss function is the square of the remainder of 360 degrees between the training recognition angle of the expanded sample and the rotation angle corresponding to the expanded sample.

24. The device according to claim 17, wherein The classification module includes: A node encoding module is used to encode the text content of each identified text field to obtain a content code; encode the text position of the text field to obtain a position code; and fuse the content code and the position code to obtain a node code corresponding to the text field; The classification processing module is used to use the node codes corresponding to each of the text fields as nodes of the graph neural network, adopt two layers of the graph neural network to perform classification processing on each of the node codes, calculate the probability of each node belonging to a different category in a cross-entropy manner, and obtain the classification category of each text field.

25. The device according to claim 24, characterized in that The node coding module includes: A content encoding module, configured to encode the text content of the text field to obtain a content code; a position encoding module, configured to normalize the text position of the text field to a predetermined size, and then encode the normalized text position to obtain a position code of a predetermined dimension; The coding fusion module is used to fuse the content coding and the position coding to obtain the node coding.

26. The device according to claim 25, characterized in that The coding fusion module connects the content code and the position code to obtain the node code.

27. The device according to claim 17, characterized in that: The result determination module is further configured to perform field correction processing on the text fields based on the text categories of the text fields to obtain a final document image recognition result.

28. The device according to claim 27, characterized in that: The result determination module is used to check whether the ID number in the text content of the text field complies with the ID number rules when the text category is the ID number. If it does not comply with the ID number rules, it is determined that the text field is incorrectly recognized or the ID is illegal.

29. The device according to claim 27, characterized in that: The result determination module is used to check whether the text content of the text field belongs to a predetermined enumeration type when the text category is a gender field, and if not, perform error correction on the text content of the text field.

30. The device according to claim 27, characterized in that: The result determination module is configured to verify and correct errors in the text content of the text field based on an address library when the text category is an address category.

31. The device according to claim 27, characterized in that: The result determination module is used to verify whether the first date information is earlier than the text content of the text field whose text category is the issuance date category when the text category is the birth date category and the text content of the text field is the first date information. If not, it is determined that the document recognition is incorrect.

32. The device according to any one of claims 17 to 31, characterized in that: In the process of classifying the node codes using the two-layer graph neural network, the weights between nodes are determined according to the distance between the nodes. The distance between two nodes is determined using the following formula: Among them, l ij represents the distance between node i and node j, h max Indicates the height of the image to be recognized, w max Indicates the width of the image to be recognized, i x Indicates the position coordinate of node i on the x-axis, i y Indicates the position coordinate of node i on the y-axis, j x Indicates the position coordinate of node j on the x-axis, j y Represents the position coordinate of node j on the y-axis.

33. The device according to claim 23, characterized in that The loss function is: ((predict-Target)mod360°) 2 Among them, predict is the training recognition angle of the expanded sample to be recognized by the recognition model to be trained, Target is the corresponding predetermined angle of the expanded sample, and mod represents the remainder operation.

34. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 16 are implemented.

35. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

36. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

Images