Certificate information automatic identification and extraction method and related equipment thereof

Abstract

The embodiment of the invention belongs to the technical field of image information processing, is applied to a certificate information automatic extraction scene, and relates to a certificate information automatic identification and extraction method and related equipment thereof, and the method comprises the steps: obtaining a target certificate image to be subjected to information extraction; uniformly extracting a text information identification template and an image information identification template in the model through certificate information; extracting text data and image data; screening out to-be-corrected text content; correcting the to-be-corrected text content to obtain corrected text content; the image data, the text data and the corrected text content are output, and information extraction of the target certificate image is completed. According to the method and the device, the certificate information unified extraction model is learned and trained, and the information identification template and the extraction text correction strategy are introduced, so that high-accuracy information extraction can be carried out on certificates of dozens of certificate types and formats on the premise of not knowing the certificate types, manpower is effectively saved, and the method and the device are more intelligent and automatic.

Description

Technical Field

[0001] The present application relates to the field of image information processing technology, and is applied to scenarios of automatic extraction of certificate information, and in particular to a method for automatic identification and extraction of certificate information and related equipment. Background Art

[0002] In financial business scenarios such as insurance, claims, and bank loans, customers are often required to provide various information, and in many cases, it is necessary to identify the relevant certificate images provided by the customer. Since the types of certificates are not uniform, there may be dozens of certificate types involved, and each certificate type may have multiple layouts. Considering the wide range of products and scenarios involved and the huge business volume, relying entirely on manual review will consume a lot of manpower, and because the types of certificates submitted are not standardized and unified, it will cause confusion in customer information management, leading to serious business problems such as duplicate insurance.

[0003] Existing solutions usually develop a targeted extraction method for a specific type of certificate. Since there are many types of certificates that can provide information, and the type of certificate is not disclosed when the customer submits the certificate image, it may not be applicable. Therefore, there is currently a lack of a solution that can automatically extract information from the target certificate without distinguishing the certificate type in advance. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a method for automatically identifying and extracting certificate information and related equipment to solve the problem that the prior art lacks a solution for automatically extracting information from a target certificate without distinguishing the certificate type in advance.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a method for automatically identifying and extracting certificate information, which adopts the following technical solution:

[0006] A method for automatically identifying and extracting certificate information comprises the following steps:

[0007] Obtaining a target document image for information extraction;

[0008] Inputting the target document image into a preset document information unified extraction model, wherein the document information unified extraction model contains information recognition templates corresponding to document images of different formats and categories as prior knowledge;

[0009] Identify text features, image features, and position features of the target document image using the document information unified extraction model;

[0010] Filtering out a text information recognition template and an image information recognition template corresponding to the target document image according to the text features, image features and position features;

[0011] Identify and extract text data in the target document image based on the text information recognition template;

[0012] Identify and extract image data from the target document image based on the image information recognition template;

[0013] Filtering out the text content to be corrected contained in the text data through a preset filtering list;

[0014] Correcting the text content to be corrected based on a preset correction strategy to obtain corrected text content;

[0015] The image data, the uncorrected text content in the text data, and the corrected text content are output to complete the information extraction of the target document image.

[0016] In order to solve the above technical problems, the embodiment of the present application also provides an automatic identification and extraction device for certificate information, which adopts the following technical solution:

[0017] A device for automatically identifying and extracting certificate information, comprising:

[0018] A target document image acquisition module is used to acquire the target document image to be extracted;

[0019] A target certificate image input module, used to input the target certificate image into a preset certificate information unified extraction model, wherein the certificate information unified extraction model contains information recognition templates corresponding to certificate images of different formats and categories as prior knowledge;

[0020] A feature recognition module, used to recognize text features, image features and position features of the target document image through the document information unified extraction model;

[0021] An information recognition template screening module, used to screen out a text information recognition template and an image information recognition template corresponding to the target document image according to the text features, image features and position features;

[0022] A text data extraction module, used to identify and extract text data in the target document image based on the text information recognition template;

[0023] An image data extraction module, used to identify and extract image data from the target document image based on the image information recognition template;

[0024] A text content screening module to be corrected, used to screen out the text content to be corrected contained in the text data through a preset screening list;

[0025] A text content correction module to be corrected, used to correct the text content to be corrected based on a preset correction strategy to obtain the corrected text content;

[0026] The information output module is used to output the image data, the uncorrected text content in the text data, and the corrected text content to complete the information extraction of the target document image.

[0027] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0028] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned method for automatic identification and extraction of certificate information when executing the computer-readable instructions.

[0029] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0030] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the method for automatic identification and extraction of certificate information as described above.

[0031] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0032] The method for automatic identification and extraction of certificate information described in the embodiment of the present application obtains the target certificate image to be extracted; uses the text information identification template and the image information identification template in the certificate information unified extraction model; extracts text data and image data; screens out the text content to be corrected; corrects the text content to be corrected to obtain the corrected text content; outputs the image data, text data and the corrected text content to complete the information extraction of the target certificate image. The present application learns and trains the certificate information unified extraction model, introduces information identification templates and extracted text correction strategies, and realizes high-accuracy information extraction of dozens of certificate types and formats without knowing the certificate type, effectively saving manpower and being more intelligent and automated. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0035] Figure 2 is a flow chart of an embodiment of a method for automatically identifying and extracting certificate information according to the present application;

[0036] Figure 3 It is a flowchart of an embodiment of learning and training the unified extraction model of certificate information in the method for automatic identification and extraction of certificate information described in the present application;

[0037] Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 305 is shown;

[0038] Figure 5 yes Figure 3 A flowchart of a specific embodiment of step 306 is shown;

[0039] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 203 is shown;

[0040] Figure 7 It is a structural schematic diagram of an embodiment of a device for automatically identifying and extracting certificate information according to the present application;

[0041] Figure 8 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0043] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0044] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0045] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0046] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0047] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0048] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0049] It should be noted that the method for automatic identification and extraction of certificate information provided in the embodiment of the present application is generally executed by a server, and accordingly, the device for automatic identification and extraction of certificate information is generally set in the server.

[0050] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0051] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for automatically identifying and extracting certificate information according to the present application. The method for automatically identifying and extracting certificate information comprises the following steps:

[0052] Step 201, obtaining a target document image for information extraction.

[0053] In this embodiment, the target certificate image to be extracted includes the certificate image to be extracted uploaded by the user terminal. The target certificate image includes various business scenarios in the financial field, such as insurance, claims, bank loan processing, etc., which often require customers to provide various certificate images, such as: identity card image, business license image, real estate certificate image, marriage certificate image, vehicle registration image, driver's license image, etc. The automatic identification and extraction method of certificate information described in the present application, by acquiring the target certificate image to be extracted, combined with a preset unified extraction model of certificate information, can automatically extract image information from the target certificate image, which is more intelligent and assists relevant financial institutions to efficiently extract image information.

[0054] Step 202: input the target certificate image into a preset certificate information unified extraction model, wherein the certificate information unified extraction model contains information recognition templates corresponding to certificate images of different formats and categories as prior knowledge.

[0055] In this embodiment, the information recognition template includes a text information recognition template and an image information recognition template. Due to different categories of certificates, such as identity card images, business license images, real estate certificate images, marriage certificate images, vehicle registration images, driver's license images, etc., the text data, image data, and text position information contained therein are different. Moreover, even for certificates of the same category, due to different formats, such as the second-generation resident identity card images and the first-generation resident identity card images, the text data, image data, and text position information contained therein are different. Therefore, the information recognition templates corresponding to certificate images of different formats and categories are respectively deployed as prior knowledge into the unified certificate information extraction model, which can facilitate the extraction of certificate image information and improve data extraction efficiency.

[0056] In this embodiment, the unified certificate information extraction model includes an input image tilt angle prediction component, a text feature extraction component, an image feature extraction component and a feature encoding component.

[0057] In this embodiment, before executing the step of inputting the target document image into the preset document information unified extraction model, the method further includes the step of training the document information unified extraction model. In essence, the step of training the document information unified extraction model aims to learn the knowledge mapping relationship between the document category and document format and the image feature information, and the knowledge mapping relationship between the document category and document format and the text feature information, so as to construct information recognition templates corresponding to document images of different categories and formats, and deploy them in the document information unified extraction model.

[0058] Continue to refer Figure 3 , Figure 3 It is a flowchart of an embodiment of learning and training the unified extraction model of certificate information in the automatic identification and extraction method of certificate information described in the present application, including:

[0059] Step 301, obtaining a batch of certificate images, wherein the batch of certificate images includes certificate images of different categories and formats;

[0060] Step 302, inputting the batch of certificate images into the certificate information unified extraction model;

[0061] Step 303, training the input image tilt angle prediction component through the batch of document images to obtain a trained input image tilt angle prediction component, and obtaining tilt angles corresponding to all images in the batch of document images;

[0062] Specifically, since the acquisition of ID images is mostly uploaded by user terminals, users are required to take photos themselves before uploading, or financial industry salespersons take photos and upload them. Therefore, the uploaded ID images are often not aligned with the shooting frame. In this embodiment, before information is extracted from the ID image, an input image tilt angle prediction component is first set up to facilitate rapid prediction of the tilt angle of the input ID image in subsequent use, and timely direction correction before information extraction, so as to facilitate subsequent efficient extraction of data information in the ID image.

[0063] In this embodiment, the step of training the input image tilt angle prediction component specifically includes: identifying the line where the text is located, calculating the angle between the line where the text is located and a preset horizontal line, and using the angle as the tilt angle.

[0064] Step 304, performing direction correction on the input batch of certificate images according to the tilt angles corresponding to all images in the batch of certificate images, to obtain batch of certificate images after direction correction;

[0065] By performing direction correction on the input batch of certificate images, the batch of certificate images after direction correction are obtained, which facilitates the subsequent extraction of data information in the certificate images.

[0066] Step 305, extracting text features from the batch of certificate images after orientation correction by the text feature extraction component to obtain text feature information, wherein the text feature information includes text features and position features, the text features include text field content, and the position features include position information of each text field content;

[0067] In this embodiment, the text feature extraction component includes a text extraction component based on OCR text recognition technology. Specifically, the position information can be obtained according to the frame diagram corresponding to each text field content, and the coordinates of the four vertices of the frame are obtained, and the coordinates of the four vertices of the frame are used to represent the position information. Correspondingly, the coordinates of the center point of the frame can also be obtained according to the frame diagram corresponding to each text field content, and the coordinates of the center point of the frame are used to represent the position information. Of course, the distance information of the text field content from the top, bottom, left, and right of the four sides in the entire document image frame can also be used to represent the position information.

[0068] Continue to refer Figure 4 , Figure 4 yes Figure 3 The flowchart of a specific embodiment of step 305 shown includes:

[0069] Step 401, inputting the batch of certificate images after direction correction into the text extraction component based on OCR text recognition technology;

[0070] Step 402, extracting the text field contents in all the certificate images respectively by using the text extraction component based on the OCR text recognition technology;

[0071] Step 403, identifying the position information of the text field contents in all the certificate images in the corresponding certificate images respectively through the text extraction component based on the OCR text recognition technology;

[0072] Step 404, grouping and sorting the text field content and the position information to sort out each text field content and the position information corresponding to each text field content;

[0073] In this embodiment, the text extraction component based on the OCR text recognition technology is only used to extract the content of each text field and the position information corresponding to the content of each text field.

[0074] Step 405, input the grouping results one by one into a preset embedding layer, and convert through the embedding layer to obtain the word vectors, one-dimensional position vectors of the words and two-dimensional position vectors of the words corresponding to different grouping results;

[0075] Step 406, accumulating the word vector, the one-dimensional position vector of the word, and the two-dimensional position vector of the word corresponding to the same grouping result to obtain a text vector;

[0076] Step 407: Use the text vector as text feature information of the text field content in the corresponding grouping and sorting result.

[0077] Specifically, the content of each text field and the position information corresponding to each text field content are extracted and input into a preset embedding layer, and converted through the embedding layer to obtain the corresponding word vector, the one-dimensional position vector of the word, and the two-dimensional position vector of the word, that is, the extracted text field content and position information are subjected to feature vector encoding processing, so as to obtain the text vector representation corresponding to each text field content, and the text vector representation is used as the text feature information of the text field content. The text content information and position information are vectorized, and feature vector encoding processing is introduced, so that the corresponding text vector representation can be obtained directly through decoding processing in the subsequent process.

[0078] Step 306, extracting image features from the batch of certificate images after direction correction through the image feature extraction component to obtain image feature information;

[0079] Continue to refer Figure 5 , Figure 5 yes Figure 3 The flowchart of a specific embodiment of step 306 shown includes:

[0080] Step 501, uniformly scaling the batch of document images after the direction correction to the extraction frame size of the image feature extraction component;

[0081] Step 502, segmenting the scaled batch of document images respectively, and position coding the segmented image blocks, wherein the position coding of the segmented image blocks follows the principle that the coding characters gradually increase from top to bottom and from left to right, and the coding characters are 1 to N, N is a positive integer, and the maximum value of N is the number of the image blocks;

[0082] Step 503, inputting the segmented image blocks into a preset linear mapping layer for linear mapping, and outputting a linear mapping sequence according to the position code;

[0083] Step 504, performing vectorization processing on the linear mapping sequence to obtain a vectorization processing result as an image vector corresponding to the document image;

[0084] Step 505: Use the image vector as image feature information of the corresponding document image.

[0085] Similarly, the extracted image blocks are linearly mapped to obtain a linear mapping sequence, which is then vectorized. A feature vector encoding method is introduced to encode the image features, so that the corresponding image vector representation can be obtained directly through subsequent decoding.

[0086] Step 307, grouping the text feature information and the image feature information of the certificate images of the same category and the same format to obtain the grouping result of the batch of certificate images, wherein the grouping result of the batch of certificate images is different according to the different categories and formats of the certificate images;

[0087] Step 308, inputting the grouping result of the batch of certificate images into the feature coding component to be trained, training the feature coding component, and obtaining a trained feature coding component;

[0088] In this embodiment, the feature encoding component includes a Transformer-based feature encoding component, and the Transformer-based feature encoding component can perform multimodal feature encoding, that is, it can encode text features and image features.

[0089] Specifically, a random masking method is adopted to input the grouping results of the batch of certificate images into the feature coding component to be trained, and the feature coding component is trained to obtain a trained feature coding component, wherein the random masking method includes randomly masking a certain proportion of text feature information in the grouping results and / or randomly masking a certain proportion of image feature information in the grouping results when training the feature coding component.

[0090] In this embodiment, a random masking method is adopted to randomly mask a certain proportion of text feature information in the grouped sorting results and / or randomly mask a certain proportion of image feature information in the grouped sorting results, for example: randomly masking 30% of the text feature information and randomly masking 40% of the image feature information, thereby realizing masked text modeling and masked image modeling, and then obtaining corresponding multimodal feature encoding and decoding components through training, so that in subsequent actual use, even if the input target document image is missing, the fixed image data and fixed text data in the document can be recognized as much as possible.

[0091] Step 309, obtaining feature encoding results corresponding to the document images of different categories and formats output by the trained feature encoding component;

[0092] Step 310 , decoding the feature coding results corresponding to the certificate images of the same category and format, thereby obtaining the image information and text information corresponding to the certificate images of the same category and format.

[0093] In this embodiment, after executing the step of decoding the feature encoding results corresponding to the certificate images of the same category and the same format to obtain the image information and text information corresponding to the certificate images of the same category and the same format, the method also includes: constructing a knowledge mapping relationship between the certificate category and the certificate format and the image feature information based on the image information corresponding to the certificate images of the same category and the same format; generating image information recognition templates corresponding to certificate images of different formats and categories based on the knowledge mapping relationship between the certificate category and the certificate format and the image feature information; constructing a knowledge mapping relationship between the certificate category and the certificate format and the text feature information based on the text information corresponding to the certificate images of the same category and the same format; generating text information recognition templates corresponding to certificate images of different formats and categories based on the knowledge mapping relationship between the certificate category and the certificate format and the text feature information.

[0094] In essence, based on the processing of the batch of certificate images, the knowledge mapping relationship between the certificate category and certificate format and the image feature information, as well as the knowledge mapping relationship between the certificate category and certificate format and the text feature information are learned, so as to construct information recognition templates corresponding to certificate images of different categories and formats, and deploy them in the unified certificate information extraction model. For practical application, after obtaining the text feature information and image feature information of the target certificate image, the corresponding information recognition template can be directly screened out for text data recognition and image data recognition, so as to facilitate subsequent rapid extraction based on the recognition results.

[0095] Step 203: Identify the text features, image features and position features of the target certificate image through the certificate information unified extraction model.

[0096] Continue to refer Figure 6 , Figure 6 yes Figure 2 The flowchart of a specific embodiment of step 203 shown includes:

[0097] Step 601, using the trained input image tilt angle prediction component to predict the tilt angle of the target document image;

[0098] Step 602, performing direction correction on the target document image based on the tilt angle to obtain a target document image after direction correction;

[0099] Step 603, extracting text features from the target document image after orientation correction by the text feature extraction component to obtain text feature information, wherein the text feature information includes text features and position features, the text features include text field content, and the position features include position information of each text field content;

[0100] Step 604: extract image features from the target document image after direction correction through the image feature extraction component to obtain image feature information.

[0101] The text feature extraction component is used to extract text features from the target document image after orientation correction to obtain text feature information, and the image feature extraction component is used to extract image features from the target document image after orientation correction to obtain image feature information, so as to facilitate subsequent information recognition template screening and screen out text information recognition templates and image information recognition templates corresponding to the target document image.

[0102] Step 204: Filter out a text information recognition template and an image information recognition template corresponding to the target document image according to the text features, image features and position features.

[0103] In this embodiment, the step of screening out the text information recognition template and the image information recognition template corresponding to the target document image according to the text features, image features and position features specifically includes: identifying the text information recognition template corresponding to the target document image according to the knowledge mapping relationship between the text feature information, document category and document format and the text feature information; identifying the image information recognition template corresponding to the target document image according to the knowledge mapping relationship between the image feature information, document category and document format and the image feature information.

[0104] Specifically, directly based on the text feature information and image feature information of the target document image, the corresponding information recognition template is screened out to perform text data recognition and image data recognition, so as to facilitate subsequent rapid extraction based on the recognition results.

[0105] In this embodiment, after executing the step of selecting the text information recognition template and the image information recognition template corresponding to the target certificate image according to the text features, image features and position features, the method further includes: if the text information recognition template and the image information recognition template corresponding to the target certificate image cannot be selected according to the text features, image features and position features, amplifying the target certificate image by image amplification technology to obtain a batch of target certificate images; inputting the batch of target certificate images into the certificate information unified extraction model; continuing to execute steps 303 to 310 to obtain image information and text information corresponding to the batch of target certificate images; constructing a knowledge mapping relationship between the category and layout of the target certificate image and the image feature information and the text feature information respectively according to the image information and text information corresponding to the batch of target certificate images; generating an image information recognition template corresponding to the target certificate image according to the knowledge mapping relationship between the category and layout of the target certificate image and the image feature information; and generating a text information recognition template corresponding to the target certificate image according to the knowledge mapping relationship between the category and layout of the target certificate image and the text feature information.

[0106] In essence, an incremental update method is adopted. When extracting information from a new document image, a corresponding image information recognition template and a text information recognition template can be constructed for the new document image, so that when the new document image is subsequently subjected to information recognition and extraction, the image information recognition template and the text information recognition template can be directly screened for information recognition and extraction.

[0107] Step 205: Identify and extract text data in the target document image based on the text information recognition template.

[0108] Specifically, the text information recognition template can be understood as, based on the processing of step 202, according to the text information extraction results, constructing text information recognition templates for different versions and categories of certificate images respectively, which can identify the location of the text in the certificate image, thereby quickly identifying and extracting text data.

[0109] Step 206: Identify and extract image data from the target document image based on the image information recognition template.

[0110] Correspondingly, the image information recognition template can be understood as, based on the processing of step 202, according to the image information extraction results, constructing image information recognition templates for different versions and categories of certificate images respectively, which can identify the location of the image in the certificate image, thereby quickly identifying and extracting image data.

[0111] Step 207: Filter out the text content to be corrected contained in the text data through a preset filter list.

[0112] In this embodiment, the preset filtering list is provided with identification information of the text content to be corrected.

[0113] Specifically, the step of filtering out the text content to be corrected contained in the text data through a preset filtering list includes: determining whether the text data contains identification information of the text content to be corrected; if the text data contains the identification information, identifying the text content to be corrected corresponding to the identification information; if the text data does not contain the identification information, the text content to be corrected contained in the text data is a NULL value.

[0114] Specifically, for example: company name, legal person information, etc., because it involves changes in company name or legal person information, when extracting information from previous business licenses, this application provides a correction plan, the purpose of which is to make the extracted text content information the changed information, thereby improving the accuracy of text information extraction.

[0115] Step 208: correct the text content to be corrected based on a preset correction strategy to obtain corrected text content.

[0116] In this embodiment, the step of correcting the text content to be corrected based on a preset correction strategy to obtain the corrected text content specifically includes: determining whether the text content to be corrected is a NULL value; if the text content to be corrected is a NULL value, outputting the NULL value as the corrected text content; if the text content to be corrected is not a NULL value, obtaining the authoritative text content corresponding to the text content to be corrected from a preset authoritative information source, and correcting the text content to be corrected according to the authoritative text content to obtain the corrected text content.

[0117] Specifically, the step of correcting the text content to be corrected according to the authoritative text content to obtain the corrected text content specifically includes: calculating the edit distance between the authoritative text content and the text content to be corrected by using the Levenshtein distance algorithm; judging whether the edit distance exceeds a preset distance threshold by comparison; if the edit distance exceeds the preset distance threshold, directly replacing the text content to be corrected with the authoritative text content to obtain the corrected text content; if the edit distance does not exceed the preset distance threshold, identifying the difference text between the authoritative text content and the text content to be corrected, and correcting the text content to be corrected by using the difference text to obtain the corrected text content.

[0118] Specifically, the Levenshtein distance algorithm is an edit distance algorithm, which mainly calculates the edit distance between two compared strings based on the difference between the two strings. It can be understood as the correction step required to convert a string into the string to be compared with it. If the step is large, the authoritative text content is directly used to replace the text content to be corrected. If the step is small, the difference text can be considered to correct the text content to be corrected.

[0119] Step 209: output the image data, the uncorrected text content in the text data, and the corrected text content to complete the information extraction of the target document image.

[0120] This application can extract information from dozens of types and formats of certificates with high accuracy without knowing the type of certificate, effectively saving manpower and being more intelligent and automated.

[0121] This application completes the information extraction of the target document image by acquiring the target document image to be extracted; extracting text data and image data through the text information recognition template and image information recognition template in the document information unified extraction model; screening out the text content to be corrected; correcting the text content to be corrected to obtain the corrected text content; and outputting the image data, text data, and corrected text content. This application learns and trains the document information unified extraction model, introduces information recognition templates and extracted text correction strategies, and realizes high-accuracy information extraction for dozens of document types and formats without knowing the document type, effectively saving manpower and being more intelligent and automated.

[0122] The embodiments of the present application can acquire and process relevant data based on image processing technology. Among them, image processing (Artificial Intelligence, AI) is the theory, method, technology and application system of using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0123] Image processing basic technologies generally include sensors, dedicated image processing chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. Image processing software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0124] In the embodiment of the present application, the target document image to be extracted is obtained; the text information recognition template and the image information recognition template in the unified document information extraction model are used; the text data and the image data are extracted; the text content to be corrected is screened out; the text content to be corrected is corrected to obtain the corrected text content; the image data, the text data and the corrected text content are output to complete the information extraction of the target document image. The present application learns and trains the unified document information extraction model, introduces information recognition templates and extracted text correction strategies, and realizes high-accuracy information extraction of dozens of document types and formats without knowing the document type, effectively saving manpower and being more intelligent and automated.

[0125] Further references Figure 7 , as a response to the above Figure 2 The present application provides an embodiment of a device for automatically identifying and extracting certificate information. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0126] like Figure 7 As shown, the automatic identification and extraction device 700 of certificate information described in this embodiment includes: a target certificate image acquisition module 701, a target certificate image input module 702, a feature recognition module 703, an information recognition template screening module 704, a text data extraction module 705, an image data extraction module 706, a to-be-corrected text content screening module 707, a to-be-corrected text content correction module 708 and an information output module 709. Among them:

[0127] The target document image acquisition module 701 is used to acquire the target document image to be extracted;

[0128] The target document image input module 702 is used to input the target document image into a preset document information unified extraction model, wherein the document information unified extraction model contains information recognition templates corresponding to document images of different formats and categories as prior knowledge;

[0129] A feature recognition module 703 is used to recognize text features, image features and position features of the target document image through the document information unified extraction model;

[0130] The information recognition template screening module 704 is used to screen out the text information recognition template and the image information recognition template corresponding to the target document image according to the text features, image features and position features;

[0131] A text data extraction module 705, configured to identify and extract text data in the target document image based on the text information recognition template;

[0132] An image data extraction module 706, configured to identify and extract image data from the target document image based on the image information recognition template;

[0133] The to-be-amended text content screening module 707 is used to screen out the to-be-amended text content contained in the text data through a preset screening list;

[0134] The text content correction module 708 is used to correct the text content to be corrected based on a preset correction strategy to obtain the corrected text content;

[0135] The information output module 709 is used to output the image data, the uncorrected text content in the text data, and the corrected text content to complete the information extraction of the target document image.

[0136] This application completes the information extraction of the target document image by acquiring the target document image to be extracted; extracting text data and image data through the text information recognition template and image information recognition template in the document information unified extraction model; screening out the text content to be corrected; correcting the text content to be corrected to obtain the corrected text content; and outputting the image data, text data, and corrected text content. This application learns and trains the document information unified extraction model, introduces information recognition templates and extracted text correction strategies, and realizes high-accuracy information extraction for dozens of document types and formats without knowing the document type, effectively saving manpower and being more intelligent and automated.

[0137] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0138] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0139] To solve the above technical problems, the present application also provides a computer device. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.

[0140] The computer device 8 includes a memory 8a, a processor 8b, and a network interface 8c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 8 having components 8a-8c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0141] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0142] The memory 8a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 8a can be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 8a can also be an external storage device of the computer device 8, such as a plug-in hard disk equipped on the computer device 8, a smart memory card (Smart Med i aCard, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 8a can also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 8a is generally used to store the operating system and various application software installed on the computer device 8, such as a computer-readable instruction of a method for automatic identification and extraction of certificate information. In addition, the memory 8a can also be used to temporarily store various types of data that have been output or are to be output.

[0143] The processor 8b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 8b is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 8b is used to run the computer-readable instructions stored in the memory 8a or process data, such as running the computer-readable instructions of the document information automatic identification and extraction method.

[0144] The network interface 8c may include a wireless network interface or a wired network interface. The network interface 8c is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0145] The computer device proposed in this embodiment belongs to the field of image information processing technology and is applied to the scenario of automatic extraction of certificate information. This application obtains the target certificate image to be extracted; uses the text information recognition template and the image information recognition template in the unified certificate information extraction model; extracts text data and image data; screens out the text content to be corrected; corrects the text content to be corrected to obtain the corrected text content; outputs the image data, text data and the corrected text content to complete the information extraction of the target certificate image. This application learns and trains the unified certificate information extraction model, introduces information recognition templates and extracted text correction strategies, and realizes high-accuracy information extraction for dozens of certificate types and formats without knowing the certificate type, effectively saving manpower and being more intelligent and automated.

[0146] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the above-mentioned method for automatic identification and extraction of certificate information.

[0147] The computer-readable storage medium proposed in this embodiment belongs to the field of image information processing technology and is applied to the scenario of automatic extraction of certificate information. This application obtains the target certificate image to be extracted; uses the text information recognition template and the image information recognition template in the unified certificate information extraction model; extracts text data and image data; screens out the text content to be corrected; corrects the text content to be corrected to obtain the corrected text content; and outputs the image data, text data, and corrected text content to complete the information extraction of the target certificate image. This application learns and trains the unified certificate information extraction model, introduces information recognition templates and extracted text correction strategies, and realizes high-accuracy information extraction for dozens of certificate types and formats without knowing the certificate type, effectively saving manpower and being more intelligent and automated.

[0148] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0149] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for automatically identifying and extracting certificate information, characterized in that: The steps include: Obtaining a target document image for information extraction; Inputting the target document image into a preset document information unified extraction model, wherein the document information unified extraction model contains information recognition templates corresponding to document images of different formats and categories as prior knowledge; Identify text features, image features, and position features of the target document image using the document information unified extraction model; Filtering out a text information recognition template and an image information recognition template corresponding to the target document image according to the text features, image features and position features; Identify and extract text data in the target document image based on the text information recognition template; Identify and extract image data from the target document image based on the image information recognition template; Filtering out the text content to be corrected contained in the text data through a preset filtering list; Correcting the text content to be corrected based on a preset correction strategy to obtain corrected text content; The image data, the uncorrected text content in the text data, and the corrected text content are output to complete the information extraction of the target document image.

2. The method for automatic identification and extraction of certificate information according to claim 1, characterized in that: The document information unified extraction model includes an input image tilt angle prediction component, a text feature extraction component, an image feature extraction component and a feature encoding component. Before executing the step of inputting the target document image into the preset document information unified extraction model, the method further includes: Acquire a batch of certificate images, wherein the batch of certificate images includes certificate images of different categories and formats; Inputting the batch of certificate images into the certificate information unified extraction model; The input image tilt angle prediction component is trained by the batch of document images to obtain a trained input image tilt angle prediction component, and the tilt angles corresponding to all images in the batch of document images are obtained; Directionally correcting the input batch of document images according to the tilt angles corresponding to all images in the batch of document images to obtain the batch of document images after direction correction; The text feature extraction component extracts text features from the batch of certificate images after direction correction to obtain text feature information, wherein the text feature information includes text features and position features, the text features include text field content, and the position features include position information of each text field content; The image feature extraction component extracts image features from the batch of certificate images after direction correction to obtain image feature information; Grouping the text feature information and the image feature information of the certificate images of the same category and the same format to obtain the grouping result of the batch of certificate images, wherein the grouping result of the batch of certificate images is different according to the different categories and formats of the certificate images; Inputting the grouping result of the batch of certificate images into the feature coding component to be trained, training the feature coding component, and obtaining the trained feature coding component; Obtaining feature encoding results corresponding to document images of different categories and formats output by the trained feature encoding component; By decoding the feature encoding results corresponding to the certificate images of the same category and the same format, the image information and text information corresponding to the certificate images of the same category and the same format are obtained.

3. The method for automatic identification and extraction of certificate information according to claim 2, characterized in that: The text feature extraction component includes a text extraction component based on OCR text recognition technology. The step of extracting text features from the batch of document images after direction correction by the text feature extraction component to obtain text feature information specifically includes: Inputting the batch of certificate images after direction correction into the text extraction component based on OCR text recognition technology; Extract the text field contents in all the certificate images respectively by using the text extraction component based on OCR text recognition technology; Identify the position information of the text field contents in all the certificate images in the corresponding certificate images by using the text extraction component based on the OCR text recognition technology; The text field content and the position information are grouped and sorted to obtain each text field content and the position information corresponding to each text field content; The grouping results are input into the preset embedding layer one by one, and converted through the embedding layer to obtain the word vectors, one-dimensional position vectors and two-dimensional position vectors of the words corresponding to different grouping results; Accumulate the word vector, the one-dimensional position vector of the word, and the two-dimensional position vector of the word corresponding to the same grouping result to obtain the text vector; The text vector is used as text feature information of the text field content in the corresponding grouping and sorting result.

4. The method for automatic identification and extraction of certificate information according to claim 2, characterized in that: The step of extracting image features from the batch of certificate images after direction correction by the image feature extraction component to obtain image feature information specifically includes: Uniformly scaling the batch of document images after the direction correction to the extraction frame size of the image feature extraction component; The scaled batch of certificate images are segmented respectively, and the segmented image blocks are position-coded, wherein the position coding of the segmented image blocks follows that the coding characters gradually increase from top to bottom and from left to right, and the coding characters are 1 to N, N is a positive integer, and the maximum value of N is the number of the image blocks; Inputting the segmented image blocks into a preset linear mapping layer for linear mapping, and outputting a linear mapping sequence according to the position code; Performing vectorization processing on the linear mapping sequence to obtain a vectorization processing result as an image vector corresponding to the document image; The image vector is used as image feature information of the corresponding document image.

5. The method for automatic identification and extraction of certificate information according to claim 2, characterized in that: After executing the step of decoding the feature encoding results corresponding to the certificate images of the same category and the same format to obtain the image information and text information corresponding to the certificate images of the same category and the same format, the method further includes: According to the image information corresponding to the certificate images of the same category and the same format, a knowledge mapping relationship between the certificate category and certificate format and the image feature information is constructed; According to the knowledge mapping relationship between the document category and document format and the image feature information, image information recognition templates corresponding to document images of different formats and categories are generated; According to the text information corresponding to the certificate images of the same category and the same format, a knowledge mapping relationship between the certificate category and certificate format and the text feature information is constructed; According to the knowledge mapping relationship between the document category and document format and text feature information, text information recognition templates corresponding to document images of different formats and categories are generated.

6. The method for automatic identification and extraction of certificate information according to claim 2, characterized in that: The step of identifying the text features, image features and position features of the target document image by using the document information unified extraction model specifically includes: Predicting the tilt angle of the target document image using the trained input image tilt angle prediction component; Performing direction correction on the target document image based on the tilt angle to obtain a target document image after direction correction; The text feature extraction component extracts text features from the target document image after direction correction to obtain text feature information, wherein the text feature information includes text features and position features, the text features include text field content, and the position features include position information of each text field content; The image feature extraction component extracts image features from the target document image after direction correction to obtain image feature information.

7. The method for automatic identification and extraction of certificate information according to claim 5, characterized in that: The step of selecting a text information recognition template and an image information recognition template corresponding to the target document image according to the text features, image features and position features specifically includes: Identify a text information recognition template corresponding to the target document image according to the knowledge mapping relationship between the text feature information, document type and document format and the text feature information; According to the knowledge mapping relationship between the image feature information, the certificate type and certificate format and the image feature information, an image information recognition template corresponding to the target certificate image is identified.

8. The method for automatic identification and extraction of certificate information according to claim 1, characterized in that: The preset filtering list is provided with identification information of the text content to be corrected. The step of filtering out the text content to be corrected contained in the text data through the preset filtering list specifically includes: Determining whether the text data contains identification information of the text content to be corrected; If the text data contains the identification information, identifying the text content to be corrected corresponding to the identification information; If the text data does not contain the identification information, the text content to be corrected contained in the text data is a NULL value; The step of correcting the text content to be corrected based on a preset correction strategy to obtain the corrected text content specifically includes: Determine whether the text content to be corrected is a NULL value; If the text content to be corrected is a NULL value, outputting a NULL value as the corrected text content; If the text content to be revised is not a NULL value, the authoritative text content corresponding to the text content to be revised is obtained from a preset authoritative information source, and the text content to be revised is revised according to the authoritative text content to obtain the revised text content.

9. A device for automatically identifying and extracting certificate information, characterized in that: include: A target document image acquisition module is used to acquire the target document image to be extracted; A target certificate image input module, used to input the target certificate image into a preset certificate information unified extraction model, wherein the certificate information unified extraction model contains information recognition templates corresponding to certificate images of different formats and categories as prior knowledge; A feature recognition module, used to recognize text features, image features and position features of the target document image through the document information unified extraction model; An information recognition template screening module, used to screen out a text information recognition template and an image information recognition template corresponding to the target document image according to the text features, image features and position features; A text data extraction module, used to identify and extract text data in the target document image based on the text information recognition template; An image data extraction module, used to identify and extract image data from the target document image based on the image information recognition template; A text content screening module to be corrected, used to screen out the text content to be corrected contained in the text data through a preset screening list; A text content correction module to be corrected, used to correct the text content to be corrected based on a preset correction strategy to obtain the corrected text content; The information output module is used to output the image data, the uncorrected text content in the text data, and the corrected text content to complete the information extraction of the target document image.

10. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the method for automatic identification and extraction of certificate information as described in any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the method for automatic identification and extraction of certificate information as described in any one of claims 1 to 8.

Images