A method and system for extracting element information of fixed-format target image

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to a method and system for extracting element information of a fixed-format target image. The specific steps are: acquiring the image information to be extracted, and identifying the contour and proportion of the acquired image information to obtain the target image; detecting the target image using a feature matching algorithm, calculating the offset of the target image, and confirming whether the target image needs to be rotated; using a super-resolution algorithm to clarify the target image, and positioning and identifying it; extracting fields from the recognition results, and cross-validating the recognition results using the association rules between big data and the extracted fields. This method can achieve higher accuracy and noise resistance for fixed-format text recognition in complex scenarios such as low resolution, blur, watermarks, and background text interference; it improves the accuracy in units of the number of bills, and at the same time gives a recognition error information prompt for bills that fail the verification, which is convenient for manual correction.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and relates to a method and a system for extracting element information of a fixed-format target image. Background Art

[0002] In the daily business scenarios of many enterprises such as commercial banks and real estate companies, a large number of documents and bills are generated. Among them, a lot of important valuable information needs to be entered into the database. If it relies on manual entry, there are problems such as low time efficiency, high labor costs, and poor accuracy. In today's artificial intelligence, using OCR technology to realize automatic information extraction has become a mainstream trend.

[0003] However, there are some challenges in OCR recognition of certificates and bills. On the one hand, in order to prevent information leakage or being used for theft for personal gain, watermarks are usually added to the target image, especially dark text watermarks, such as "For xxx use only, photocopying is invalid". These watermarks overlap with the text on the recognition object, making text recognition difficult. On the other hand, the quality (clarity) of the target image depends on the hardware conditions of the scanning machine, which will lead to uneven image clarity quality, which increases the difficulty of OCR information extraction of the target image. Most existing OCR recognition methods are suitable for high-resolution images and cannot be applied to the above scenarios. Summary of the invention

[0004] The present invention discloses a method and system for extracting information of fixed-format target image elements, so as to solve any of the above and other potential problems in the prior art.

[0005] In order to achieve the above technical solution, the technical solution of the present invention is: a method for extracting element information of a fixed-format target image, the method specifically comprising the following steps:

[0006] Step 1: Adaptive cutout to separate the foreground and background of the identified object. In actual scenes, the target image may have some complex backgrounds, such as text interference information of non-extracted elements, etc. Therefore, the object to be identified can be accurately extracted through cutout technology, which can eliminate the interference of background text and help us better utilize the relative position information of the identified elements. Usually, the accuracy of cutout based on contour detection is higher than feature matching. Therefore, the present invention adopts an adaptive cutout method, that is, cutout is first realized through contour detection, and feature matching is performed if it is unsuccessful. This method greatly improves the accuracy of cutout. Details are as follows:

[0007] a) Use digital processing techniques such as image convolution and morphology to extract contour information from images;

[0008] b) Determine the size or proportion of the extracted contour, and the one that meets the requirements is the target contour;

[0009] c) If all the extracted contour area or proportion conditions are not met, then use feature detection methods (SIFT feature detection, SURF feature detection, etc.) and homography matching to complete the cutout. The implementation idea is to make a matching template corresponding to the bill type, and use feature matching algorithms such as SIFT to calculate the transformation matrix between the template feature point set and the target image feature point set.

[0010] Through this transformation matrix, the perspective transformation of the identified object is realized, thereby completing the cutout.

[0011] Step 2: Use feature detection with rotation invariance to perform homography matching to achieve image direction correction. Implementation idea: By calculating the transformation matrix of the template feature point set and the front and back image feature point set, the offset of the upper left corner vertex coordinates of the template image is obtained. Assume that the width and height of the target image are w and h respectively. If the offset > min(h,w), the image is rotated 180 degrees.

[0012] Step 3: Locate the watermark position by matching the features of the watermark template and the target image, perform Kmeans clustering on the pixel values ​​in the watermark area, determine the watershed threshold between the text and watermark pixel values ​​by the difference between the text pixels and the watermark pixels, fill the area belonging to the watermark pixel value with the background pixel value, and then remove the watermark;

[0013] Step 4: Use GAN (Generative Adversarial Network) technology to achieve image enhancement and improve image clarity. Specific operation: Use the custom blind image quality evaluation index iqa_score =

[0014] w1*niqe_score+w2*brisque_score+w3*laplace_score (niqe, brisque, laplace are all known image quality evaluation indicators), select the clear target image, add blur, noise, etc. to the clear target image to generate a blurred target image. The blurred target image is used as the input of the GAN network model, and the clear target image is used as the network output, so as to conduct supervised learning network training, and rely on the training model to achieve the clarity of the target image

[0015] Step 5: For the target image, extract the rectangular frame to be identified according to the relative position of the target element;

[0016] Step 6: Use the text recognition model to perform text recognition on the rectangular frame image of the element to be recognized, and obtain the recognition result;

[0017] Step 7: Use big data to cross-validate the recognition results, and correct the information that is incorrectly recognized for some elements, so as to reduce the recognition error. For example, for address element information, the first three levels (province, city, district) of the recognition result can be matched with the three-level administrative address database, and the edit distance between each address in the address database and the recognition result can be calculated to calculate the error size, and the address with the smallest edit distance can be selected to correct the first three levels of the address;

[0018] Step 8: Use the joint correction method of factor information recognition to further ensure the accuracy of factor information extraction. Some identification factors are related to each other, such as the date of birth and ID number in personal information, and the capitalized amount and digital amount in the bill.

[0019] The beneficial effects of the present invention are as follows: the method can achieve higher accuracy and noise resistance for fixed plate text recognition in complex scenarios such as low resolution, blur, watermark, and background text interference; the accuracy per number of bills is improved, and at the same time, information prompts of recognition errors are given for bills that fail the verification, which is convenient for manual correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The present invention is a flowchart of a method for extracting element information of a fixed-format target image.

[0021] Figure 2 The present invention is a logic block diagram of a fixed-format target image element information extraction system.

[0022] Figure 3 The present invention is a flowchart for identifying an ID card using the method of the present invention.

[0023] Figure 4 The present invention is a flowchart for bill recognition using the method of the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0025] like Figure 1 As shown, the present invention provides a method for extracting element information of a fixed-format target image, the method specifically comprising the following steps:

[0026] S1) acquiring image information to be extracted, and identifying the contour and proportion of the acquired image information to obtain a target image;

[0027] S2) detecting the target image using a feature matching algorithm, calculating the offset of the target image, and determining whether the target image needs to be rotated according to the offset;

[0028] S3) using super-resolution algorithms to make the target image clear, and to locate and identify it;

[0029] S4) Use the association rules between big data and information to cross-validate the recognition results.

[0030] The method also includes a watermark removal step: judging whether a watermark exists in the target image obtained in S1) based on feature matching, locating the watermark if a watermark exists, separating the identification element information and the watermark information through pixel clustering, and removing the watermark information.

[0031] The steps are as follows:

[0032] The watermark position is located by matching the features of the watermark template and the target image.

[0033] Perform Kmeans clustering on the pixel values ​​in the watermark area, and determine the watershed threshold between the text and watermark pixel values ​​based on the difference between the text pixels and the watermark pixels;

[0034] The area belonging to the watermark pixel value is filled with the background pixel value to remove the watermark.

[0035] The specific steps of S1) are:

[0036] S1.1) Extracting contour information from images using image digital processing technology;

[0037] S1.1) Use image digital processing technology to extract contour information in the image. S1.2) If the area of ​​the extracted contour is between 0.1 and 0.95 times the area of ​​the image, it is considered to meet the requirements and is an image target. A perspective transformation is performed and S2) is executed; otherwise, S1.3) is executed;

[0038] S1.3) using a feature matching algorithm to calculate the transformation matrix between the template feature point set and the target image feature point set, using the transformation matrix to achieve the perspective transformation of the identified object, obtaining a corrected target image, and executing S3);

[0039] The specific steps of S2) are:

[0040] S2.1) Calculate the transformation matrix of the template feature point set and the target image feature point set using the feature matching algorithm,

[0041] S2.2) Obtain the offset of the coordinates of the upper left corner vertex of the target image according to the transformation matrix. Assume that the width and height of the target image are w and h respectively. If the offset > min(h,w), the target image is rotated 180 degrees; if the offset < min(h,w), the target image remains unchanged.

[0042] The specific steps of S3) are:

[0043] S3.1) processing the target image processed by S2.2) using an image quality evaluation method to obtain a fused blind image quality evaluation index value;

[0044] The image quality evaluation method refers to calculating the NIQE, BRISQUE, and LAPLACE index values ​​of the image, normalizing them, and using the average value as the quality evaluation index.

[0045] S3.2) Compare the fusion blind image quality evaluation index value obtained in S3.1) with the threshold. If it is greater than or equal to the threshold, it is a clear target image and execute S3.4) Otherwise, it is a blurred target image and execute S3.3)

[0046] S3.3) performs a clearing process on the blurred target image obtained in S3.2):

[0047] In S3.2), the blurred target image is used as the input of the GAN network model, and the clear target image is used as the network output to achieve the clarity of the blurred target image, and then S3.4) is executed.

[0048] S3.4) Use clear target images for positioning and identification.

[0049] The specific steps of S4) are:

[0050] S4.1) extracting keyword fields according to the recognition results of S3.4);

[0051] S4.2) Identify the content of the keyword field extracted in S4.1), and match the identification result with the information in the history library. If the match is successful, it is used as the identification result; if the match is unsuccessful, similarity calculation is performed, and correction is completed to obtain the identification result;

[0052] S4.3) performing rule mapping according to the recognition result to obtain the final result;

[0053] S4.4) If the mapping of some recognition results is empty, return to S4.2) to obtain results from other fields with associated relationships through association rules.

[0054] The regular mapping includes format mapping, truncation mapping and arithmetic mapping.

[0055] like Figure 2 As shown, the present invention provides an extraction system for the above-mentioned fixed-format target image element information extraction method, the extraction system comprising:

[0056] The information acquisition module is used to acquire the image information to be extracted, and to identify the outline and proportion of the acquired image information to obtain the target image;

[0057] A preprocessing module is used to detect the target image using a feature matching algorithm, calculate the offset of the target image, and determine whether the target image needs to be rotated according to the offset;

[0058] The positioning and recognition module is used to use the super-resolution algorithm to make the target image clear, and to locate and recognize it;

[0059] The verification module is used to cross-validate the recognition results by using the association rules between big data and information.

[0060] The verification module includes a semantic correction module and an information mapping and rule verification module;

[0061] The semantic correction module includes a regularized regular correction method and a nearest neighbor correction method for comparison with the information database; the information mapping and rule verification module maps the corrected answer information into the range or format required by the business according to certain rules, and at the same time, data verification can be performed on the relevant information according to the rules (the ID card number and gender, birth, and address are verified against each other), and these rules usually need to be obtained from the metadata database.

[0062] The system also includes: the metadata database stores the rule information of each field, mainly including the value domain or value range of the field, relative position and keywords; the information database includes the historical data of the field to be identified, the data information crawled from the network or a third party and the data obtained from the current identification task;

[0063] The extraction system also includes: a watermark recognition module, which is used to determine whether a watermark exists in the target image based on feature matching, and if a watermark exists, complete the watermark positioning, and separate the identification element information and the watermark information through the pixel clustering method to remove the watermark information.

[0064] An information processing terminal for implementing the above-mentioned method for extracting element information of a fixed-format target image.

[0065] A computer-readable storage medium includes instructions, which, when executed on a computer, enable the computer to execute the above-mentioned method for extracting element information of a fixed-format target image.

[0066] Embodiment 1:

[0067] like Figure 3 As shown, taking identity card element recognition as an example, the specific steps of the technical solution are explained.

[0068] Step 1: Use digital image processing and other technologies to perform convolution operations on the ID card image (Figure a) to obtain Figure b, and obtain Figure c through contour detection and perspective transformation;

[0069] Step 2: Use SIFT feature detection to calculate the image offset and rotate the image to obtain Figure D;

[0070] Steps 3 and 4: Locate the watermark position, perform kmeans clustering on the pixel values ​​in the watermark area, and divide them into three categories (watermark, text to be recognized, and background, where the pixel value corresponding to the background is the highest and the pixel value corresponding to the text to be recognized is the lowest), and replace the pixel value in the middle with the background pixel value. Then use the gan network to achieve image clarity, and get Figure E;

[0071] Step 5: Extract text lines at relative positions. As shown in Figure f, extract the red frame area according to the relative proportion to obtain the text frame of each recognition element.

[0072] Step 6: Perform text recognition on each text box to obtain the recognition result;

[0073] Steps 7 and 8: Through big data cross-validation and joint correction of element information, the recognition results are post-processed and optimized, specifically:

[0074] a) The first three levels (province, city, district) of the address are matched with the three-level administrative address database, and the error of the edit distance between each address in the address database and the recognition result can be calculated, and the address with the smallest edit distance is selected to correct the first three levels of the address;

[0075] b) Date of birth - ID card number correction: In the ID card information, according to the ID card number coding rules, the numbers corresponding to the 7th to 14th digits correspond to the date of birth one by one, so they can be corrected with each other; c) ID card address - issuing authority correction: In the ID card information, there is a certain correlation between the ID card address and the issuing authority. When one of the elements cannot be effectively identified, the other element can be used for correction;

[0076] d) Gender - ID card number correction: In the ID card information, according to the ID card number coding rules, the number corresponding to the 17th digit corresponds to the gender, odd numbers represent males, and even numbers represent females, so they can be corrected against each other.

[0077] e) Ethnicity - Ethnic dictionary correction: In the identity card information, the ethnic information can be verified and corrected based on the ethnic dictionary database.

[0078] 2> Mutual correction of validity period and starting time: In the ID card information, the validity period element is connected by two time points: effective and expiration. There is a specific logical relationship between the two time points. For example, the expiration node is either "-long-term" or is separated from the effective node by an integer number of years. The two ends of the validity period can be corrected accordingly.

[0079] Embodiment 2:

[0080] like Figure 4 As shown, an example of extracting invoice element information is provided, which explains the specific steps of the technical solution.

[0081] Step 1: Using digital image processing and other technologies, we first perform convolution operations on the invoice image (Figure A) to extract the image contour information. It is found that the contour area does not meet the requirements, so the SIFT feature detection method is used to cut out the image. Figure B is the selected template, and the cut image C is obtained;

[0082] Step 2: SIFT feature detection has rotation-invariant features, so the direction has been corrected;

[0083] Step 3: This example usually does not contain watermarks in the actual scene, so this step is skipped;

[0084] Step 4: Use the GAN network to make the image clearer and get Figure D (since the image itself has good clarity, the results are almost the same)

[0085] Step 5: Extract text lines at relative positions, as shown in Figure f, extract the red frame area according to the relative proportion, and obtain the text frame of each recognition element, as shown in Figure g;

[0086] Step 6: Perform text recognition on each text box to obtain the recognition result;

[0087] Steps 7 and 8: Through big data cross-validation and joint correction of element information, the recognition results are post-processed and optimized, specifically:

[0088] a) The QR code in the upper left corner of the invoice can be located and the recognition result can be cross-verified by calling the QR code recognition interface;

[0089] b) Jointly correct the recognition results through the arithmetic logic relationship among the amount, tax amount and total amount.

[0090] The above is a detailed introduction to a method and system for extracting information of a fixed-format target image element provided by an embodiment of the present application. The description of the above embodiment is only used to help understand the method and its core idea of ​​the present application; at the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0091] If certain words are used in the specification and claims to refer to specific components, hardware manufacturers may use different nouns to refer to the same component. This specification and claims do not use differences in names as a way to distinguish components, but use differences in the functions of components as the criteria for distinction. As mentioned throughout the specification and claims, "including" and "comprising" are open-ended terms, so they should be interpreted as "including / including but not limited to". "Approximately" means that within an acceptable error range, those skilled in the art can solve the technical problem within a certain error range and basically achieve the technical effect. The subsequent description of the specification is a preferred embodiment of the present application, but the description is for the purpose of illustrating the general principles of the present application, and is not used to limit the scope of the present application. The scope of protection of the present application shall be determined by the attached claims.

[0092] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a product or system. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the product or system including the elements.

[0093] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0094] The above description shows and describes several preferred embodiments of the present application, but as mentioned above, it should be understood that the present application is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the application concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art do not deviate from the spirit and scope of the present application, and should be within the scope of protection of the claims attached to the present application.

Claims

1. A method for extracting element information of a fixed-format target image, characterized in that: The method specifically comprises the following steps: S1) acquiring image information to be extracted, and identifying the contour and proportion of the acquired image information to obtain a target image; The specific steps are: S1.1) Use image digital processing technology to extract contour information in the image; S1.2) If the area of ​​the extracted contour is between 0.1 and 0.95 times the area of ​​the image, it is considered to meet the requirements and is an image target. A perspective transformation is performed and S2) is executed; otherwise, S1.3) is executed; S1.3) using a feature matching algorithm to calculate the transformation matrix between the template feature point set and the target image feature point set, using the transformation matrix to achieve the perspective transformation of the identified object, obtain a corrected target image, and execute S3); S2) detecting the target image using a feature matching algorithm, calculating the offset of the target image, and determining whether the target image needs to be rotated according to the offset; The specific steps are: S2.1) using feature matching algorithm to calculate the transformation matrix of the template feature point set and the target image feature point set, S2.2) Obtain the offset of the coordinates of the upper left corner vertex of the target image according to the transformation matrix. Assume that the width and height of the target image are w and h respectively. If the offset is greater than min(h, w), the target image is rotated 180 degrees; if the offset is less than min(h, w), the target image remains unchanged. S3) using super-resolution algorithms to make the target image clear, and to locate and identify it; S4) Use the association rules between big data and information to cross-validate the recognition results.

2. The method according to claim 1, characterized in that The method also includes a watermark removal step: judging whether a watermark exists in the target image obtained in S1) based on feature matching, and if no watermark exists, executing S2); if a watermark exists, completing watermark positioning, removing the watermark information, and then executing S2).

3. The method according to claim 2, characterized in that The steps are as follows: locating the watermark position by matching the features of the watermark template and the target image, performing Kmeans clustering on the pixel values ​​of the watermark area, and determining the watershed threshold between the text pixel value and the watermark pixel value through the difference between the text pixel value and the watermark pixel value; filling the area belonging to the watermark pixel value with the background pixel value to achieve watermark removal.

4. The method according to claim 1, characterized in that: The specific steps of S3) are: S3.1) the target image processed by S2.2) is processed by an image quality evaluation method to obtain a fused blind image quality evaluation index value; S3.2) the fused blind image quality evaluation index value obtained by S3.1) is compared with a threshold value, if it is greater than or equal to the threshold value, it is a clear target image, and S3.4) is executed; otherwise, it is a blurred target image, and S3.3) is executed; S3.3) the blurred target image confirmed by S3.2) is used as the input of the GAN network model, and the clear target image is used as the network output to achieve the clarity of the blurred target image; S3.4) the clear target image confirmed by S3.2) and the clear target image obtained by S3.3) are located and identified.

5. The method according to claim 1, characterized in that The specific steps of S4) are: S4.1) extracting keyword fields according to the recognition results of S3.4); S4.2) Identify the content of the keyword field extracted in S4.1), and match the identification result with the information in the history library. If the match is successful, it is used as the identification result; if the match is unsuccessful, similarity calculation is performed, and correction is completed to obtain the identification result; S4.3) performing rule mapping according to the recognition result to obtain the final result; S4.4) If the mapping of some recognition results is empty, return to S4.2) to obtain results from other fields with associated relationships through association rules.

6. An extraction system for the fixed-format target image element information extraction method according to any one of claims 1 to 5, characterized in that: The extraction system includes: an information acquisition module, which is used to acquire image information to be extracted, and identify the contour and proportion of the acquired image information to obtain a target image; a preprocessing module, which is used to detect the target image using a feature matching algorithm, calculate the offset of the target image, and confirm whether the target image needs to be rotated according to the offset; a positioning and recognition module, which is used to clarify the target image using a super-resolution algorithm, and perform positioning and recognition; and a verification module, which is used to cross-verify the recognition result using the association rules between big data and information.

7. The extraction system according to claim 6, characterized in that The extraction system also includes: a watermark recognition module, which is used to determine whether a watermark exists in the target image based on feature matching, and if a watermark exists, complete the watermark positioning, and separate the identification element information and the watermark information through the pixel clustering method to remove the watermark information.

8. A computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute a method for extracting element information of a fixed-format target image as described in any one of claims 1 to 5.

Images