Certificate layout analysis method based on deep learning and traditional algorithms

Abstract

The present invention discloses a certificate layout analysis method based on deep learning and traditional algorithms, comprising obtaining four coordinates of edge corner points of a machine-readable code area of a certificate layout, aligning the machine-readable code area of the certificate layout to be collected using a standard certificate size; extracting the visual reading area coordinate position using a preset coordinate point of a standard size, and identifying characters in the visual reading area of the certificate layout using a target detection algorithm; adjusting the external frame of each recognition area among the preset coordinate points of the standard size using a bottom-up padding method; extracting the accurate area requiring character recognition, feeding it into a convolutional recurrent neural network structure for recognition, and obtaining a final character recognition result; and matching a known column area with the character recognition result to obtain a final structured output result.

Description

Technical Field

[0001] The present invention relates to the field of text recognition technology, and in particular to a certificate layout analysis method based on deep learning and traditional algorithms. Background Art

[0002] In conventional document recognition, the accuracy of document recognition is greatly challenged by the complex layout of documents. Currently, there are two methods for structured document output:

[0003] With template matching, when pre-processing is required, try to straighten the document as much as possible, and then use the prepared standard parts to directly extract the framed area. This method is only suitable for strictly coordinated shooting. When the document is incomplete or distorted, the output result cannot be accurately obtained, which can easily lead to problems with structured output recognition. The method based on direct template matching is to strictly process the document in the early stage, such as perspective change and affine transformation, to make the document to be identified into a document with a size and area that strictly matches the template. This method needs to find four key coordinate points to stretch and deform the document during use. The larger the area covered by each key point, the better the image standardization effect. However, when the key points of some passport documents are not obvious, the above method is easy to fail. The image to be matched is distorted, the matching result will be missing content, and the structured output will fail.

[0004] The algorithm based on deep learning directly identifies the small print column of the ID item, and then obtains the structured output result of the ID identification content based on the distribution of text content on each type of ID. The method requires high hardware computing power and has high usage costs. At the same time, when the ID photo is not clear enough and the small print column recognition fails, the structured output of the relevant ID information fails. In detail, due to the limitations of the shooting conditions, the small print column is easily lost during recognition. After the column is lost, the recognized content cannot find the corresponding column affiliation, and the output recognition result is missing. For example, in the recognition of the small print column of the Korean passport, due to the blurry photo of the ID, the small print recognition confidence is low, which directly affects the final recognition result. Summary of the Invention

[0005] One purpose of the present invention is to provide a passport layout analysis method based on deep learning and traditional algorithms. Based on the advantages of deep learning, traditional algorithms are integrated to perform passport layout analysis to achieve accurate output of document recognition results.

[0006] Other advantages and features of the present invention will become more apparent from the following detailed description and will be realized by means of the instrumentalities and combinations particularly pointed out in the appended claims.

[0007] According to one aspect of the present invention, a method for analyzing the layout of a certificate based on deep learning and traditional algorithms, which can achieve the aforementioned objectives and other objectives and advantages, includes:

[0008] Obtain the four coordinates of the edge corner points of the machine-readable code area of the certificate page, and align the machine-readable code area of the certificate page to be collected using the size of the standard certificate;

[0009] Using the coordinate positions of the visual reading area of the preset coordinate points of a standard size, extracting the visual reading area, and using an object detection algorithm to identify the characters in the visual reading area of the certificate page;

[0010] Use the bottom-up padding method to adjust the bounding box of each recognition area among the preset coordinate points of standard size;

[0011] Extract the exact area where character recognition is required and feed it into the convolutional recurrent neural network structure for recognition to obtain the final character recognition result;

[0012] The known column areas of the standard certificate are matched with the character recognition results to obtain the final structured output result.

[0013] According to one embodiment of the present invention, after adjusting the bounding box of each recognition area among the preset coordinate points of the standard size, it also includes: obtaining the accurate bounding box of the recognition area by moving the bounding box of the character area to be recognized in the visual reading area.

[0014] According to one embodiment of the present invention, before obtaining the four coordinates of the edge corner points of the machine-readable code area of the certificate page, the collected biometric information page of the certificate page is preprocessed, the machine-readable code area is data enhanced, and character recognition of the machine-readable code area is performed.

[0015] According to one embodiment of the present invention, after character recognition of the machine-readable code area is performed, the country and type of the certificate page are confirmed based on the character information of the machine-readable code area.

[0016] According to one embodiment of the present invention, when extracting the accurate area where character recognition is required, Gaussian filtering, grayscale image conversion, and median filtering are first performed on the character area to be recognized.

[0017] According to one embodiment of the present invention, after the character area to be identified is subjected to Gaussian filtering, grayscale image conversion, and median filtering, the character area to be identified is subjected to Sauvola binarization, erosion and dilation, and projection algorithm processing.

[0018] According to one embodiment of the present invention, the step of extracting the recognition area through the coordinate information of preset coordinate points of standard size includes: obtaining the coordinates of the machine-readable code area of the image to be processed in the entire image; using the four vertex coordinates of the machine-readable code to perform affine transformation and straighten the image; resizing the original image according to the width and height of the machine-readable code to be processed and the aspect ratio of the standard machine-readable code; at the same time, generating the coordinates of the standard machine-readable code area in the entire image; cropping the redundant edges of the image and filling the missing edges with pixels.

[0019] According to one embodiment of the present invention, the coordinate frame of the passport visual reading area is adjusted in the order starting from the closest distance to the machine readable code area and then proceeding upwards.

[0020] According to one embodiment of the present invention, when performing median filtering processing, a (2n+1)(2n+1) window is selected (n is a value greater than or equal to 1), and the window is used to slide along each pixel from left to right and from top to bottom in the row and column directions of the image data; each time it slides, the grayscale of the pixels in the window is sorted, and the grayscale value of the pixel at the center of the window is replaced by the middle value.

[0021] According to one embodiment of the present invention, when performing projection algorithm processing, horizontal projection is performed by accumulating pixels of the binarized image in the y-axis direction, and then dividing each row of characters by a threshold value; vertical projection is performed by accumulating pixels of the binarized image in the x-axis direction, and then dividing each character by a threshold value, and at the same time removing the blank areas without characters on the left and right ends, thereby generating the final character recognition result.

[0022] This invention builds on the advantages of deep learning and integrates traditional algorithms to perform passport layout analysis to achieve accurate output of document recognition results, thereby accurately achieving structured output. Even if the document is not clear enough or there is distortion, it can still accurately achieve structured output recognition results. With the development of deep learning methods, combined with traditional image processing, the pixel-by-pixel analysis of the document layout has brought a new level of improvement to the document recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is an algorithm flow chart of a certificate layout analysis method based on deep learning and traditional algorithms according to an embodiment of the present invention.

[0024] Figure 2 This is a flowchart of the steps for extracting and identifying areas in the certificate layout analysis method based on deep learning and traditional algorithms in the above embodiment of the present invention.

[0025] Figure 3This is a flowchart of the steps in the coordinate frame adjustment sequence of the certificate layout analysis method based on deep learning and traditional algorithms in the above embodiment of the present invention.

[0026] Figure 4 This is a flowchart of the steps for extracting the accurate area of the certificate layout analysis method based on deep learning and traditional algorithms in the above embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are for illustrative purposes only, and those skilled in the art will readily appreciate other obvious variations. The basic principles of the present invention defined in the following description may be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the present invention.

[0028] Those skilled in the art should understand that, in the disclosure of the present invention, the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like to indicate orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the above terms should not be understood as limiting the present invention.

[0029] It is understood that the term "a" should be understood as "at least one" or "one or more." That is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "a" should not be understood as limiting the quantity. In addition, in the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0030] In the description of the present invention, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through a medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0031] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0032] This paper is based on the target detection deep learning algorithm and traditional image processing, and outputs the recognition results of passport documents in a structured manner, such as Figure 1 FIG. 1 is a flow chart of an algorithm for analyzing a license layout based on deep learning and traditional algorithms according to an embodiment of the present invention, which mainly includes the following steps:

[0033] Step 1: Pre-processing the collected biometric information page of the certificate, performing data enhancement on the machine-readable code area, and performing character recognition on the machine-readable code area;

[0034] Step 2: Determine the country and type of the certificate based on the character information in the machine-readable code area;

[0035] Step 3: Obtain the four coordinates of the edge corner points of the machine-readable code area, use the standard certificate size, and align the machine-readable code area with the certificate page to be collected. In detail, taking the passport machine-readable code as an example, the passport machine-readable code is divided into two rows of characters. Take the four vertex coordinates, starting from the upper left corner, in order, the upper left point, the upper right point, the lower right point, and the lower left point. Then process the passport into a standard size, so that the coordinate information of the preset coordinate points of the standard size can be used to extract the recognition area. Figure 2 As shown, the processing steps are as follows: 1) obtain the coordinates of the machine-readable code area of the image to be processed in the entire image; 2) use the four vertex coordinates of the machine-readable code to perform affine transformation and straighten the image; 3) resize the original image according to the width and height of the machine-readable code to be processed and the aspect ratio of the standard machine-readable code; 4) before step 3), generate the coordinates of the standard machine-readable code area in the entire image; 5) crop the redundant edges of the image and fill the missing edges with pixels.

[0036] Step 4: Using the coordinate positions of the visual reading zone of the preset coordinate points of a standard size, extract the visual reading zone, and use an object detection algorithm to identify the characters in the visual reading zone of the certificate, such as uppercase letters, numbers, and slashes;

[0037] Step 5: Use the bottom-up filling method to adjust the bounding box of each recognition area among the preset coordinate points of the standard size. By fine-tuning the bounding box of the character area to be recognized in the visual reading area, the accurate bounding box of the recognition area is obtained. In detail, this step can also handle the problem of inaccurate area extraction caused by deformation of the document due to squeezing during the shooting process. The specific steps are as follows: The order of adjusting the coordinate frame of the passport visual reading area is to start from the distance closest to the machine-readable code area, and then process upwards in sequence. Taking the Chinese ordinary passport as an example, Figure 3 As shown, the adjustment order is: issuing authority, validity period, issuing place, issuing date, birth place, birth date, nationality, gender, name, passport number, country code, and type. The adjustment order relationship diagram is as follows:

[0038] That is, based on the coordinates of the letters in the "Valid Until" column, calculate the coordinate difference with the bounding box of the "Valid Until" column in the standard document, and adjust the bounding boxes of all other content to be identified. Then, in the next adjustment, the bounding boxes of "Valid Until", "Authorized Authority", and "Issuing Place" will not be adjusted. Then, based on the coordinates of the letters in the "Issuing Date" column, continue to adjust the coordinates of the remaining bounding boxes, and so on, until all areas to be identified are adjusted;

[0039] Step 6: Use traditional image processing algorithms, such as Gaussian filtering, image graying, and median filtering, to extract the accurate area where character recognition is required, and feed it into the convolutional recurrent neural network structure for recognition to obtain the final character recognition result. For details, please refer to the attached Figure 4 The detailed steps of the algorithm of step six are as follows: first, the area to be recognized is cropped, and then processed by Gaussian filtering, grayscale image, median filtering, Sauvola binarization, erosion and expansion, and projection algorithm to generate the final character recognition result. In step six, after the area to be recognized is cropped, the following Gaussian filtering formula is used for calculation:

[0040]

[0041] Where μ is the mean of x and δ is the standard deviation of x. Due to dust and contamination of the camera, the captured ID image can be filtered using Gaussian filtering to remove many pixels that are significantly different from the surrounding pixels. After Gaussian filtering, the image is grayscaled and calculated according to the following grayscale formula:

[0042] gray=0.2989*R+0.5870*G+0.1140*B

[0043] Among them, R is the red channel pixel, G is the green channel pixel, and B is the blue image pixel;

[0044] After the image is grayed out, it is calculated according to the following Sauvola binarization formula:

[0045] Calculate the grayscale mean m(x,y) and standard deviation S(x,y) in the area r*r.

[0046]

[0047]

[0048] The threshold T(x,y) of the sauvola binarization algorithm at the pixel point (x,y) is

[0049]

[0050] Where R is the dynamic range of the standard deviation; k∈(0, 1), sauvola binarization is used to achieve local mean brightness to better obtain the text area.

[0051] After the sauvola binarization process, the median filter is calculated according to the following formula:

[0052] g(x,y)=med{f(xi,yi)},(i,j)∈S

[0053] Where, let g(x,y), f(x,y) be the pixel grayscale value, and S be the template window;

[0054] Specifically, a (2n+1)(2n+1) window (usually 3*3 or 5*5) is selected (n is a value greater than or equal to 1), and the window is used to slide along each pixel in the row and column directions of the image data from left to right and from top to bottom. Each time it slides, the grayscale of the pixels in the window is sorted, and the grayscale value of the pixel at the center of the window is replaced by the median value. This median filter effectively removes salt and pepper noise from the image.

[0055] Through median filtering, corrosion expansion is performed and calculated according to the following corrosion operator formula:

[0056] [εB(X)](x)=min{XB}

[0057] Where [εB(X)](x) represents the method of eroding the subelement x from the set X by the structuring element B, min represents the minimum value, and XB represents the value of x within the structuring element B. The erosion step is used to remove small noise pixels in the binary image.

[0058] Calculated according to the following expansion operator formula:

[0059] [εB(X)](x)=max{XB}

[0060] Where [εB(X)](x) represents the dilation operation of the structural element B on the sub-element x of the set X, and max represents the maximum value. The dilation step is used to fill the small holes in the characters in the image to make the characters more complete.

[0061] After the erosion and dilation processing, the projection algorithm is used for processing. The horizontal projection is to accumulate the pixels of the binary image in the y-axis direction, and then segment each row of characters by the threshold value; the vertical projection is to accumulate the pixels of the binary image in the x-axis direction, and then segment each character by the threshold value, and at the same time remove the blank areas on the left and right ends to generate the final character recognition result;

[0062] Step 7: Match the known column area with the character recognition result to obtain the final structured output result.

[0063] like Figure 2 FIG. 1 is a flowchart of a method for analyzing a certificate layout based on deep learning and traditional algorithms according to an embodiment of the present invention, which mainly includes the following steps:

[0064] 1) Collect image data of the certificate page through imaging equipment;

[0065] 2) extracting image feature information of the machine-readable code area of the certificate page by processing the image data, and simultaneously feeding the image feature information into an encoder to obtain a prediction result including letters and numbers;

[0066] 2) Analyzing the machine-readable code information on the certificate page and aligning the machine-readable code areas on all the certificate pages;

[0067] 3) fine-tuning the content area frame of the certificate page to obtain the character recognition area of the final structured page;

[0068] 4) extracting the character recognition area, and performing traditional image processing algorithms on the extracted character recognition area, such as Gaussian filtering, image graying, and median filtering, to generate an accurate area of the character to be recognized;

[0069] 5) The accurate area of the character to be recognized is sent to the deep neural network to obtain the final character result, and combined with the preset template area, a structured recognition result is output.

Claims

1. A certificate layout analysis method based on deep learning and traditional algorithms, characterized by: The following steps are involved: Obtain the four coordinates of the edge corner points of the machine-readable code area of the certificate page, and align the machine-readable code area of the certificate page to be collected using the size of the standard certificate; Using the coordinate positions of the visual reading area of the preset coordinate points of a standard size, extracting the visual reading area, and using an object detection algorithm to identify the characters in the visual reading area of the certificate page; Using a bottom-up approach, adjust the bounding boxes of each recognition area within the preset coordinate points of standard size. The order of coordinate box adjustment for the passport visual area starts with the closest distance to the machine-readable code area, and then proceeds upwards. Extract the exact area where character recognition is required and feed it into the convolutional recurrent neural network structure for recognition to obtain the final character recognition result; The known column areas of the standard certificate are matched with the character recognition results to obtain the final structured output result.

2. The analysis method according to claim 1, characterized in that After adjusting the bounding box of each recognition area among the preset coordinate points of the standard size, it also includes: obtaining the accurate bounding box of the recognition area by moving the bounding box of the character area to be recognized in the visual reading area.

3. The analysis method according to claim 2, characterized in that Before obtaining the four coordinates of the edge corner points of the machine-readable code area of the certificate page, the collected biometric information page of the certificate page is preprocessed, the machine-readable code area is data enhanced, and character recognition is performed on the machine-readable code area.

4. The analysis method according to claim 3, characterized in that After character recognition of the machine-readable code area is performed, the country and type of certificate to which the certificate page belongs are confirmed based on the character information of the machine-readable code area.

5. The analysis method according to claim 4, characterized in that When extracting the accurate area where character recognition is required, the character area to be recognized is first subjected to Gaussian filtering, grayscale image conversion, and median filtering.

6. The analysis method according to claim 5, characterized in that After Gaussian filtering, grayscale image conversion, and median filtering are performed on the character area to be identified, Sauvola binarization, corrosion expansion, and projection algorithm processing are performed on the character area to be identified.

7. The analysis method according to claim 6, characterized in that The steps of extracting the recognition area using the coordinate information of the preset coordinate points of a standard size include: obtaining the coordinates of the machine-readable code area of the image to be processed in the entire image; using the four vertex coordinates of the machine-readable code to perform an affine transformation to straighten the image; resizing the original image according to the width and height ratio of the machine-readable code to be processed and the aspect ratio of the standard machine-readable code; at the same time, generating the coordinates of the standard machine-readable code area in the entire image; cropping the redundant edges of the image and filling the missing edges with pixels.

8. The analysis method according to claim 5, characterized in that When performing median filtering, select a (2n+1)(2n+1) window, where n≥1, and use the window to slide along each pixel from left to right and from top to bottom along the row and column directions of the image data; each time it slides, the grayscale of the pixels in the window is sorted, and the grayscale value of the pixel at the center of the window is replaced by the median value.

9. The analysis method according to claim 6, characterized in that When performing the projection algorithm, horizontal projection is performed by accumulating pixels of the binary image in the y-axis direction, and then dividing each row of characters by the threshold value; vertical projection is performed by accumulating pixels of the binary image in the x-axis direction, and then dividing each character by the threshold value, and at the same time removing the blank areas without characters on the left and right ends, thereby generating the final character recognition result.

Images