Key information extraction method based on spatial position relation of OCR document and table recognition results

Abstract

The invention provides a key information extraction method based on an OCR (Optical Character Recognition) document table recognition result spatial position relationship, which comprises the following steps: S1, taking a document table as a picture, and carrying out image preprocessing on the picture; s2, performing OCR (Optical Character Recognition) on the pre-processed document table, and reading data; s3, constructing a text object set with spatial attributes based on the read data; s4, extracting key data in the text object set according to predefined keywords; and S5, recursively executing the step S4 until a recursion ending condition is met, and completing text extraction. The method and the device can be widely applied to retrieval of key business data in table type or mixed type document pictures.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for extracting key information based on the spatial position relationship of OCR document form recognition results. Background Art

[0002] In the process of informatization, enterprises often need to process large amounts of documents and extract key information from them. Currently, the mainstream technology for document recognition is OCR (Optical Character Recognition). For example, Baidu's open-source model PaddleOCR can effectively identify the text content and spatial coordinates in documents. However, there are still some problems when processing documents such as mixed forms of various documents:

[0003] 1. OCR focuses on text recognition but cannot understand the logical relationships between text, especially in documents with complex layouts;

[0004] 2. The document image has quality issues (such as tilt, blur, etc.), which greatly affects the OCR recognition effect;

[0005] 3. Lack of key information extraction capabilities, and manual screening is still required for identified results;

[0006] 4. The number of sample documents is small and there is no unified template format, making it difficult to extract them through model training. Summary of the Invention

[0007] In order to solve the technical problems existing in the background technology, the present invention proposes a key information extraction method based on the spatial position relationship of OCR document form recognition results.

[0008] The present invention proposes a method for extracting key information based on the spatial position relationship of OCR document form recognition results, comprising the following steps:

[0009] S1. Take the document form as an image and perform image preprocessing on it;

[0010] S2. The pre-processed document form is recognized and data is read through OCR;

[0011] S3. Construct a text object set with spatial attributes based on the read data;

[0012] S4. Extract key data from the text object set based on predefined keywords;

[0013] S5. Recursively execute S4 until the recursive end condition is met and the text extraction is completed.

[0014] Preferably, the image preprocessing in S1 includes grayscale conversion, line detection, and direction correction.

[0015] Preferably, the specific steps of S1 are as follows:

[0016] S11, decoding the base64-encoded document form image to convert the document form image into data that can be processed by OpenCV;

[0017] S12. Use the cvtColor, GaussianBlur, and threshold functions provided by OpenCV to perform grayscale conversion, Gaussian blur processing, and binarization on the image.

[0018] S13. Use the Canny algorithm provided by OpenCV to perform edge detection on the binary image, and then use Hough transform to detect straight lines in it;

[0019] S14, using the starting point coordinates and the end point coordinates of the straight line outputted in S13 to calculate its tilt angle, retaining angles within ±45° for screening horizontal straight lines, and taking the median of the angles as the tilt angle of the entire image;

[0020] S15. Calculate the affine transformation matrix using the getRotationMatrix2D function according to the tilt angle, and use the calculated matrix to correct the image using the warpAffine function.

[0021] Preferably, the OCR engine in S2 uses Baidu's PaddleOCR for text recognition.

[0022] Preferably, each text object in the text object set constructed in S3 includes text content, upper left corner coordinates, lower right corner coordinates, and midpoint coordinates.

[0023] Preferably, the specific steps of S4 are as follows:

[0024] S41, predefined keyword K={k1, k2, ..., k n}, and use the predefined keyword K to match the text object set T = {t1, t2, ..., t n}, if an object t i The text content belongs to the predefined keywords, that is, t.text∈K, then the object t i As the initial input text for retrieval;

[0025] S42, with the current input text t i The midpoint coordinate C i As a benchmark, calculate the next text object t that meets the requirements in the text object set T j , and after retrieving t j Then remove t from the text object set T j;

[0026] S43, the text object t retrieved in S42 j Re-use the text as the input text of S42 for recursive iteration until the condition judgment is not satisfied and the recursion ends, and the text objects retrieved at each step are saved;

[0027] S44. Output search results

[0028] The output retrieval result is a set of all text objects found in the iterative process of S42 and S43, and this set serves as the key data result set of the search.

[0029] Preferably, in S42, according to the midpoint coordinate C i Calculate the next text object t that meets the requirements j , where t j The following conditions are met:

[0030]

[0031] Among them, x 1i x 1j C 1x C 1y They are the x-coordinate of the upper left corner, the x-coordinate of the lower right corner, and the x and y coordinates of the midpoint of the input text, and the rest are the coordinates of the search text.

[0032] Preferably, in S43, the condition that is judged to be unsatisfied refers to the search result being empty or the result not meeting the regular expression of the key data.

[0033] Preferably, the keywords predefined in S4 are headers to be extracted from the document table.

[0034] The present invention can improve the OCR recognition accuracy by performing image preprocessing operations on document forms; and the data extraction process does not rely on fixed document templates, but is extracted based on keywords, thereby improving the processing flexibility; thereby, it can improve the traditional OCR processing of recognition results in an environment with a small number of document samples and non-uniform formats, and can be widely used to retrieve key business data in table-type or mixed document images. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flowchart of a key information extraction method based on the spatial position relationship of OCR document form recognition results proposed by the present invention;

[0036] Figure 2 This is a flowchart of image preprocessing of a document form in a method for extracting key information based on the spatial position relationship of OCR document form recognition results proposed by the present invention;

[0037] Figure 3 This is a schematic diagram of the key data extraction process in the key information extraction method based on the spatial position relationship of OCR document form recognition results proposed by the present invention. DETAILED DESCRIPTION

[0038] Reference Figure 1-3 The present invention proposes a key information extraction method based on the spatial position relationship of OCR document form recognition results, comprising the following steps:

[0039] S1. Take the document form as an image and perform image preprocessing on it;

[0040] Specifically: the image preprocessing includes grayscale, line detection, and direction correction, such as Figure 2 The specific steps are as follows:

[0041] S11. Decode the base64-encoded document form image. Specifically, use Python and Flask framework to provide a RESTful interface to the outside world, receive the base64-encoded document form image taken by the device as input, and convert it into data that can be processed by OpenCV through the base64 module and NumPy module.

[0042] S12, grayscale conversion, Gaussian blur processing and binarization, specifically: use the cvtColor, GaussianBlur, and threshold functions provided by OpenCV to perform grayscale conversion, Gaussian blur processing, and binarization on the image to enhance image features and reduce noise interference.

[0043] S13, edge detection and line extraction, specifically: use the Canny algorithm provided by OpenCV to perform edge detection on the binary image, and then use Hough transform to detect the lines therein.

[0044] S14. Calculate the image tilt angle, specifically: use the starting point coordinates and the end point coordinates of the straight line output in S13 to calculate its tilt angle, retain the angle within ±45° for screening horizontal straight lines, and take the median of the angles as the tilt angle of the entire image.

[0045] S15. Correct the image by affine transformation, specifically: calculate the affine transformation matrix by using the getRotationMatrix2D function according to the tilt angle, and correct the image by using the calculated matrix by using the warpAffine function.

[0046] S2. The pre-processed document form is recognized and data is read through OCR;

[0047] Specifically: The OCR engine used is Baidu's PaddleOCR. The specific steps are as follows:

[0048] The pre-processed image is input into the PaddleOCR model for text recognition, and the recognized text is output to the text extraction algorithm model for processing. After processing, it contains the following information: text content (text), coordinates of the upper left corner of the text (x 1i ,y 1i ), the coordinate of the lower right corner of the text (x 2i ,y 2i ).

[0049] S3. Construct a text object set with spatial attributes based on the read data. The specific steps are as follows:

[0050] Use Java to build a collection: read each individual text block identified by S2 and its coordinate information to build it, and make the constructor of each object in the collection include text content (text), upper left corner coordinates (x 1i ,y 1i ), the lower right corner coordinate (x 2i ,y 2i ), and midpoint coordinates (C 1x ,C 1y ), that is, t i ={text,x 1i ,y 1i ,x 2i ,y 2i ,C 1x ,C 1y}, the midpoint coordinates

[0051] S4. Extract key data from the text object set based on predefined keywords;

[0052] Specifically, the predefined keywords are manually formulated header data to be retrieved and then enumerated. For example, the enumeration result of the quantity keyword can be {"quantity", "stock quantity", "order quantity"}; then, the text collection is searched for records whose text objects belong to the enumeration result, and this record is used as the retrieval basis. The specific steps of S4 are as follows:

[0053] S41, predefined keyword K={k1, k2, ..., k n}, and use the predefined keyword K to match the text object set T = {t1, t2, ..., t n}, if an object t i The text content belongs to the predefined keywords, that is, t.text∈K, then the object t i As the initial input text for retrieval;

[0054] S42, with the current input text t i The midpoint coordinate C i As a benchmark, calculate the next text object t that meets the requirements in the text object set T j , and after retrieving t j Then remove t from the text object set T j , to avoid retrieving itself in the next iteration and reduce the number of loops.

[0055] where t j The following conditions are met:

[0056]

[0057] Among them, x 1i x 1j C 1x C 1y They are the x-coordinate of the upper left corner, the x-coordinate of the lower right corner, and the x and y coordinates of the midpoint of the input text, and the rest are the coordinates of the search text, namely: Condition 1 is vertical detection, judging whether the search text is below the basic text; Condition 2 is horizontal detection, judging whether there is any overlapping part in the horizontal direction of the text. If there is overlap, the result is greater than 0; Condition 3 is to calculate the Euclidean distance and horizontal distance between the center points of the text, where the Euclidean distance is the distance between two nodes, and the horizontal distance is used to increase the weight of the horizontal direction, making the result more inclined to the vertical direction result.

[0058] S43, the text object t retrieved in S42 j Re-use the text as the input text of S42 for recursive iteration until the condition judgment is not satisfied and the recursion ends, and the text objects retrieved at each step are saved;

[0059] Specifically: the text object t retrieved by S42 j As new input text, according to the text object t j The midpoint coordinate C j Calculate the next text object t that meets the requirements in the text object set T k , and then use the retrieved text object t k As new input text, and then according to the text object t k The midpoint coordinate C k Calculate the next text object that meets the requirements in the text object set T...recursively iterate until the condition is not met and end the recursion.

[0060] The recursive iterative judgment that the condition is not satisfied means that the search result is empty or the result does not meet the regular expression of the key data, that is, the next object that meets the condition cannot be found.

[0061] S44. Output search results

[0062] The output retrieval result is a set of all text objects found in the iterative process of S42 and S43, and this set serves as the key data result set of the search.

[0063] S5, recursively execute S4 until the recursive end condition is met and the text extraction is completed;

[0064] Specifically: The recursive end condition is that the search result is empty.

[0065] Compared with the prior art, the present invention has the following advantages:

[0066] 1. The present invention can improve the OCR recognition accuracy by performing image preprocessing operations on document forms;

[0067] 2. The present invention does not rely on fixed document templates, but extracts information based on keywords, thus improving processing flexibility;

[0068] 3. This invention can improve the processing of traditional OCR recognition results in an environment with a small number of document samples and non-uniform formats;

[0069] 4. The present invention decouples the identification business function module and the data processing business function module through a RESTful interface.

[0070] As can be seen from the above, the present invention can be widely used to retrieve key business data in form-type or mixed-type document images.

[0071] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A key information extraction method based on the spatial position relationship of OCR document form recognition results, characterized by: The following steps are involved: S1. Take the document form as an image and perform image preprocessing on it; S2. The pre-processed document form is recognized and data is read through OCR; S3, constructing a text object set with spatial attributes based on the read data; S4. Extract key data from the text object set based on predefined keywords; S5. Recursively execute S4 until the recursive end condition is met and the text extraction is completed.

2. The key information extraction method based on the spatial position relationship of OCR document form recognition results according to claim 1 is characterized in that: The image preprocessing in S1 includes grayscale conversion, line detection, and direction correction.

3. The key information extraction method based on the spatial position relationship of OCR document form recognition results according to claim 2 is characterized in that: The specific steps of S1 are as follows: S11, decoding the base64-encoded document form image to convert the document form image into data that can be processed by OpenCV; S12. Use the cvtColor, GaussianBlur, and threshold functions provided by OpenCV to perform grayscale conversion, Gaussian blur processing, and binarization on the image. S13. Use the Canny algorithm provided by OpenCV to perform edge detection on the binary image, and then use Hough transform to detect straight lines in it; S14, using the starting point coordinates and the end point coordinates of the straight line outputted in S13 to calculate its tilt angle, retaining angles within ±45° for screening horizontal straight lines, and taking the median of the angles as the tilt angle of the entire image; S15. Calculate the affine transformation matrix using the getRotationMatrix2D function according to the tilt angle, and use the calculated matrix to correct the image using the warpAffine function.

4. The key information extraction method based on the spatial position relationship of OCR document form recognition results according to claim 1 is characterized in that: The OCR engine in S2 uses Baidu's PaddleOCR for text recognition.

5. The key information extraction method based on the spatial position relationship of OCR document form recognition results according to claim 1 is characterized in that: Each text object in the text object set constructed by S3 includes text content, upper left corner coordinates, lower right corner coordinates, and midpoint coordinates.

6. The key information extraction method based on the spatial position relationship of OCR document form recognition results according to claim 5 is characterized in that: The specific steps of S4 are as follows: S41, predefined keyword K={k1, k2, ..., k n }, and use the predefined keyword K to match the text object set T = {t1, t2, ..., t n }, if an object t i The text content belongs to the predefined keywords, that is, t.text∈K, then the object t i As the initial input text for retrieval; S42, with the current input text t i The midpoint coordinate C i As a benchmark, calculate the next text object t that meets the requirements in the text object set T j , and after retrieving t j Then remove t from the text object set T j ; S43, the text object t retrieved by S42 j Re-use the text as the input text of S42 for recursive iteration until the condition judgment is not satisfied and the recursion ends, and the text objects retrieved at each step are saved; S44. Output search results The output retrieval result is a set of all text objects found in the iterative process of S42 and S43, and this set serves as the key data result set of the search.

7. The key information extraction method based on the spatial position relationship of OCR document form recognition results according to claim 6 is characterized in that: In S42, according to the midpoint coordinate C i Calculate the next text object t that meets the requirements j , where t j The following conditions are met: Among them, x 1i x 1j C 1x C 1y They are the x-coordinate of the upper left corner, the x-coordinate of the lower right corner, and the x and y coordinates of the midpoint of the input text, and the rest are the coordinates of the search text.

8. The key information extraction method based on the spatial position relationship of OCR document form recognition results according to claim 6 is characterized in that: In S43, the condition that is judged to be unsatisfied refers to the search result being empty or the result not meeting the regular expression of the key data.

9. The key information extraction method based on the spatial position relationship of OCR document form recognition results according to any one of claims 1 to 8, characterized in that: The predefined keywords in S4 are the headers that need to be extracted from the document table.

Images