Data intelligent recognition and extraction system, method and medium based on deep learning

Abstract

The application provides a kind of data intelligent recognition extraction system, method and medium based on deep learning, including background server containing picture text recognition, table recognition and text layout intelligent recognition restoration function, built-in load balancing;Web page contains text recognition, table recognition, PDF recognition and scanning PDF text comparison function;Windows 10 client calls background server to carry out picture text recognition, screenshot recognition and PDF document recognition, all recognition functions contain intelligent text layout restoration capability.The application converts text box into undirected graph model in graph theory according to the relative position of text box in the picture, and restores the original text layout intelligently using rectangle sorting and relative position judgment algorithm;The application has low requirements for system hardware, does not require GPU required by normal OCR service, and can achieve close to GPU recognition performance on ordinary personal PC.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a data intelligent identification and extraction system, method and medium based on deep learning, and especially to an unstructured data intelligent identification and extraction method based on deep learning and geometric layout intelligent analysis. BACKGROUND

[0002] Deep learning is a new research direction in the field of machine learning, which is introduced into machine learning to make it closer to the original goal-artificial intelligence. Deep learning is to learn the internal rules and representation levels of sample data, and the information obtained in these learning processes is very helpful for the interpretation of data such as text, images and sounds. The ultimate goal of deep learning is to enable machines to have analysis and learning ability like humans, and to be able to recognize text, images and sound data. Deep learning is a complex machine learning algorithm, which has achieved much better results in speech and image recognition than previous related technologies. Deep learning has achieved many results in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech, recommendation and personalization technology, and other related fields. Deep learning enables machines to imitate human activities such as vision, hearing and thinking, solves many complex pattern recognition problems, and makes great progress in artificial intelligence related technologies.

[0003] A non-structured data extraction method based on deep learning is disclosed in the patent document with publication number CN110795543A, which is applied to the field of natural language processing of artificial intelligence. The method includes: obtaining input knowledge text; calling a head entity identification model to identify head entities from the knowledge text, and calling a relationship identification model to identify relationship entities from the knowledge text; constructing a question according to the head entities and the relationship entities; calling a reading comprehension model to extract an answer from the knowledge text according to the question, and determining the answer as a tail entity; and constructing a knowledge graph according to the triplets composed of the head entities, the relationship entities and the tail entities.

[0004] The inventor believes that the mainstream OCR system interface on the market only realizes the text positioning and recognition function, but cannot restore the text layout of the whole image. Therefore, a technical solution is needed to improve the above technical problems. SUMMARY

[0005] In view of the defects in the prior art, the present application aims to provide a data intelligent identification and extraction system, method and medium based on deep learning.

[0006] According to the data intelligent identification and extraction method based on deep learning provided by the present application, the method comprises the following steps:

[0007] Step S1: Image preprocessing;

[0008] Step S2: Detect and correct the text box positions on the preprocessed image;

[0009] Step S3: Take a screenshot of the text box area based on the detected text box position;

[0010] Step S4: Correct the text box edges of the obtained screenshot;

[0011] Step S5: Perform text recognition using a CRNN model on the screenshot after text box edge correction to obtain the recognition result;

[0012] Step S6: Perform text correction based on a Bayesian model on the recognition results to obtain the corrected text;

[0013] Step S7: Perform layout analysis on the text box to obtain the layout analysis results;

[0014] Step S8: Output the error correction text to the corresponding position of the image based on the layout analysis results.

[0015] Preferably, step S7 includes the following steps:

[0016] Step S7.1: Estimate the global text line height of the image based on the total height of all text boxes;

[0017] Step S7.2: Calculate the two-dimensional positional relationship based on the text box coordinates;

[0018] Step S7.3: Estimate the line information occupied by all text boxes based on the text box height;

[0019] Step S7.4: Estimate the column information occupied by all text boxes based on the width of the text boxes;

[0020] Step S7.5: Estimate the position of the text in the text box based on the row and column information of the text box;

[0021] Step S7.6: Sort all texts according to the order calculated in step S7.2;

[0022] Step S7.7: Based on the text sorting result, insert the appropriate number of blank lines and leading spaces between two adjacent lines of text according to the vertical spacing;

[0023] Step S7.8: Assemble the text layout.

[0024] Preferably, step S7.2 simulates the process of a person reading text. If two text boxes are on the same line, their order is determined according to their left and right positions; if they are not on the same line, their order is determined according to their top and bottom positions. The common area of ​​the projections of the text boxes on the Y-axis is used to estimate whether they are on the same line. An empirical value is that if the common part of the projections of the two text boxes on the Y-axis exceeds 70% of their height, they are on the same line of text.

[0025] Preferably, step S7.8 involves assembling all the text and corresponding spaces into a complete text in sequence.

[0026] This invention also provides a data intelligent recognition and extraction system based on deep learning, the system comprising the following modules:

[0027] Module M1: Image preprocessing;

[0028] Module M2: Performs text box position detection and correction on the preprocessed image;

[0029] Module M3: Captures a screenshot of the text box area based on the detected text box location;

[0030] Module M4: Corrects the text box edges of the obtained screenshot;

[0031] Module M5: Performs text recognition using a CRNN model on the screenshot after text box edge correction to obtain the recognition result;

[0032] Module M6: Performs text correction based on a Bayesian model on the recognition results to obtain the corrected text;

[0033] Module M7: Performs layout analysis on text boxes and obtains the layout analysis results;

[0034] Module M8: Outputs the error correction text to the corresponding position of the image based on the layout analysis results.

[0035] Preferably, module M7 includes the following modules:

[0036] Module M7.1: Estimates the global text line height of an image based on the total height of all text boxes;

[0037] Module M7.2: Calculates two-dimensional positional relationships based on text box coordinates;

[0038] Module M7.3: Estimates the line information occupied by all text boxes based on the height of the text boxes;

[0039] Module M7.4: Estimates the column information occupied by all text boxes based on their width;

[0040] Module M7.5: Estimates the position of text in a text box based on the row and column information of the text box;

[0041] Module M7.6: Sorts all text according to the order calculated by module M7.2;

[0042] Module M7.7: Based on the text sorting result, insert the appropriate number of blank lines and leading spaces between two adjacent lines of text according to the vertical spacing;

[0043] Module M7.8: Assemble text layout.

[0044] Preferably, module M7.2 simulates the process of a person reading text. If two text boxes are on the same line, their order is determined by their left and right positions; if they are not on the same line, their order is determined by their top and bottom positions. The order of text boxes is estimated based on the common area of ​​their projections on the Y-axis. An empirical value is that if the common part of their projections on the Y-axis exceeds 70% of their height, they are on the same line of text.

[0045] Preferably, step S7.8 involves assembling all the text and corresponding spaces into a complete text in sequence.

[0046] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described above.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. This invention fully utilizes the text box position information based on convolutional neural networks and their outputs, and surpasses mainstream optical character recognition (OCR) systems in terms of layout restoration after image text recognition and overall computing performance. In particular, the accurate text layout restoration algorithm greatly facilitates users' subsequent information processing. It also includes customized error correction capabilities based on domain dictionaries for specific business scenarios.

[0049] 2. Based on the relative position of the text box in the image, this invention converts the text box into an undirected graph model in graph theory, and uses a rectangle sorting and relative position judgment algorithm to intelligently restore its original text layout.

[0050] 3. This invention has low system hardware requirements. It does not require the GPU typically needed for OCR services and can achieve near-GPU recognition performance on a regular personal PC. Attached Figure Description

[0051] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0052] Figure 1 This is a sample image showing the output results of mainstream text recognition products on the market.

[0053] Figure 2 This is a diagram illustrating the text recognition and layout restoration effects of the present invention.

[0054] Figure 3 This is a flowchart illustrating the principle of the present invention.

[0055] Figure 4 This is a flowchart illustrating step S7 of the present invention. Detailed Implementation

[0056] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0057] This invention provides a deep learning-based intelligent recognition system for unstructured data, comprising:

[0058] Backend server: Includes image text recognition, table recognition, intelligent text layout recognition and restoration, etc. Built-in load balancing ensures high availability and can run on Windows 10 and various mainstream Linux distributions. Deployment is quick and easy, requiring no runtime dependencies; simply copy with one click.

[0059] Web version: Includes functions such as text recognition, table recognition, PDF recognition, and text comparison of scanned PDFs. All recognition functions include intelligent text layout restoration capabilities.

[0060] Windows 10 client: It realizes functions such as image text recognition, screenshot recognition, and PDF document recognition by calling the background server. All recognition functions include intelligent text layout restoration capabilities.

[0061] This invention also provides a data intelligent recognition and extraction method based on deep learning, comprising the following steps:

[0062] Step S1: Image preprocessing;

[0063] Step S2: Text box position detection and correction;

[0064] Step S3: Take a screenshot of the text box area;

[0065] Step S4: Text box edge correction;

[0066] Step S5: The CRNN model performs text recognition;

[0067] Step S6: Text correction based on Bayesian model;

[0068] Step S7: Text box layout analysis; estimate the global text line height of the image based on the height of all text boxes; calculate the two-dimensional positional relationship based on the text box coordinates; in principle, it simulates the process of a person reading text from top to bottom and from left to right. If two text boxes are on the same line, their order is determined according to their left-right position. If they are not on the same line, their order is determined according to their top-bottom position. Whether they are on the same line is estimated based on the common area of ​​their projections on the Y-axis. An empirical value is that if the common part of their projections on the Y-axis exceeds 70% of their heights, they are considered to be on the same line of text; estimate the line information occupied by all text boxes based on the text box height; estimate the column information occupied by all text boxes based on the text box width; estimate the position of the text in the text box based on the line and column information of the text box; sort all the text according to the order calculated in step S7.2; based on the text sorting result, insert the corresponding number of blank lines and leading spaces between two adjacent different lines of text according to the vertical spacing; assemble the text layout, assembling all the text and corresponding spaces into complete text in order.

[0069] Step S8: Output the original text.

[0070] This invention also provides a data intelligent recognition and extraction system based on deep learning. The system includes the following modules: Module M1: image preprocessing; Module M2: text box position detection and correction of the preprocessed image; Module M3: text box region screenshot based on the detected text box position; Module M4: text box edge correction of the obtained screenshot; Module M5: text recognition using a CRNN model on the screenshot with corrected text box edges to obtain the recognition result; Module M6: text correction based on a Bayesian model on the recognition result to obtain the corrected text.

[0071] Module M7: Performs layout analysis on text boxes and obtains the layout analysis results; Module M7.1: Estimates the global text line height of the image based on the height of all text boxes.

[0072] Module M7.2: Calculates two-dimensional positional relationships based on text box coordinates; simulates the process of human reading text. If two text boxes are on the same line, their order is determined by their left-right positions; if they are not on the same line, their order is determined by their top-bottom positions; estimates whether they are on the same line based on the common area of ​​their projections on the Y-axis. An empirical value is that if the common part of their projections on the Y-axis exceeds 70% of their heights, they are on the same line of text.

[0073] Module M7.3: Estimates the row information occupied by all text boxes based on the height of the text boxes; Module M7.4: Estimates the column information occupied by all text boxes based on the width of the text boxes; Module M7.5: Estimates the position of the text in the text boxes based on the row and column information of the text boxes; Module M7.6: Sorts all text according to the order calculated by Module M7.2; Module M7.7: Based on the text sorting result, inserts an appropriate number of blank lines and leading spaces between two adjacent lines of text according to the vertical spacing.

[0074] Module M7.8: Assembles text layouts by assembling all text and corresponding spaces into complete text in order.

[0075] Module M8: Outputs the error correction text to the corresponding position of the image based on the layout analysis results.

[0076] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described above.

[0077] This invention fully utilizes the text box position information based on convolutional neural networks and their outputs, surpassing mainstream optical character recognition (OCR) systems in layout restoration and overall computational performance after image text recognition. In particular, its accurate text layout restoration algorithm greatly facilitates subsequent information processing for users. It also includes customized error correction capabilities based on domain dictionaries for specific business scenarios. Based on the relative position of the text boxes in the image, this invention transforms the text boxes into an undirected graph model from graph theory, and intelligently restores the original text layout using a rectangle sorting and relative position judgment algorithm. This invention has low system hardware requirements, eliminating the need for the GPU typically required for OCR services, and can achieve near-GPU recognition performance directly on ordinary personal PCs.

[0078] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0079] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A data intelligent recognition and extraction method based on deep learning, characterized in that, The method includes the following steps: Step S1: Image preprocessing; Step S2: Detect and correct the text box positions on the preprocessed image; Step S3: Take a screenshot of the text box area based on the detected text box position; Step S4: Correct the text box edges of the obtained screenshot; Step S5: Perform text recognition using a CRNN model on the screenshot after text box edge correction to obtain the recognition result; Step S6: Perform text correction based on a Bayesian model on the recognition results to obtain the corrected text; Step S7: Perform layout analysis on the text box to obtain the layout analysis results; Step S8: Output the correction text to the corresponding position in the image based on the layout analysis results; Step S7 includes the following steps: Step S7.1: Estimate the global text line height of the image based on the total height of all text boxes; Step S7.2: Calculate the two-dimensional positional relationship based on the text box coordinates; Step S7.3: Estimate the line information occupied by all text boxes based on the text box height; Step S7.4: Estimate the column information occupied by all text boxes based on the width of the text boxes; Step S7.5: Estimate the position of the text in the text box based on the row and column information of the text box; Step S7.6: Sort all texts according to the order calculated in step S7.2; Step S7.7: Based on the text sorting result, insert the appropriate number of blank lines and leading spaces between two adjacent lines of text according to the vertical spacing; Step S7.8: Assemble the text layout; Step S7.2 simulates the process of a person reading text. If two text boxes are on the same line, their order is determined by their left and right positions; if they are not on the same line, their order is determined by their top and bottom positions. The common area of ​​the projections of the text boxes on the Y-axis is used to estimate whether they are on the same line. An empirical value is that if the common part of the projections of the two text boxes on the Y-axis exceeds 70% of their height, they are on the same line of text. S7.8 assembles all the text and corresponding spaces into a complete text in order.

2. A data intelligent recognition and extraction system based on deep learning, characterized in that, The system includes the following modules: Module M1: Image preprocessing; Module M2: Performs text box position detection and correction on the preprocessed image; Module M3: Captures a screenshot of the text box area based on the detected text box location; Module M4: Corrects the text box edges of the obtained screenshot; Module M5: Performs text recognition using a CRNN model on the screenshot after text box edge correction to obtain the recognition result; Module M6: Performs text correction based on a Bayesian model on the recognition results to obtain the corrected text; Module M7: Performs layout analysis on text boxes and obtains the layout analysis results; Module M8: Outputs the error correction text to the corresponding position in the image based on the layout analysis results; Module M7 includes the following modules: Module M7.1: Estimates the global text line height of an image based on the total height of all text boxes; Module M7.2: Calculates two-dimensional positional relationships based on text box coordinates; Module M7.3: Estimates the line information occupied by all text boxes based on the height of the text boxes; Module M7.4: Estimates the column information occupied by all text boxes based on their width; Module M7.5: Estimates the position of text in a text box based on the row and column information of the text box; Module M7.6: Sorts all text according to the order calculated by module M7.2; Module M7.7: Based on the text sorting result, insert the appropriate number of blank lines and leading spaces between two adjacent lines of text according to the vertical spacing; Module M7.8: Assemble text layout; The module M7.2 simulates the process of a person reading text. If two text boxes are on the same line, their order is determined by their left and right positions; if they are not on the same line, their order is determined by their top and bottom positions. The common area of ​​the projections of the text boxes on the Y-axis is used to estimate whether they are on the same line. An empirical value is that if the common part of the projections of the two text boxes on the Y-axis exceeds 70% of their height, they are on the same line of text. S7.8 assembles all the text and corresponding spaces into a complete text in order.

3. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method of claim 1.

Images