Method, device, storage medium and equipment for extracting structured information from images

Abstract

The present application discloses a method, device, storage medium and equipment for extracting structured information from an image, and belongs to the field of image processing. The method includes: obtaining a digital image to be extracted; performing OCR processing on the digital image to obtain an OCR recognition result; if structured information needs to be extracted, determining an extraction area in the digital image according to the type corresponding to the structured information, and the types of structured information include smart tag type, table type and fixed position data type; extracting data from the extraction area according to preset extraction rules and OCR recognition results, and combining the data and the corresponding document attribute names in the extraction rules into structured information, and the extraction rules are obtained after analyzing the text layout in the digital image. The present application obtains extraction rules by analyzing the text layout in the digital image, and then determines the extraction area based on the type of structured information, and extracts structured information from the extraction area using the extraction rules.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, device, storage medium and equipment for extracting structured information from an image. Background Art

[0002] The digital images we commonly see contain a lot of data that is useful to people. For example, the scanned image of a business license contains information such as the company name and legal representative; the scanned image of a student information card contains information such as the student's name, gender, and major; the scanned image of an official document contains information such as the document number and title. These data are usually stored in a (Key, Value) structure, and these data are referred to as structured information below. Among them, the Value part also includes two-dimensional table data (Grid), such as multi-row and multi-column resume information in a personnel information card, and archive file directory tables. If the structured information can be extracted from the image, it is possible to automatically classify and organize the scanned files, help build the database of the information system, efficiently establish the archive directory, and greatly optimize the office process and efficiency of the information system.

[0003] In related technologies, we can digitize (scan / photograph) archived paper documents to obtain digital images; then, use OCR (Optical Character Recognition) technology to identify text and the position coordinates of the text in the image from the digital image; finally, staff manually create structured information based on the OCR recognition results.

[0004] The OCR technology can only identify the text and text location in the digital image, but cannot analyze the structured information. Summary of the invention

[0005] The present application provides a method, device, storage medium and equipment for extracting structured information from an image, which is used to solve the problem in the related art that structured information cannot be identified from a digital image. The technical solution is as follows:

[0006] According to a first aspect of the present application, a method for extracting structured information from an image is provided, the method comprising:

[0007] Acquire a digital image to be extracted;

[0008] Performing optical character recognition (OCR) processing on the digital image to obtain an OCR recognition result, wherein the OCR recognition result includes the text in the digital image and the position of the text in the digital image;

[0009] If it is necessary to extract structured information, an extraction area is determined in the digital image according to the type corresponding to the structured information, wherein the type of the structured information includes a smart tag type, a table type, and a fixed position data type;

[0010] Data is extracted from the extraction area according to preset extraction rules and the OCR recognition result, and the data and the document attribute names corresponding to the extraction rules are combined into structured information. The extraction rules are obtained after analyzing the text layout in the digital image.

[0011] In a possible implementation manner, determining the extraction area in the digital image according to the type corresponding to the structured information includes:

[0012] When the type is a smart label type or a fixed position data type, detecting whether the digital image meets the trigger condition of the trigger, and if the digital image meets the trigger condition, acquiring the rectangular area set in the trigger, and determining the rectangular area as the extraction area;

[0013] When the type is a table type, table lines are detected in the digital image, and cells in the table are determined according to the table lines to obtain an extraction area.

[0014] In a possible implementation, extracting data from the extraction area according to a preset extraction rule and the OCR recognition result includes:

[0015] When the type is a smart label type and the extraction area belongs to a non-regular matching type, the label text is identified according to the OCR recognition result, and in the extraction area, the data indicated by the label text is extracted according to the extraction rule.

[0016] In a possible implementation, extracting data from the extraction area according to a preset extraction rule and the OCR recognition result includes:

[0017] When the type is a smart tag type and the extraction area belongs to a regular matching type, a regular expression is obtained according to the extraction rule, and in the extraction area, data matching the regular expression is extracted according to the OCR recognition result.

[0018] In a possible implementation, extracting data from the extraction area according to a preset extraction rule and the OCR recognition result includes:

[0019] When the type is a table type, detecting whether there is a table header in the extraction area according to the OCR recognition result;

[0020] If there is a table header in the extraction area, extract the data in each cell corresponding to the table header from the table header row downward;

[0021] If there is no header in the extraction area, a virtual header is generated, and data in each cell corresponding to the virtual header is extracted.

[0022] In a possible implementation, extracting data from the extraction area according to a preset extraction rule and the OCR recognition result includes:

[0023] When the type is a fixed position data type, detecting whether the digital image meets the applicable conditions according to the extraction rule;

[0024] If the digital image meets the applicable condition, the data in the extraction area is extracted according to the OCR recognition result.

[0025] In a possible implementation, the method further includes:

[0026] The digital image is preprocessed, and the preprocessing includes at least one of automatic deflection correction, background removal and deformation correction, and automatic segmentation of specific certificates.

[0027] According to a second aspect of the present application, a device for extracting structured information from an image is provided, the device comprising:

[0028] An acquisition module, used for acquiring a digital image to be extracted;

[0029] A recognition module, configured to perform optical character recognition (OCR) processing on the digital image to obtain an OCR recognition result, wherein the OCR recognition result includes the text in the digital image and the position of the text in the digital image;

[0030] A setting module, for determining an extraction area in the digital image according to a type corresponding to the structured information if structured information needs to be extracted, wherein the types of the structured information include a smart tag type, a table type, and a fixed position data type;

[0031] The extraction module is used to extract data from the extraction area according to the preset extraction rules and the OCR recognition results, and to form structured information with the data and the document attribute names corresponding to the extraction rules, wherein the extraction rules are obtained after analyzing the text layout in the digital image.

[0032] According to a third aspect of the present application, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the method for extracting structured information from an image as described above.

[0033] According to a fourth aspect of the present application, a computer device is provided, characterized in that the computer device comprises: the apparatus for extracting structured information from an image as described above.

[0034] The beneficial effects of the technical solution provided by this application include at least:

[0035] By analyzing the text layout in the digital image to obtain the extraction rules, and then determining the extraction area based on the type of structured information, the structured information is extracted from the extraction area using the extraction rules. There is no need to use a large number of samples for annotation and training of the neural network model, which saves the process of collecting samples, manual annotation and training, and can quickly and flexibly increase the extraction capabilities of new types of files / digital images. In addition, the training and reasoning of neural networks require the computing resources of GPUs, while the algorithm for analyzing text layout does not require GPU support and has much lower hardware requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 is a flowchart of a method for extracting structured information from an image provided by an embodiment of the present application;

[0038] Figure 2 is a flowchart of a method for extracting structured information from an image provided by an embodiment of the present application;

[0039] Figure 3 It is a structural block diagram of a device for extracting structured information from an image provided by an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0041] This application provides a way for users to easily customize the structured information to be extracted according to business needs. Specifically, the OCR recognition results can be further analyzed, and the structure of the page can be analyzed according to user-defined settings or national standards to obtain the structured information in the digital image. Based on the above algorithm, we have developed desktop and server versions of the software product, which can output the extraction results in a variety of configurable ways and can be used as a network service to support business systems.

[0042] The methods and products in this application can be applied but not limited to: building a government service window system to extract information from certificates and business forms; applying to office automation systems to extract information from official documents (title, document number, issuing agency logo, confidentiality level, copy number, etc.); applying to archive information management systems to extract information required for archive directories and quickly generate archive directories.

[0043] The following is an explanation of the process of the method for extracting structured information from an image provided by the present application.

[0044] like Figure 1 As shown, it shows a method flow chart of a method for extracting structured information from an image provided by an embodiment of the present application. The method for extracting structured information from an image can be applied to a computer device installed with the above-mentioned desktop version or server version software product. The method for extracting structured information from an image may include:

[0045] Step 101: Acquire a digital image to be extracted.

[0046] Digital images can be obtained by digitizing (scanning / photographing) paper documents, and the digital images contain structured information. According to different file types and different data to be extracted, we divide structured information into three categories, namely smart tags, tables, and fixed position data. Among them, the relative position of smart tags in digital images is not fixed, but there are feature tags indicating or conforming to specific patterns, such as directories. Tables are data with attributes of two-dimensional table data types. Fixed position data is data with relatively fixed data positions and is not convenient to define in a smart tag manner.

[0047] We can set up an extraction interface for the desktop version or server version of the software product and provide the extraction interface to the user. When the user needs to extract structured information from a digital image, he can upload at least one digital image through the extraction interface, and the computer device can obtain at least one digital image.

[0048] Step 102, performing OCR processing on the digital image to obtain an OCR recognition result, wherein the OCR recognition result includes the text in the digital image and the position of the text in the digital image.

[0049] The OCR technology in the prior art is already very mature, and this embodiment does not elaborate on the process of processing digital images using the OCR technology.

[0050] Step 103: if it is necessary to extract structured information, then determine the extraction area in the digital image according to the type corresponding to the structured information, and the types of structured information include smart tag type, table type and fixed position data type.

[0051] The extracted region is the region in the digital image that contains structural information.

[0052] Structured information includes smart tags, tables, and fixed-position data. We need to determine the corresponding extraction area based on the specific type of structured information. The specific determination method is described in the following.

[0053] Step 104, extracting data from the extraction area according to preset extraction rules and OCR recognition results, and combining the data and the document attribute names corresponding to the extraction rules into structured information, wherein the extraction rules are obtained by analyzing the text layout in the digital image.

[0054] The extraction rules are obtained by analyzing the text layout in the digital image. For different types of digital images, we can set extraction rules to specify the data to be extracted and how to extract the data. Users can define multiple different extraction rules.

[0055] Specifically, the extraction rules may include several document attribute definitions. Document attribute definitions are the definitions of columns in a database table. For each document attribute, the user needs to specify the document attribute name and data type (text type, two-dimensional table type). The user can add, delete, and modify the definition of document attributes. Each document attribute has its own value, and the data extracted from the digital image is ultimately assigned to a certain document attribute. In other words, document attributes and data constitute structured information.

[0056] When we need to extract structured information from a file with a new type of text layout, we only need to analyze the text layout of the file to obtain the extraction rules, and then we can extract the structured information based on the extraction rules. There is no need to use a large number of samples for annotation and training of neural network models, which saves the process of collecting samples, manual annotation and training, and can quickly and flexibly increase the extraction capabilities of new types of files / digital images.

[0057] In summary, the method for extracting structured information from an image provided by the embodiment of the present application obtains extraction rules by analyzing the text layout in a digital image, and then determines the extraction area based on the type of structured information, and extracts structured information from the extraction area using the extraction rules. There is no need to use a large number of samples for annotation and training of a neural network model, which eliminates the process of collecting samples, manual annotation and training, and can quickly and flexibly increase the extraction capabilities of new types of files / digital images. In addition, the training and reasoning of neural networks require the computing resources of a GPU, while the algorithm for analyzing text layout does not require GPU support, and the hardware requirements are much lower.

[0058] like Figure 2As shown, it shows a flow chart of a method for extracting structured information from an image provided by an embodiment of the present application. The method for extracting structured information from an image can be applied to a computer device installed with the above-mentioned desktop version or server version software product. The method for extracting structured information from an image can include:

[0059] Step 201: Acquire a digital image to be extracted.

[0060] Digital images can be obtained by digitizing (scanning / photographing) paper documents, and the digital images contain structured information. According to different file types and different data to be extracted, we divide structured information into three categories, namely smart tags, tables, and fixed position data. Among them, the relative position of smart tags in digital images is not fixed, but there are feature tags indicating or conforming to specific patterns, such as directories. Tables are data with attributes of two-dimensional table data types. Fixed position data is data with relatively fixed data positions and is not convenient to define in a smart tag manner.

[0061] We can set an extraction interface for the desktop version or server version of the software product and provide the extraction interface to the user. When the user needs to extract structured information from a digital image, he can upload at least one digital image through the extraction interface, and the computer device can obtain at least one digital image. For each digital image, steps 202-205 can be executed to extract structured information from it.

[0062] Step 202, preprocessing the digital image, the preprocessing including at least one of automatic deflection correction, background removal and deformation correction, and automatic segmentation of specific certificates.

[0063] Before extracting structured information, the user can be provided with a preprocessing type, and the user configures the required preprocessing steps based on the preprocessing type. Each step is executed in the configured order, and the next preprocessing step uses the output of the previous step as input. Common preprocessing includes:

[0064] (1) Automatic deflection correction: It can detect the deflection angle of the digital image according to the direction of the text lines in the digital image and automatically correct the deflection of the digital image. This image processing algorithm is based on a more mature deflection correction algorithm.

[0065] (2) Photo background removal and distortion correction: It can remove the background in the digital image and correct its perspective distortion. For example, some certificates are placed on a table when taking photos. The certificate itself is not placed correctly in the digital image and has perspective distortion. In order to extract the information of the certificate, it is necessary to separate the certificate from the background of the table and correct its perspective distortion.

[0066] (3) Automatic segmentation of specific certificates: Some commonly used certificates (such as ID cards and driver's licenses) have 1-2 pages. They are sometimes copied or photographed on paper with a brightness close to that of the background. It is difficult to obtain good results by preprocessing the background removal and deformation correction of non-specific certificates. Inferring the boundaries of the certificate image based on the characteristic text on the certificate has become a more reliable image segmentation method. We have developed this more reliable image segmentation algorithm for some certificates with known standard formats and characteristic text. The algorithm can segment certificates such as ID cards (front and back) and driver's licenses (main and secondary pages) from digital images. If there are two pages, the two pages are spliced ​​into a digital image.

[0067] Step 203, performing OCR processing on the digital image to obtain an OCR recognition result, which includes the text in the digital image and the position of the text in the digital image.

[0068] The OCR technology in the prior art is already very mature, and this embodiment does not elaborate on the process of processing digital images using the OCR technology.

[0069] Step 204: if it is necessary to extract structured information, then determine the extraction area in the digital image according to the type corresponding to the structured information, and the types of structured information include smart tag type, table type and fixed position data type.

[0070] The extracted region is the region in the digital image that contains structural information.

[0071] Structured information includes smart tags, tables, and fixed-position data. We need to determine the corresponding extraction area based on the specific type of structured information. The determination method is as follows:

[0072] (1) When the type is a smart label type or a fixed position data type, detect whether the digital image meets the trigger condition of the trigger. If the digital image meets the trigger condition, obtain the rectangular area set in the trigger and determine the rectangular area as the extraction area.

[0073] The extraction rules can include several triggers. The user selects a sample image, delimits a rectangular area with the mouse, and specifies the trigger text. The computer device records the relative coordinates of the rectangular area (relative to the length and width of the digital image, with a value range of 0...1) and the trigger text to form a trigger. The user can add, delete, and modify the definition of the trigger.

[0074] When the computer device finds that the corresponding rectangular area in the digital image contains the trigger text specified by the trigger, the trigger is in a trigger-effective state, and the area is determined as the extraction area.

[0075] (2) When the type is a table, the horizontal and vertical lines of the table are detected in the digital image, and the cells in the table are determined according to the horizontal and vertical lines to obtain the extraction area.

[0076] The computer device can detect table lines (horizontal lines and vertical lines) in the digital image, and if the table lines are detected, the extraction area is determined.

[0077] Step 205, extracting data from the extraction area according to preset extraction rules and OCR recognition results, and combining the data and the document attribute names corresponding to the extraction rules into structured information, wherein the extraction rules are obtained by analyzing the text layout in the digital image.

[0078] The extraction rules are obtained by analyzing the text layout in the digital image. For different types of digital images, we can set extraction rules to specify the data to be extracted and how to extract the data. Users can define multiple different extraction rules.

[0079] Specifically, the extraction rules may include several document attribute definitions. Document attribute definitions are the definitions of columns in a database table. For each document attribute, the user needs to specify the document attribute name and data type (text type, two-dimensional table type). The user can add, delete, and modify the definition of document attributes. Each document attribute has its own value, and the data extracted from the digital image is ultimately assigned to a certain document attribute. In other words, document attributes and data constitute structured information.

[0080] The following describes the extraction process based on the type of structured information.

[0081] (1) Structured information is a smart tag

[0082] When the type is a smart label type and the extraction area belongs to a non-regular matching type, the label text is recognized according to the OCR recognition result, and in the extraction area, the data indicated by the label text is extracted according to the extraction rules.

[0083] In this case, the extraction rules also include information such as the specified label text, data content location, number of skips, corresponding document attribute name, whether the label is multi-line, whether the data content is multi-line, applicable pages, etc. Users can add, delete, modify smart label extraction areas, or modify their order in the collection.

[0084] in:

[0085] Label text: a symbol text indicating data or a regular expression containing target data.

[0086] Data content position: can be one of "left-right", "top-bottom", or "regular match", which represents the positional relationship between the data to be extracted and the label text.

[0087] Skip count: It is a non-negative integer N, which means ignoring the first N tags that meet the tag matching conditions.

[0088] Document attribute name: If data is extracted, this document attribute will be assigned. Multiple extraction areas can correspond to the same document attribute. During an extraction process, if it is found that the document attribute has been assigned a value in this process, the value of the document attribute remains unchanged.

[0089] Is the label multi-line: Indicates whether the label text must be on the same line. For example, the text in a table cell may wrap due to the width of the column. If "yes", the search for matching label text will consider multiple lines.

[0090] Whether the data content is multi-line: whether the extracted data can wrap.

[0091] Content filter conditions: Select one from the following list. Filter conditions can automatically filter invalid text or correct misidentified text when extracting data, and get better extraction results when the format is not standardized. Content filter conditions can include but are not limited to the following: no filter, gender (male, female), nationality (Han,...), whether Chinese, x year x month x day: October 8, 2016 or October 8, 2016, Western date: 1978-12-1 or 1978-04", numerical value (including decimal point, symbol), all numbers, letters and numbers, Chinese uppercase amount, lowercase amount.

[0092] Applicable pages: When an image file contains multiple pages, it is used to specify which pages the extraction search scope applies to. Users can specify one of "First page", "Last page", "Any page", or "Pages that meet the trigger conditions". If the user selects "Pages that meet the trigger conditions", you need to continue to specify the name of the trigger.

[0093] For each non-regular matching extraction area, search and locate the position of its label text in the page; if the page is not an applicable page for the smart label, ignore the smart label; if the page is an applicable page for the smart label, perform extraction.

[0094] When the type is smart tag type and the extraction area belongs to regular matching type, a regular expression is obtained according to the extraction rule, and in the extraction area, data matching the regular expression is extracted according to the OCR recognition result.

[0095] In the OCR recognition results, for each regular matching extraction area, search and locate the position of the text that meets the regular expression matching. The matching algorithm will consider factors such as whether the content is multi-line, the number of skips, etc. Take the first matching group from the matching results as the extracted data.

[0096] (2) Structured information is a table

[0097] When the type is table type, detect whether there is a header in the extraction area based on the OCR recognition result; if there is a header in the extraction area, extract the data in each cell corresponding to the header from the header row downward; if there is no header in the extraction area, generate a virtual header and extract the data in each cell corresponding to the virtual header.

[0098] In this case, the extraction rule also includes the table header, document property name, and applicable page.

[0099] Header: If the header is at the top row of the table, it can be omitted and the computer device will automatically discover it.

[0100] Document attributes: If data is extracted, this document attribute will be assigned, but the type of this attribute must be a two-dimensional table type.

[0101] Applicable pages: When an image file contains multiple pages, it is used to specify which pages the extraction search scope applies to. Users can specify one of "First page", "Last page", "Any page", or "Pages that meet the trigger conditions". If the user selects "Pages that meet the trigger conditions", you need to continue to specify the name of the trigger.

[0102] Specifically, the identified horizontal and vertical lines can be used to analyze the cells of the table on each page to determine the extraction area. Then, for each extraction area, check whether the page is applicable according to the order of the pages. If the page is applicable and the table has a header, then according to the header found previously, from the header row downward, obtain the data in each cell that has a corresponding relationship with the header; if the table does not have a header (pure data table), a virtual header is generated, and from the virtual header row downward, obtain the data in each cell that has a corresponding relationship with the virtual header.

[0103] (3) Structured information is fixed-position data

[0104] When the type is a fixed position data type, whether the digital image meets the applicable conditions is detected according to the extraction rules; if the digital image meets the applicable conditions, the data located in the extraction area is extracted according to the OCR recognition result.

[0105] In this case, the extraction rule may also include an applicable condition. For each extraction area, check whether the current page meets its applicable condition. If so, convert the rectangular relative coordinates defined by the extraction area into absolute coordinates based on the size image, and obtain the text in this rectangular area as the extracted data.

[0106] In this embodiment, a document type hint may also be set. The document type hint is a string that indicates the document type, so as to adjust the layout analysis algorithm more in line with the file type and implement expansion of the image type.

[0107] In summary, users can define an extraction rule for each type of digital image that needs to be extracted, such as defining an extraction rule for ID cards, defining an extraction rule for driver's licenses, and so on. Multiple data extraction rules are managed through the user interface provided by the software product. In the desktop version of the software product, data extraction of a type of files can be processed in batches, and each batch uses an extraction rule; in the server version of the software product, the service caller calls the extraction function through the network service interface, calling the specified extraction rule identifier and an image file (which can contain multiple page images) at a time, and the extraction results are returned to the caller in JSON format.

[0108] It should be noted that when we need to extract structured information from a file with a new type of text layout, we only need to analyze the text layout of the file to obtain the extraction rules, and then we can extract the structured information based on the extraction rules. There is no need to use a large number of samples for annotation and training of neural network models, which saves the process of collecting samples, manual annotation and training, and can quickly and flexibly increase the extraction capabilities of new types of files / digital images.

[0109] In summary, the method for extracting structured information from an image provided by the embodiment of the present application obtains extraction rules by analyzing the text layout in a digital image, and then determines the extraction area based on the type of structured information, and extracts structured information from the extraction area using the extraction rules. There is no need to use a large number of samples for annotation and training of a neural network model, which eliminates the process of collecting samples, manual annotation and training, and can quickly and flexibly increase the extraction capabilities of new types of files / digital images. In addition, the training and reasoning of neural networks require the computing resources of a GPU, while the algorithm for analyzing text layout does not require GPU support, and the hardware requirements are much lower.

[0110] like Figure 3 As shown, it shows a structural block diagram of a structured information extraction device in an image provided by an embodiment of the present application. The structured information extraction device in the image can be applied to a computer device installed with the above-mentioned desktop version or server version software product. The structured information extraction device in the image may include:

[0111] An acquisition module 310 is used to acquire a digital image to be extracted;

[0112] The recognition module 320 is used to perform OCR processing on the digital image to obtain an OCR recognition result, wherein the OCR recognition result includes the text in the digital image and the position of the text in the digital image;

[0113] A setting module 330 is used to determine an extraction area in the digital image according to a type corresponding to the structured information if structured information needs to be extracted, where the types of structured information include smart tag type, table type, and fixed position data type;

[0114] The extraction module 340 is used to extract data from the extraction area according to the preset extraction rules and the OCR recognition results, and to form structured information with the data and the document attribute names corresponding to the extraction rules. The extraction rules are obtained by analyzing the text layout in the digital image.

[0115] In an optional embodiment, the setting module 330 is further used to:

[0116] When the type is a smart label type or a fixed position data type, detecting whether the digital image meets the trigger condition of the trigger, if the digital image meets the trigger condition, obtaining the rectangular area set in the trigger, and determining the rectangular area as the extraction area;

[0117] When the type is a table type, the horizontal and vertical lines of the table are detected in the digital image, and the cells in the table are determined according to the horizontal and vertical lines to obtain the extraction area.

[0118] In an optional embodiment, the extraction module 340 is further configured to:

[0119] When the type is a smart label type and the extraction area belongs to a non-regular matching type, the label text is recognized according to the OCR recognition result, and in the extraction area, the data indicated by the label text is extracted according to the extraction rules.

[0120] In an optional embodiment, the extraction module 340 is further configured to:

[0121] When the type is smart tag type and the extraction area belongs to regular matching type, a regular expression is obtained according to the extraction rule, and in the extraction area, data matching the regular expression is extracted according to the OCR recognition result.

[0122] In an optional embodiment, the extraction module 340 is further configured to:

[0123] When the type is table type, detect whether there is a header in the extraction area based on the OCR recognition result;

[0124] If there is a table header in the extraction area, extract the data in each cell corresponding to the table header from the table header row downwards;

[0125] If there is no header in the extraction area, a virtual header is generated and the data in each cell corresponding to the virtual header is extracted.

[0126] In an optional embodiment, the extraction module 340 is further configured to:

[0127] When the type is a fixed position data type, detecting whether the digital image meets the applicable conditions according to the extraction rules;

[0128] If the digital image meets the applicable conditions, the data within the extraction area is extracted according to the OCR recognition result.

[0129] In an optional embodiment, the device further comprises:

[0130] The preprocessing module is used to preprocess the digital image, and the preprocessing includes at least one of automatic correction, background removal and deformation correction, and automatic segmentation of specific certificates.

[0131] In summary, the structured information extraction device in the image provided by the embodiment of the present application obtains extraction rules by analyzing the text layout in the digital image, and then determines the extraction area based on the type of structured information, and extracts structured information from the extraction area using the extraction rules. There is no need to use a large number of samples for annotation and training of neural network models, which saves the process of collecting samples, manual annotation and training, and can quickly and flexibly increase the extraction capabilities of new types of files / digital images. In addition, the training and reasoning of neural networks require the computing resources of GPUs, while the algorithm for analyzing text layout does not require GPU support, and the hardware requirements are much lower.

[0132] An embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the method for extracting structured information from an image as described above.

[0133] An embodiment of the present application provides a computer device, wherein the computer device includes the above-mentioned device for extracting structured information from any image.

[0134] It should be noted that: when the structured information extraction device in the image provided by the above embodiment performs structured information extraction in the image, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the structured information extraction device in the image is divided into different functional modules to complete all or part of the functions described above. In addition, the structured information extraction device in the image provided by the above embodiment and the structured information extraction method embodiment in the image belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0135] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0136] The above description is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A method for extracting structured information from an image, characterized in that: The method comprises: Acquire a digital image to be extracted; Performing optical character recognition (OCR) processing on the digital image to obtain an OCR recognition result, wherein the OCR recognition result includes the text in the digital image and the position of the text in the digital image; If it is necessary to extract structured information, an extraction area is determined in the digital image according to the type corresponding to the structured information, wherein the type of the structured information includes a smart tag type, a table type, and a fixed position data type; Extracting data from the extraction area according to a preset extraction rule and the OCR recognition result, and composing structured information with the data and the document attribute name corresponding to the extraction rule, wherein the extraction rule is obtained by analyzing the text layout in the digital image, and the extraction rule is obtained by analyzing the structure of the page according to user-defined settings or national standards; The step of determining the extraction area in the digital image according to the type corresponding to the structured information includes: when the type is a smart tag type or a fixed position data type, detecting whether the digital image satisfies a trigger condition of a trigger, and if the digital image satisfies the trigger condition, obtaining a rectangular area set in the trigger, and determining the rectangular area as the extraction area; wherein the trigger is formed by a user selecting a sample image, delimiting a rectangular area with a mouse, and specifying and setting a trigger text, and a computer device recording relative coordinates of the rectangular area and the trigger text, wherein the relative coordinates of the rectangular area are coordinates relative to the length and width of the sample image; The extracting data from the extraction area according to the preset extraction rules and the OCR recognition results includes: when the type is a smart label type and the extraction area belongs to a non-regular matching type, identifying the label text according to the OCR recognition results, and in the extraction area, extracting the data indicated by the label text according to the extraction rules, wherein the extraction rules are provided with information of specifying the label text, data content position, skipping times, corresponding document attribute name, whether the label is multi-line, whether the data content is multi-line, and applicable pages, the label text indicating the data is a marker text or a regular expression containing the target data, the data content position indicates the positional relationship between the data to be extracted and the label text, the skipping times indicates ignoring the first N labels that meet the label matching conditions, whether the label is multi-line indicates whether the label text must be on the same text line, whether the data content is multi-line indicates whether the extracted data can wrap, and the applicable pages are used to specify which pages the extraction search range applies to; The extracting data from the extraction area according to the preset extraction rules and the OCR recognition result includes: when the type is a table type, detecting whether there is a header in the extraction area according to the OCR recognition result; if there is a header in the extraction area, extracting data in each cell corresponding to the header from the header row downward; if there is no header in the extraction area, generating a virtual header, and extracting data in each cell corresponding to the virtual header; The extracting data from the extraction area according to the preset extraction rules and the OCR recognition result includes: when the type is a fixed position data type, detecting whether the digital image meets the applicable conditions according to the extraction rules; if the digital image meets the applicable conditions, extracting the data located in the extraction area according to the OCR recognition result.

2. The method for extracting structured information from an image according to claim 1, characterized in that: The step of determining an extraction area in the digital image according to the type corresponding to the structured information includes: When the type is a table type, table lines are detected in the digital image, and cells in the table are determined according to the table lines to obtain an extraction area.

3. The method for extracting structured information from an image according to claim 1, characterized in that: The extracting data from the extraction area according to the preset extraction rule and the OCR recognition result includes: When the type is a smart tag type and the extraction area belongs to a regular matching type, a regular expression is obtained according to the extraction rule, and in the extraction area, data matching the regular expression is extracted according to the OCR recognition result.

4. The method for extracting structured information from an image according to any one of claims 1 to 3, characterized in that: The method further comprises: The digital image is preprocessed, and the preprocessing includes at least one of automatic deflection correction, background removal and deformation correction, and automatic segmentation of specific certificates.

5. A device for extracting structured information from an image, characterized in that: The device comprises: An acquisition module, used for acquiring a digital image to be extracted; A recognition module, configured to perform optical character recognition (OCR) processing on the digital image to obtain an OCR recognition result, wherein the OCR recognition result includes the text in the digital image and the position of the text in the digital image; A setting module, for determining an extraction area in the digital image according to a type corresponding to the structured information if structured information needs to be extracted, wherein the types of the structured information include a smart tag type, a table type, and a fixed position data type; An extraction module, for extracting data from the extraction area according to a preset extraction rule and the OCR recognition result, and combining the data and the document attribute name corresponding to the extraction rule into structured information, wherein the extraction rule is obtained by analyzing the text layout in the digital image, and the extraction rule is obtained by analyzing the structure of the page according to user-defined settings or national standards; The setting module is further used for: when the type is a smart label type or a fixed position data type, detecting whether the digital image meets the trigger condition of the trigger, and if the digital image meets the trigger condition, obtaining the rectangular area set in the trigger, and determining the rectangular area as the extraction area; wherein the trigger is formed by the user selecting a sample image, delimiting a rectangular area with a mouse, and specifying and setting a trigger text, and the computer device records the relative coordinates of the rectangular area and the trigger text, and the relative coordinates of the rectangular area are the coordinates relative to the length and width of the sample image; The extraction module is further used for: when the type is a smart label type and the extraction area belongs to a non-regular matching type, identifying the label text according to the OCR recognition result, and in the extraction area, extracting the data indicated by the label text according to the extraction rule, wherein the extraction rule is provided with information of specifying the label text, the data content position, the number of skips, the corresponding document attribute name, whether the label is multi-line, whether the data content is multi-line, and the applicable page, the label text indicating the data is a marker text or a regular expression containing the target data, the data content position indicates the positional relationship between the data to be extracted and the label text, the number of skips indicates ignoring the first N labels that meet the label matching conditions, whether the label is multi-line indicates whether the label text must be on the same text line, whether the data content is multi-line indicates whether the extracted data can wrap, and the applicable page is used to specify which pages the extraction search range applies to; The extraction module is further used for: when the type is a table type, detecting whether there is a header in the extraction area according to the OCR recognition result; if there is a header in the extraction area, extracting data in each cell corresponding to the header from the header row downward; if there is no header in the extraction area, generating a virtual header and extracting data in each cell corresponding to the virtual header; The extraction module is further used to: when the type is a fixed position data type, detect whether the digital image meets the applicable conditions according to the extraction rules; if the digital image meets the applicable conditions, extract the data located in the extraction area according to the OCR recognition result.

6. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method for extracting structured information from an image as described in any one of claims 1 to 4.

7. A computer device, characterized in that: The computer device comprises: the device for extracting structured information from an image as described in claim 5.

Images