Data processing method and system for files

Abstract

The application provides a data processing method and system for a file, the data processing method for a file comprising: judging whether a file can directly perform text extraction, if yes, entering the next step; performing text extraction on the file to obtain text content of the file; locating a target page by using a regular matching technology on the obtained text content; extracting the located target page as an image file; removing a page watermark of the image file by using an image processing technology; extracting page data of the image file by using a computer vision recognition technology, and extracting a data arrangement structure of the page data by using a layout recognition technology; and performing a format correction operation on the extracted page data to perfect the data arrangement structure. Compared with a simple text extraction method and an image extraction method, the data processing method and system for a file according to the embodiment of the application can greatly improve extraction efficiency and extraction accuracy, and can generate structured page data.

Description

Technical Field

[0001] This invention relates to the field of document data processing, and more specifically to a data processing method and system for documents. Background Technology

[0002] Tender document data processing refers to extracting structured data such as itemized price lists and basic company information from tender documents. In today's highly competitive market, each tender process involves a massive volume of tender documents. Filtering out crucial information from this vast amount of data using traditional manual reading methods is not only inefficient but sometimes even impractical.

[0003] Existing document data processing methods can be mainly divided into two types: 1. Text extraction method: Directly extracting text information and corresponding position information from multiple lines of text in the document, and performing operations such as encoding and decoding on the text and position information to obtain unstructured data. 2. Image extraction method: Extracting document pages into image files, using computer vision recognition technology to extract page content, and using layout recognition technology to restore the data structure to obtain structured data. Generally speaking, the text extraction method is fast, has high accuracy, and is not affected by other factors (such as watermarks). Image extraction method, on the other hand, has a wide range of applications; by converting documents into image files, it can process all types of PDF documents.

[0004] However, text extraction methods can only process ordinary PDF documents using standard character encoding and cannot process scanned PDF documents. As for image extraction methods, tender documents generally have many pages, and each page contains a watermark. If image recognition and layout recognition technologies are used directly to extract the content of each page, the extraction efficiency and accuracy are very low, and the recognition accuracy is easily affected by other types of factors (such as watermarks), making it difficult to meet the needs of large-scale applications.

[0005] This invention aims to solve the above problems by proposing a data processing method and system for tender documents, which combines the advantages of text extraction and image extraction methods while avoiding their disadvantages. Summary of the Invention

[0006] This invention proposes a data processing method and system for bid documents, which can efficiently extract structured data from unstructured data in bid documents and improve bid evaluation efficiency.

[0007] A first aspect of the present invention provides a data processing method for a file, comprising: Step 1: determining whether a file can be directly extracted for text; if so, proceeding to the next step; Step 2: extracting text from the file to obtain the text content of the file; Step 3: locating a target page from the obtained text content using regular expression matching technology; Step 4: extracting the located target page into an image file; Step 5: removing the page watermark from the image file using image processing technology; Step 6: extracting page data from the image file using computer vision recognition technology, and extracting the data layout structure of the page data using layout recognition technology; and Step 7: performing format correction operations on the extracted page data to improve the data layout structure.

[0008] Furthermore, step 1 also includes: determining whether a document can be directly extracted as text; if not, extracting each page of the document as an image file, and then proceeding to step 5.

[0009] Further, step 1 may also include: step 8: extracting the first n pages of text content from the file, where n is a positive integer; and step 9: determining whether the extracted first n pages of text content contain more than m characters. If so, it is determined that text extraction technology can be applied to the file; otherwise, it is determined that text extraction technology cannot be applied to the file. The characters can be Chinese or foreign language characters. Further, n equals 10, and m equals 10.

[0010] According to some embodiments of the present invention, the file may be a PDF document.

[0011] Furthermore, in step 7, the format correction operation includes filling, shifting, and completing operations on the extracted page data.

[0012] Furthermore, the image processing technology includes optical character recognition (OCR), and the computer vision recognition technology includes optical character recognition (OCR).

[0013] A second aspect of the present invention provides a data processing system, including a processor and a memory, the memory storing instructions for performing a data processing method for a file according to any of the preceding first aspect embodiments, the processor executing the instructions stored in the memory.

[0014] This invention converts PDF documents into image files and uses image processing technology to remove page watermarks, thereby filtering out interfering images and improving image recognition accuracy. For tender documents that can be extracted using text extraction methods, regular expression extraction technology is first used to locate the target page, and then image recognition and layout analysis are performed. Compared with performing image recognition on all pages to locate the target page and then extracting text, this invention can greatly improve the efficiency of tender document data extraction. Attached Figure Description

[0015] Figure 1 This is a flowchart of a data processing method for a file according to an embodiment of the present invention; and

[0016] Figure 2 This is a partial schematic diagram of step 1 according to another embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and beneficial effects of the present invention clearer, the technical solutions of the present invention will be described in detail below with reference to the accompanying drawings and a specific embodiment.

[0018] See Figure 1 The first aspect of the present invention provides a data processing method for files, comprising the following steps.

[0019] Step 1: Determine if a document can be directly extracted for text. If so, proceed to Step 2.

[0020] The text extraction method can only process ordinary PDF documents using standard character encoding and cannot process scanned PDF documents. If a document can be directly extracted for text, then text extraction is performed directly. If a document cannot be directly extracted for text, then each page of the document is extracted as an image file before proceeding to step 5.

[0021] For example, see Figure 2 Step 1 may further include: Step 8: Extracting the first n = 10 pages of text content from the file, where n is a positive integer; and Step 9: Determining whether the extracted first n = 10 pages of text content contain m = 10 or more characters. If yes, it is determined that text extraction technology can be applied to the file, i.e., text extraction can be performed directly on the file; otherwise, it is determined that text extraction technology cannot be applied to the file, i.e., text extraction cannot be performed directly on the file. The characters can be Chinese or foreign language characters.

[0022] Step 2: Extract text from the file to obtain its text content.

[0023] Step 3: Use regular expression matching to locate the target page from the obtained text content.

[0024] For the acquired text content, regular expression matching technology is used to locate the target page. The target page is the page from which page data is extracted. For example, the target page often contains content that the tenderer is interested in, such as the bidder's qualifications, the bidder's business capabilities, the bidder's qualification certificates, the bidder's production conditions and production capacity, etc. It can be a field, the name of a table, or the content of a cell.

[0025] Step 4: Extract the located target page as an image file.

[0026] Step 5: Use image processing techniques to remove the page watermark from the image file.

[0027] For various reasons such as anti-counterfeiting or confidentiality, tender documents often contain page watermarks, which may include the bidder's signature, name, or other content. The presence of page watermarks hinders the efficiency and accuracy of text extraction. The extracted text is affected by the watermark, resulting in poor accuracy, and the watermark itself may also be extracted as unwanted data. Image extraction methods are labor-intensive, with low extraction efficiency and accuracy, further hampered by the watermark. Here, we first extract the target page as an image file, then use image processing techniques to remove the watermark from the image file, thus eliminating this obstacle to data extraction. Image processing techniques can be various general computer image processing techniques, such as Optical Character Recognition (OCR). OCR refers to the process of using electronic devices (such as scanners or digital cameras) to examine characters printed on paper, determining their shape by detecting dark and light patterns, and then translating the shape into computer text using character recognition methods.

[0028] Step 6: Use computer vision recognition technology to extract page data from the image file, and use layout recognition technology to extract the data arrangement structure of the page data.

[0029] Computer vision recognition technology is used to extract page data from image files. However, this page data is initially unstructured. For example, in a table, the content of each cell is extracted, but the overall layout structure—the table's format—is not. Then, layout recognition technology is used to extract the data layout structure, thus obtaining the page data's format. For instance, company name, turnover, staff size, production capacity, and technical qualifications are unstructured page data. After obtaining the data layout structure (e.g., table format), this data becomes structured data (e.g., a complete table), as shown in Table 1 below.

[0030] Table 1

[0031] Company Name Turnover Staff size Production capacity Technical Qualification

[0032] The computer vision recognition technology mentioned here can refer to various general-purpose computer vision recognition technologies, such as Optical Character Recognition (OCR). Page layout recognition technology is the process of automatically analyzing, recognizing, and understanding images, text, table information, and positional relationships within a page. The page layout recognition technology mentioned here can refer to various general-purpose page layout recognition technologies.

[0033] Step 7: Correct the format of the extracted page data and improve the data layout structure.

[0034] Even after the extracted page data is structured, there may still be some formatting issues. Therefore, format correction operations are needed, such as filling, shifting, and completing the extracted page data, to further improve the data layout structure.

[0035] A second aspect of the present invention provides a data processing system, including a processor and a memory, the memory storing instructions for performing a data processing method for a file according to any of the preceding first aspect embodiments, the processor executing the instructions stored in the memory.

[0036] According to the data processing method and system for documents of the present invention, by extracting PDF documents into image files and removing page watermarks using image processing technology, interference images can be filtered out, and the accuracy of image recognition can be improved. For tender documents that can be extracted using text extraction methods, the target page is first located using regular expression extraction technology, and then image recognition and layout analysis are performed. Compared with performing image recognition on all pages to locate the target page and then extracting text, the efficiency of tender document data extraction can be greatly improved, and finally, structured data can be generated accurately and quickly.

[0037] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A data processing method for files, comprising: Step 1: Determine if a document can be directly extracted for text. If so, proceed to the next step. Step 2: Extract text from the file to obtain its text content; Step 3: Use regular expression matching to locate the target page from the acquired text content; Step 4: Extract the located target page as an image file; Step 5: Use image processing techniques to remove the page watermark from the image file; Step 6: Use computer vision recognition technology to extract page data from the image file, and use layout recognition technology to extract the data arrangement structure of the page data; as well as Step 7: Correct the format of the extracted page data and improve the data layout structure; Step 1 further includes: determining whether a document can be directly extracted as text; if not, extracting each page of the document as an image file, and then proceeding to step 5. Step 1 further includes: Step 8: Extract the text content of the first n pages of the file, where n is a positive integer; and Step 9: Determine whether the extracted first n pages of text content contain more than m characters. If yes, then determine that text extraction technology can be applied to the file; otherwise, determine that text extraction technology cannot be applied to the file.

2. The data processing method for files according to claim 1, characterized in that, The file is a PDF document.

3. The data processing method for files according to claim 1, characterized in that, The format correction operation includes filling, shifting, and completing the extracted page data.

4. The data processing method for files according to claim 1, characterized in that, The characters can be Chinese or foreign language characters.

5. The data processing method for files according to claim 1, characterized in that, n equals 10, m equals 10.

6. The data processing method for files according to claim 1, characterized in that, The image processing technology includes Optical Character Recognition (OCR).

7. The data processing method for files according to claim 1, characterized in that, The computer vision recognition technology includes optical character recognition (OCR) technology.

8. A data processing system comprising a processor and a memory, the memory storing instructions for performing a data processing method for a file according to any one of claims 1-7, the processor executing the instructions stored in the memory.

Images