Multi-format document structured data feature extraction system based on deep learning

Abstract

The invention discloses a multi-format document structured data feature extraction system based on deep learning. The multi-format document structured data feature extraction system comprises the following steps: S1, preprocessing a document; carrying out format standardization processing on the input PDF report, wherein the format standardization processing comprises document denoising and rotation correction; s2, format self-adaptive identification is carried out; and automatically identifying the format type of the input document based on the existing document sample training model by applying a machine learning algorithm. The invention relates to the technical field of data processing, and has the beneficial effects that the efficiency is improved: the data processing speed can be remarkably improved by automatically identifying the document format and adopting a targeted data extraction method, and the data extraction speed is greatly improved compared with the prior art. And in combination with the OCR technology and the natural language processing technology, the data extraction accuracy is remarkably improved, the accuracy is improved to 90% or above, and the requirement for manual correction is effectively reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a multi-format document structured data feature extraction system based on deep learning. Background Art

[0002] With the development of information technology, data has become an important foundation for corporate decision-making and business analysis. Especially in the corporate structure of many branches, each branch will regularly submit various reports, which are usually in the form of electronic documents such as PDF. These documents contain a large amount of unstructured data, such as text and tables, and their content is crucial for the company's operational analysis and decision-making. However, due to the non-uniform report formats of different branches, data extraction and processing face huge challenges. Existing data processing technologies, such as OCR (Optical Character Recognition) technology and NLP (Natural Language Processing) technology, although they can achieve text recognition and a certain degree of data extraction, still have limitations in processing documents and tables in various formats. Therefore, it is necessary to design a multi-format document structured data extraction and optimization processing system that can automatically identify document formats and extract fixed data features for different document formats to facilitate subsequent modeling and analysis.

[0003] Although existing technologies can achieve data extraction to a certain extent, they still have the following major disadvantages: Strong format dependence: It is necessary to design and maintain extraction templates separately for each document format, which is a large workload and difficult to adapt to slight changes or updates in the format. Low processing efficiency: Especially when faced with a large number of diverse documents, existing technologies are difficult to complete data extraction quickly and accurately, affecting the efficiency of data processing. Limited accuracy: Due to the diversity of tables and texts in documents and the limitations of OCR technology, the accuracy of data extraction still needs to be improved. Poor adaptability: Existing technologies are difficult to effectively handle the diversity of report formats of different branches, especially in automatically identifying new or rare document formats.

[0004] In view of the above shortcomings, the present invention proposes a comprehensive data extraction and optimization processing system, which aims to achieve efficient and accurate extraction of unstructured data in multi-format documents through advanced data processing technology and algorithms, and improve the automation and adaptability of data processing through intelligent data processing processes, ultimately solving the problems of low efficiency and low accuracy of existing technologies in processing diverse document formats. Summary of the invention

[0005] The purpose of the present invention is to solve the above problems and to design a multi-format document structured data feature extraction system based on deep learning.

[0006] The technical solution of the present invention to achieve the above-mentioned purpose is a multi-format document structured data feature extraction system based on deep learning, comprising the following steps:

[0007] Step S1: Document preprocessing: format standardization of the input PDF report, including document denoising and rotation correction;

[0008] Step S2: format adaptive recognition: applying a machine learning algorithm to automatically recognize the format type of the input document based on the existing document sample training model;

[0009] Step S3: extracting unstructured data; according to the identified document format, using OCR technology combined with natural language processing technology to extract unstructured data from the document;

[0010] Step S4: Data post-processing: cleaning, verifying and formatting the extracted data, including removing redundant information, data type conversion and missing value processing;

[0011] Step S5: Data optimization and integration: Apply data mining and machine learning techniques to further analyze and optimize the processed data.

[0012] Furthermore, in step S1: since some tables have table fault problems, a fuzzy matching algorithm is used to solve the table fault problem. The fuzzy matching calculates the minimum number of editing operations required for the conversion between two character strings based on the Levenshtein Distance algorithm. The smaller the edit distance, the greater the similarity between the two character strings:

[0013]

[0014] in, Indicates that the string Convert to string The minimum number of single-character edits (insertions, deletions, or substitutions) required. Represents a string Length, Indicates taking the maximum value, Represents the similarity score.

[0015] Furthermore, in step S2: using a deep learning model, such as a convolutional neural network (CNN), based on a large number of annotated document samples for training, to achieve automatic recognition of the format of new input documents, preliminary classification of document types, and then cooperate with targeted data extraction solutions. This step automatically determines the type and structure of the document by identifying the structural features of the document, such as the layout of titles, paragraphs, and tables:

[0016]

[0017] in, For the input document, Train the model for existing document samples, Document format types identified by machine learning algorithms;

[0018] Furthermore, in the step S3: applying OCR (Optical Character Recognition) technology, the image text in the document is converted into editable and searchable text data.

[0019]

[0020] in Accepts two inputs: document format type and Documentation , the output is a long text .

[0021] Combined with NLP (Natural Language Processing) technology, the extracted text content is semantically analyzed and structured, unstructured sensitive data is extracted, rating data is encoded, and digital information is accurately matched.

[0022]

[0023] in Accepts long text input And output the extracted unstructured data ;

[0024] Furthermore, in step S4: the text data extracted by OCR is cleaned, including removing non-text elements and correcting recognition errors.

[0025] Use data validation techniques, such as regular expression matching, to ensure the correctness and consistency of data format.

[0026] Format the extracted data, such as standardizing date and currency data.

[0027]

[0028] in, Accept the data extracted in the previous step And output the formatted data ;

[0029] Furthermore, in step S5: data mining techniques, such as cluster analysis, are used to further analyze the extracted data to discover potential relationships between the data.

[0030] Apply machine learning algorithms to classify or predict data and improve its application value.

[0031]

[0032] in, Optimizes functions for data, accepts formatted data , the data type obtained after applying the algorithm is output and has data with higher application value.

[0033] The multi-format document structured data feature extraction system based on deep learning produced by the technical solution of the present invention has improved efficiency: by automatically identifying the document format and adopting a targeted data extraction method, the present invention can significantly improve the speed of data processing. Compared with the existing technology, the data extraction speed is greatly improved. Improved accuracy: Combining OCR technology and natural language processing technology, the present invention has significantly improved the accuracy of data extraction, and the accuracy rate has been increased to more than 90%, effectively reducing the need for manual correction.

[0034] Strong adaptability: The present invention can adapt to the format diversity of reports of different branch companies, reduce the dependence on specific format templates, and improve the versatility and flexibility of the system. Economic benefits: By improving data processing efficiency and accuracy, the present invention can help enterprises reduce the manpower and time costs of data processing, improve the efficiency and quality of decision-making, and thus bring significant economic benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a block diagram of the multi-format document structured data feature extraction system based on deep learning described in the present invention. DETAILED DESCRIPTION

[0036] The present invention will be described in detail below in conjunction with the accompanying drawings. Figure 1 As shown in the figure, a multi-format document structured data feature extraction system based on deep learning is

[0037] Step S1: Document preprocessing: The input PDF report is formatted and standardized, including document denoising, rotation correction, etc., to facilitate subsequent data extraction.

[0038] Step S2: Format adaptive recognition: Apply machine learning algorithms to automatically identify the format type of input documents based on existing document sample training models, reducing reliance on fixed templates.

[0039] Step S3: Extracting unstructured data: Based on the identified document format, OCR technology combined with natural language processing technology is used to extract unstructured data (such as text and tables) from the document.

[0040] Step S4: Data post-processing: The extracted data is cleaned, verified and formatted, including removing redundant information, data type conversion, missing value processing, etc., to ensure the accuracy and consistency of the data.

[0041] Step S5: Data optimization and integration: Apply data mining and machine learning techniques to further analyze and optimize the processed data to improve the value of the data.

[0042] Each step uses advanced data processing technology and algorithms to ensure the specificity and executability of the operation. For example, in the data post-processing step, specific rules for data cleaning can be set, such as regular expression matching for data format standardization, missing value processing can use interpolation or model-based prediction methods, etc.

[0043] In this embodiment,

[0044] Step S1 document preprocessing:

[0045] Since some tables have table faults, a fuzzy matching algorithm is used to solve the table fault problem. Fuzzy matching calculates the minimum number of editing operations required to convert two strings based on the Levenshtein Distance algorithm. The smaller the edit distance, the greater the similarity between the two strings:

[0046]

[0047] in, Indicates that the string Convert to string The minimum number of single-character edits (insertions, deletions, or substitutions) required. Represents a string Length, Indicates taking the maximum value, Represents the similarity score.

[0048] Step S2: format adaptive recognition:

[0049] Using deep learning models, such as convolutional neural networks (CNN), trained based on a large number of annotated document samples, we can automatically identify the format of new input documents, preliminarily classify document types, and then use targeted data extraction solutions. This step automatically determines the type and structure of documents by identifying the structural features of documents, such as the layout of titles, paragraphs, and tables:

[0050]

[0051] in, For the input document, Train the model for existing document samples, The document format type identified by the machine learning algorithm.

[0052] Step S3: Extract unstructured data:

[0053] Apply OCR (Optical Character Recognition) technology to convert image text in documents into editable and searchable text data.

[0054]

[0055] in Accepts two inputs: document format type and Documentation , the output is a long text .

[0056] Combined with NLP (Natural Language Processing) technology, the extracted text content is semantically analyzed and structured, unstructured sensitive data is extracted, rating data is encoded, and digital information is accurately matched.

[0057]

[0058] in Accepts long text input And output the extracted unstructured data .

[0059] Step S4: data post-processing:

[0060] Clean the text data extracted by OCR, including removing non-text elements and correcting recognition errors.

[0061] Use data validation techniques, such as regular expression matching, to ensure the correctness and consistency of data format.

[0062] Format the extracted data, such as standardizing date and currency data.

[0063]

[0064] in, Accept the data extracted in the previous step And output the formatted data .

[0065] Step S5: Data optimization and integration:

[0066] Use data mining techniques, such as cluster analysis, to further analyze the extracted data and discover potential connections between the data.

[0067] Apply machine learning algorithms to classify or predict data and improve its application value.

[0068]

[0069] in, Optimizes functions for data, accepts formatted data , the data type obtained after applying the algorithm is output and has data with higher application value.

[0070] The above technical solutions only reflect the preferred technical solutions of the technical solutions of the present invention. Some changes that may be made to certain parts thereof by technicians in this technical field all reflect the principles of the present invention and fall within the protection scope of the present invention.

Claims

1. A multi-format document structured data feature extraction system based on deep learning, characterized by: The following steps are involved: Step S1: Document preprocessing: format standardization of the input PDF report, including document denoising and rotation correction; Step S2: format adaptive recognition: applying a machine learning algorithm to automatically recognize the format type of the input document based on the existing document sample training model; Step S3: extracting unstructured data; according to the identified document format, using OCR technology combined with natural language processing technology to extract unstructured data from the document; Step S4: Data post-processing: cleaning, verifying and formatting the extracted data, including removing redundant information, data type conversion and missing value processing; Step S5: Data optimization and integration: Apply data mining and machine learning techniques to further analyze and optimize the processed data.

2. According to the multi-format document structured data feature extraction system based on deep learning according to claim 1, it is characterized in that: In step S1: since some tables have table faults, a fuzzy matching algorithm is used to solve the table fault problem. The fuzzy matching algorithm calculates the minimum number of editing operations required for the conversion between two character strings based on the Levenshtein Distance algorithm. The smaller the edit distance, the greater the similarity between the two character strings: ； in, Indicates that the string Convert to string the minimum number of single-character edits (insertions, deletions, or substitutions) required, Represents a string Length, Indicates taking the maximum value, Represents the similarity score.

3. The multi-format document structured data feature extraction system based on deep learning according to claim 1, characterized in that: In step S2, a deep learning model, such as a convolutional neural network (CNN), is used to train based on a large number of annotated document samples to automatically identify the format of a new input document, preliminarily classify the document type, and then cooperate with a targeted data extraction solution. This step automatically determines the type and structure of the document by identifying the structural features of the document, such as the layout of the title, paragraph, and table: ； in, For the input document, Train the model for an existing sample of documents, The document format type identified by the machine learning algorithm.

4. According to the multi-format document structured data feature extraction system based on deep learning in claim 1, it is characterized in that: In step S3: Optical Character Recognition (OCR) technology is applied to convert the image text in the document into editable and searchable text data. ； in Accepts two inputs: document format type and Documentation , the output is a long text , Combined with NLP (Natural Language Processing) technology, the extracted text content is semantically analyzed and structured, unstructured sensitive data is extracted, rating data is encoded, and digital information is accurately matched. ； in Accepts long text input And output the extracted unstructured data .

5. The multi-format document structured data feature extraction system based on deep learning according to claim 1, characterized in that: In step S4: the text data extracted by OCR is cleaned, including removing non-text elements and correcting recognition errors. Use data validation techniques, such as regular expression matching, to ensure the correctness and consistency of data formats; Format the extracted data, such as standardizing dates and currency data, ； in, Accept the data extracted in the previous step And output the formatted data .

6. The multi-format document structured data feature extraction system based on deep learning according to claim 1, characterized in that: In step S5: using data mining techniques, such as cluster analysis, to further analyze the extracted data to discover potential relationships between the data. Apply machine learning algorithms to classify or predict data and improve the application value of data. ； in, Optimizes functions for data, accepts formatted data , the data type obtained after applying the algorithm is output and has data with higher application value.

Images