Method and system for processing unstructured data in field of intelligent document processing

Abstract

The invention discloses an unstructured data processing method and system in the field of intelligent document processing, and the method comprises the steps: carrying out the file format recognition and content extraction of a to-be-processed document; storing the structured data of the file in the Excel format into MySQL (Structured Query Language) data; dividing document contents extracted from other formatted files into unstructured data and structured data; carrying out full-text cleaning and blocking processing on the unstructured data, and storing the unstructured data in a Milvus vector database; data cleaning and rule base analysis are carried out on the structured data; performing deep analysis on the extracted document content by utilizing a large language model, generating an abstract and constructing a document tree; and after data cleaning and rule base analysis are carried out on the structured data in the document content, the structured data are stored in the MySQL database. According to the method, efficient processing and intelligent analysis of the unstructured data can be realized, so that the data processing speed is increased, the resource utilization rate is optimized, and the data mining capability is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent document processing technology, and more specifically to a method and system for processing unstructured data in the field of intelligent document processing. The applicable fields include but are not limited to computer science, artificial intelligence, pattern recognition, natural language processing and machine learning, etc., and the purpose is to improve the automation and intelligence level of document processing and optimize the management and utilization efficiency of unstructured data. Background Art

[0002] With the rapid development of information technology, especially the widespread application of the Internet and big data technology, the amount of unstructured data (such as text, images, audio and video, etc.) has exploded. Due to the diverse formats and complex structures of these data, traditional relational databases are difficult to effectively manage and analyze them. However, these data contain rich information and knowledge. How to efficiently process and utilize these data has become a key challenge in the current technological development.

[0003] In the field of intelligent document processing, the research and application of unstructured data processing methods are particularly important. Traditional methods have problems such as slow processing speed, low efficiency, and low accuracy. In addition, they lack the understanding of the deep semantics of unstructured data and are difficult to meet the needs of complex document processing tasks. Therefore, developing new unstructured data processing methods to improve the level of intelligent data processing has become an urgent problem that technicians in this field need to solve. Summary of the invention

[0004] In view of this, the present invention provides a method and system for processing unstructured data in the field of intelligent document processing, which can at least partially solve the problems of slow processing speed, low efficiency and low accuracy existing in traditional methods; the present invention combines the latest artificial intelligence technology and machine learning algorithms to achieve efficient processing and intelligent analysis of unstructured data, so as to improve data processing speed, optimize resource utilization and enhance data mining capabilities.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] In a first aspect, an embodiment of the present invention provides a method for processing unstructured data in the field of intelligent document processing, comprising the following steps:

[0007] S10, identify the file format and extract the content of the document to be processed; generate SQL statements from the Excel format file through data cleaning and LLM, and store the structured data in MySQL data;

[0008] S20, dividing the document contents extracted from files in formats other than Excel format into unstructured data and structured data; performing full-text cleaning and block processing on the unstructured data in the document contents; performing data cleaning and rule base parsing on the structured data in the document contents;

[0009] S30, using a large language model to deeply analyze the extracted document content and generate a summary;

[0010] S40, constructing a document tree based on the finite state machine and XML attribute parsing of the extracted document content;

[0011] S50, convert the unstructured data in the document content into vector representation after full-text cleaning and block processing, and store it in the Milvus vector database; after data cleaning and rule base parsing, store the structured data in the document content in the MySQL database.

[0012] Furthermore, the method further comprises the following steps:

[0013] According to the needs of user queries, vector searches are performed in the Milvus vector database to recall text blocks, and / or target chapters are recalled by traversing the document tree, and / or SQL statements are executed through the MySQL database to recall accurate answers.

[0014] Furthermore, the method further comprises the following steps:

[0015] Display data recalled from the Milvus vector database and / or MySQL database, and / or provide a visualization interface of the document tree.

[0016] Furthermore, according to the user's query requirements, a vector search is performed in the Milvus vector database to recall text blocks, including:

[0017] Convert user query requirements into vectors and perform fine-tuning through vector normalization, feature selection and weighting, and regularization processing;

[0018] A similarity search is performed in the Milvus vector database to recall text blocks with higher similarity; the similarity is calculated using cosine similarity, Euclidean distance and inner product calculation methods.

[0019] Furthermore, content extraction is performed on the document to be processed, including:

[0020] ResNet-50 and improved U-Net architecture are used to detect the layout area and extract text, titles, tables and pictures; the improved U-Net architecture introduces residual connections based on the U-Net architecture;

[0021] The CTC loss function is used in combination with the GRU network to capture the long-term dependencies in the document and realize character recognition of the text;

[0022] Use the pre-trained graph convolutional network (GCN) to identify the row and column information in the table and extract the table data;

[0023] A multi-task learning framework combining Faster R-CNN and OCR technology is used to identify and extract text information in the seal area.

[0024] Furthermore, the unstructured data in the document content is fully cleaned, including:

[0025] The BERT model is used to understand the context of the unstructured data in the document content. The bidirectional encoder captures the long-distance dependencies in the text and automatically identifies and removes noise words and irrelevant information.

[0026] Combined with rule-based regular expressions, it cleans up known irrelevant elements including headers, footers, and page separators.

[0027] Furthermore, the unstructured data in the document content is processed in blocks, including:

[0028] An improved Transformer architecture is used to effectively capture the dependencies between the internal parts of the document and perform block processing on the document; the improved Transformer architecture introduces a self-attention mechanism.

[0029] In a second aspect, an embodiment of the present invention further provides a system for processing unstructured data in the field of intelligent document processing, including:

[0030] The document recognition and extraction module is used to identify the file format and extract the content of the documents to be processed; generate SQL statements from Excel format files through data cleaning and LLM, and store structured data in MySQL data;

[0031] The data classification and processing module is used to classify the document contents extracted from files in formats other than Excel into unstructured and structured data; perform full-text cleaning and block processing on the unstructured data in the document contents; perform data cleaning and rule base analysis on the structured data in the document contents;

[0032] Intelligent parsing and summary generation module, which uses a large language model to deeply parse the extracted document content and generate summaries;

[0033] The document tree construction module is used to construct the document tree based on the finite state machine and XML attribute parsing of the extracted document content;

[0034] The data storage and management module is used to convert the unstructured data in the document content into vector representation after full-text cleaning and block processing, and store it in the Milvus vector database; the structured data in the document content is stored in the MySQL database after data cleaning and rule base parsing.

[0035] Furthermore, it also includes:

[0036] The precise recall and query module is used to perform vector search and recall text blocks in the Milvus vector database according to user query requirements, and / or recall target chapters through document tree traversal, and / or execute SQL statements through the MySQL database to recall accurate answers.

[0037] Furthermore, it also includes:

[0038] The query and result display module is used to display the data recalled from the Milvus vector database and / or the MySQL database, and / or provide a visualization interface of the document tree.

[0039] It can be seen from the above technical solutions that compared with the prior art, the present invention has the following technical advantages:

[0040] The present invention realizes unified recognition and extraction of multiple document formats, simplifies the processing flow; improves the quality and availability of data by deep cleaning and segmenting unstructured data; generates summaries using a large language model (LLM), improves the accuracy and information content of the summaries; then, accurately analyzes complex document structures by constructing a document tree and applying a custom rule base; finally, the present invention adopts data cleaning technology and a relational database management system (RDBMS) to improve the consistency and accuracy of data.

[0041] In addition, the present invention combines the advantages of vector databases and relational databases, improves the accuracy and efficiency of recall, and realizes the visualization of query results, so that users can easily view and analyze document content. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 A flow chart of a method for processing unstructured data in the field of intelligent document processing provided by the present invention.

[0044] Figure 2 A schematic diagram of the method for processing unstructured data in the field of intelligent document processing provided by the present invention.

[0045] Figure 3 A block diagram of a system for processing unstructured data in the field of intelligent document processing provided by the present invention.

[0046] Figure 4 A schematic diagram of a system for processing unstructured data in the field of intelligent document processing provided by the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Embodiment 1:

[0049] The embodiment of the present invention discloses a method for processing unstructured data in the field of intelligent document processing, which is specially designed for processing and analyzing various types of documents, including PDF, DOCX, TXT and Excel files. Its core purpose is to convert unstructured data into structured data, while providing efficient document recognition, data extraction, content classification, intelligent parsing and accurate recall functions to meet the needs of different industries and fields. Figure 1 As shown, the method specifically includes the following steps S10 to S70:

[0050] S10. Identify the file format and extract the content of the document to be processed; generate SQL statements for the Excel format file through data cleaning and LLM, and store the structured data in MySQL data.

[0051] like Figure 2 As shown, the input document is first identified and extracted, supporting multiple file formats. For PDF, DOCX and TXT files, the document content is extracted with high accuracy. For Excel files, the product generates SQL statements through data cleaning and LLM, and directly stores structured data into the MySQL database to provide support for complex queries.

[0052] Among them, for PDF files, the PDF parsing library can be used to extract text, images and table data; for DOCX files, its XML structure is parsed to extract content; TXT files directly read plain text data; and Excel files extract values, text and formulas by parsing cell data.

[0053] For PDF and DOCX files, ensure high-precision extraction of document content, including the following four methods:

[0054] (1) In the field of document recognition, traditional layout analysis methods usually rely on manually designed rules and cannot handle complex document structures, especially when multiple elements (such as charts, tables, text blocks, etc.) coexist. In order to solve these problems, in this embodiment, an optimized layout analysis method is proposed by combining deep learning with traditional algorithms, which significantly improves the detection accuracy of complex document areas.

[0055] In order to improve the accuracy of layout area detection, we combined convolutional neural network (CNN) and image segmentation technology, using ResNet-50 and U-Net architecture. ResNet-50 is used to extract deep features of the image, while U-Net is used to perform pixel-level segmentation. This combined architecture can accurately identify different areas in the document and maintain a high recognition rate on complex layouts.

[0056] U-Net is used for pixel-level segmentation. The network layer retains more detailed information through an improved skip connection structure, avoiding the problem of inaccurate segmentation in traditional methods. The specific optimizations are as follows:

[0057] Improved U-Net architecture: Residual connections are introduced based on the traditional U-Net to help avoid the gradient vanishing problem in deep networks.

[0058] Improved accuracy: According to experimental results, the optimized network significantly improves the accuracy of region segmentation in complex documents, especially in cases where images and text are mixed and page content is highly irregular.

[0059] The formula is as follows, which represents the output of the U-Net model improved by residual connection:

[0060] y=f(X:W)+X

[0061] Where X is the input image, W is the parameter of the convolutional layer, and y is the output segmentation result. The residual connection X helps avoid information loss and ensures high-precision region segmentation.

[0062] (2) Traditional OCR technology relies heavily on template matching and character recognition, and has poor adaptability to complex text layouts and handwriting. In addition, existing OCR models cannot effectively process contextual information in documents, resulting in the inability to accurately identify synonyms and related terms.

[0063] In order to improve the accuracy of text recognition, this embodiment optimizes the CTC (Connectionist Temporal Classification) loss function based on the traditional OCR framework and combines the GRU (Gated Recurrent Unit) network to capture long-term dependencies in documents. By adding a context learning module, more semantic information can be considered when the model is output, thereby improving the accuracy of recognition.

[0064] By combining the CTC loss function and the GRU model, the model can still efficiently perform text recognition without precise alignment annotations. Through contextual prompts (such as the introduction of industry vocabulary and contextual information), the model can automatically adjust the recognition results and reduce misrecognition when processing documents in professional fields (such as finance and law).

[0065] CTC loss function optimization formula:

[0066] LCTC=-logP(O|X)

[0067] Among them, P(O|X) represents the conditional probability of the model outputting the text sequence O under the input image X. The optimized CTC loss function helps the model improve the recognition accuracy when dealing with complex typesetting and unlabeled data.

[0068] (3) Traditional table structure recognition technology often relies on manual rules or simple table boundary detection, which makes it difficult to handle complex table structures with multiple layers of nesting, especially tables containing merged cells.

[0069] In this embodiment, a table structure optimization scheme combined with a graph neural network (GNN) is proposed, where each table cell is regarded as a node in a graph, and the relationship between cells is represented by edges. By training a graph convolutional network (GCN), the model can learn the structural relationship between table cells, thereby accurately identifying row and column structures in complex tables.

[0070] GNN is used to learn the relationship between table cells. The optimized model can automatically discover the row and column information in the table and accurately extract the required data regardless of the complexity of the table. Graph neural networks effectively reduce the reliance on traditional rules and improve the recognition accuracy and robustness of table structures.

[0071] Graph convolutional network formula:

[0072]

[0073] In this formula, is the normalized adjacency matrix of the graph, H(l) is the representation of the graph nodes, W(l) is the weight of the graph convolution layer, and σ is the activation function. The optimized graph convolution network can accurately capture the structural features of the table, thereby effectively improving the efficiency of table data extraction.

[0074] (4) Traditional methods are easily affected by background interference or image blur when dealing with seal area recognition, resulting in misjudgment or omission. Especially in legal or contract documents, seal area recognition is particularly important.

[0075] In order to improve the accuracy of seal recognition, in this embodiment, a multi-task learning framework combining FasterR-CNN and OCR technology is used. In seal detection, FasterR-CNN is used for region positioning, and OCR is used to identify the text information in the seal area. Through multi-task learning, information sharing between detection and recognition tasks is ensured, and recognition accuracy is improved.

[0076] By introducing the FasterR-CNN region candidate box detection method, the detection accuracy of the seal area is effectively improved. In terms of OCR recognition, the multi-task learning framework is used to improve the recognition ability of irregular text in the seal, avoiding the problems of false detection and missed detection in traditional methods.

[0077] FasterR-CNN optimization formula:

[0078] R = {(x1, y1, x2, y2)}

[0079] Among them, (x1, y1) and (x2, y2) represent the coordinates of the seal area, and R represents all detected areas. Through the optimized Faster R-CNN model, the seal area can be detected efficiently and accurately, and the text in it can be further recognized through OCR.

[0080] S20. Divide the document contents extracted from files in formats other than Excel format into unstructured and structured data; perform full-text cleaning and block processing on the unstructured data in the document contents; and perform data cleaning and rule base parsing on the structured data in the document contents.

[0081] In this step, the document contents of PDF, DOCX and TXT files are intelligently classified into unstructured and structured data. Unstructured data is fully cleaned and segmented, combined with natural language processing (NLP) technology to extract key information and classify it, such as title, text, table, image, etc., for further analysis and storage. Structured data is converted into a format that can be used for database query through data cleaning and rule library parsing.

[0082] in:

[0083] (1) The data cleaning process is as follows:

[0084] Based on the existing text cleaning technology, this embodiment introduces a text cleaning algorithm based on deep learning, combining traditional regular expressions with custom filtering rules, fundamentally improving the effect of document cleaning.

[0085] Deep learning model optimization: The BERT (Bidirectional Encoder Representations from Transformers) model is used to understand the context of the text. The bidirectional encoder captures long-distance dependencies in the text and automatically identifies and removes noise words and irrelevant information.

[0086] BERT is used to filter irrelevant words and meaningless words in documents, reducing the error cleaning in traditional methods and maintaining the core information of the document.

[0087] Combining rule-based regular expressions with deep models: For different types of documents, we combine regular expressions and deep learning models to dynamically generate cleaning rules to further improve cleaning accuracy. Regular expressions are used to clean known irrelevant elements such as headers, footers, and page separators, while deep learning models are used to process unknown noise.

[0088] Compared with traditional methods, the accuracy of noise removal can be improved and the processing speed can be increased, especially when processing documents with chaotic structures, it can more effectively retain key information.

[0089] (2) The process of block processing is as follows:

[0090] An improved Transformer architecture is used to segment documents. The Transformer model effectively captures the dependencies between the various parts of the document through the self-attention mechanism, thereby improving the accuracy of document segmentation.

[0091] In order to further improve the segmentation effect, this embodiment adds a hierarchical segmentation strategy based on the traditional segmentation, and uses different segmentation strategies for different types of content in the document (such as paragraphs, titles, tables, etc.). Hierarchical segmentation makes the segmentation result more accurate by understanding the document structure.

[0092] Optimization algorithm formula: Let the text sequence be T = {t1, t2, …, tn}, where ti represents the i-th word in the document, and the block segmentation goal is to divide the text into different blocks B = {B1, B2, …, Bm}.

[0093] Use the improved Transformer model to calculate the relationship between words in the text through the self-attention mechanism:

[0094]

[0095] Where Q, K, and V represent query, key, and value vectors, respectively. k is the dimension of the key vector.

[0096] In this way, the model is able to consider the contextual information of the document during the chunking process, thereby improving the chunking accuracy and increasing the efficiency of document extraction.

[0097] In addition, by introducing dynamically selected small sample data, the model can adaptively adjust the learning strategy according to the retrieval results of the external knowledge base each time it is updated. Assuming that the current input text is T and the external knowledge base is K, the model is optimized according to T and K:

[0098]

[0099] Among them, θ is the model parameter, L(T,K;θ) is the loss function based on the knowledge base, and the model adaptively adjusts the optimization strategy according to external information. After combining the networking function, it can automatically adjust the cleaning and segmentation strategies according to different document types, which is more adaptable, especially when processing professional documents, the accuracy has been significantly improved.

[0100] S30. Use the large language model to deeply analyze the extracted document content and generate summaries. The large language model LLM is not only used for deep analysis of document content, but also for generating summaries to improve the accuracy and efficiency of understanding document content. It also makes full use of the latest NLP technology to ensure a deep understanding of document content.

[0101] The process of this step not only improves the speed of understanding the document content, but also provides a basis for subsequent semantic search and data retrieval.

[0102] S40, constructing a document tree based on the extracted document content based on the finite state machine and XML attribute parsing.

[0103] The hierarchical structure of the document is represented by building a document tree, which is particularly important for understanding and processing documents with complex structures. The construction of the document tree is based on finite state machines and XML attributes, ensuring the accuracy and reliability of the document structure.

[0104] S50, convert the unstructured data in the document content into vector representation after full-text cleaning and block processing, and store it in the Milvus vector database; after data cleaning and rule base parsing, store the structured data in the document content in the MySQL database.

[0105] The parsed data is stored in the Milvus vector database and the MySQL relational database. Milvus is used to store vector representations of unstructured data and supports efficient similarity searches. MySQL is used to store structured data and supports complex SQL queries. This design combines the advantages of both databases and provides a flexible data storage and retrieval solution.

[0106] S60, according to the user's query requirements, perform vector search in the Milvus vector database to recall the text block, and / or recall the target chapter by traversing the document tree, and / or execute SQL statements through the MySQL database to recall the accurate answer;

[0107] The present invention provides three precise recall methods: recalling text blocks through Milvus, accurately recalling target chapters through document trees, and recalling accurate answers through MySQL databases. These three methods together ensure that users can quickly and accurately obtain the required information, greatly improving the efficiency and accuracy of data retrieval.

[0108] The details are as follows:

[0109] 1. Recall text chunks through Milvus: Split the document into multiple text chunks, convert them into vectors using the embedding model, and store them in the Milvus vector database. When a user queries, the query is converted into a vector, and a similarity search is performed in Milvus to recall the relevant text chunks.

[0110] In the calculation of Top-N vector similarity, algorithms such as cosine similarity, Euclidean distance and inner product are used. The accuracy and efficiency of similarity calculation are improved through fine-tuning methods such as vector normalization, feature selection and weighting, and regularization processing. The comprehensive application of these technical means ensures that users can quickly and accurately obtain the required information and improves the intelligent level of data processing.

[0111] The specific algorithm for calculating Top-N vector similarity is as follows:

[0112] 1). Cosine Similarity

[0113] Cosine similarity measures the cosine of the angle between two vectors. The closer the value is to 1, the more similar the vectors are.

[0114] Fine-tuning method:

[0115] Vector normalization: Before calculating the cosine similarity, the vectors are normalized to ensure that the modulus of each vector is 1, so as to eliminate the influence of the vector length on the similarity calculation.

[0116] Feature selection: According to the task requirements, select the features that have a greater impact on the similarity calculation, remove the noise features, and improve the accuracy of the similarity calculation.

[0117] Weighted processing: Assign different weights to different features, emphasize the features that have a greater impact on similarity calculation, and improve the reliability of the calculation results.

[0118] 2). Euclidean Distance

[0119] Euclidean distance calculates the straight-line distance between two vectors. The smaller the distance, the more similar the vectors are.

[0120] Fine-tuning method:

[0121] Feature scaling: Scale the features of each dimension to ensure that different features have the same dimension in the distance calculation and avoid excessive influence of certain features on the distance calculation.

[0122] Principal Component Analysis (PCA): Use dimensionality reduction techniques such as PCA to reduce data dimensions, reduce noise, and improve the efficiency and accuracy of distance calculations.

[0123] Weighted Euclidean distance: Different weights are assigned to each dimension according to the importance of the feature, so that the distance calculation is more in line with actual needs.

[0124] 3). Dot Product

[0125] The inner product is calculated by summing the products of corresponding elements of two vectors. The larger the value, the more similar the vectors are.

[0126] Fine-tuning method:

[0127] Vector normalization: Normalize the vector to eliminate the influence of vector length on inner product calculation, so that similarity calculation focuses more on direction rather than size.

[0128] Feature selection and weighting: Select features that have an important impact on the task and assign appropriate weights to them to enhance the effect of inner product calculation.

[0129] Regularization processing: When calculating the inner product, a regularization term is added to prevent calculation deviations caused by excessively large eigenvalues ​​and improve the generalization ability of the model.

[0130] 2. Accurately recall target chapters through the document tree: Build a hierarchical structure (document tree) of the document, locate relevant chapters or paragraphs based on user queries, and quickly obtain target content.

[0131] 3. Recall accurate answers through MySQL: For structured data, store it in a MySQL database. When a user queries, use SQL statements to search the database and obtain accurate answers.

[0132] S70. Display the data recalled from the Milvus vector database and / or the MySQL database, and / or provide a visualization interface of the document tree.

[0133] This method provides a user-friendly interface for displaying query results. This interface not only displays the data recalled from Milvus and MySQL, but also provides a visualization of the document tree, allowing users to easily view and analyze document content.

[0134] The method for processing unstructured data in the field of intelligent document processing provided by the embodiment of the present invention provides users with an efficient, accurate and reliable document processing solution through its powerful document processing capabilities and flexible data management strategies. It not only improves the efficiency of document processing, but also greatly enhances the availability and operability of data through intelligent analysis and precise recall functions.

[0135] Embodiment 2:

[0136] Based on the same inventive concept, the present invention also provides a processing system for unstructured data in the field of intelligent document processing, referring to Figure 3 As shown, including:

[0137] The document recognition and extraction module is used to identify the file format and extract the content of the documents to be processed; generate SQL statements from Excel format files through data cleaning and LLM, and store structured data in MySQL data;

[0138] The data classification and processing module is used to classify the document contents extracted from files in formats other than Excel into unstructured and structured data; perform full-text cleaning and block processing on the unstructured data in the document contents; perform data cleaning and rule base analysis on the structured data in the document contents;

[0139] Intelligent parsing and summary generation, which uses large language models to deeply parse the extracted document content and generate summaries;

[0140] The document tree construction module is used to construct the document tree based on the finite state machine and XML attribute parsing of the extracted document content;

[0141] The data storage and management module is used to convert the unstructured data in the document content into vector representation after full-text cleaning and block processing, and store it in the Milvus vector database; the structured data in the document content is stored in the MySQL database after data cleaning and rule base parsing.

[0142] The precise recall and query module is used to perform vector search and recall text blocks in the Milvus vector database according to user query requirements, and / or recall target chapters through document tree traversal, and / or execute SQL statements through the MySQL database to recall accurate answers.

[0143] Query and result display module, used to display the data recalled from Milvus vector database and / or MySQL database, and / or provide a visual interface of document tree

[0144] In the field of financial analysis, accurately and quickly extracting key information from financial reports is crucial to investment decisions. Take a company's annual financial report as an example. This document contains rich financial data and business analysis, and analysts need to extract and analyze key financial indicators from it.

[0145] Reference Figure 4 As shown, an embodiment of the present invention proposes a system for processing unstructured data in the field of intelligent document processing. The system realizes efficient processing of unstructured and structured data through a carefully designed process and architecture. Figure 4 The principles of the system are described in detail, including: the various components and their interactions, from document recognition and extraction to data storage and recall.

[0146] First, the system processes the input PDF, DOCX, TXT and Excel files through an advanced document recognition and extraction module. This process is clearly demonstrated in the attached figure, where unstructured data and structured data are fed into different processing flows. For unstructured data, the system performs full-text cleaning and chunking, then generates summaries using a large language model (LLM), and stores the processed data in ES / Milvus for subsequent text chunk recall.

[0147] For structured data, the system uses text parsing and classification based on XML attributes, generates corresponding SQL statements through LLM, and then stores the data in the MySQL database. This process ensures the structured storage of data and provides support for complex queries.

[0148] The generation of the document tree is another key point in the system design. By building a hierarchical structure of the document, the system can accurately recall the target chapter. This function is shown in detail in the attached figure, which proves the system's ability to handle complex document structures.

[0149] The three recall mechanisms designed in the system - recalling text blocks through Milvus, accurately recalling target chapters through the document tree, and recalling accurate answers through MySQL, have significantly improved the system's efficiency and accuracy in data recall.

[0150] The embodiment of the present invention proposes a processing system for unstructured data in the field of intelligent document processing, which can not only effectively process and analyze a large amount of unstructured and structured document data, but also performs well in accuracy, recall rate and processing speed. These characteristics make the system very suitable for application scenarios that require efficient and accurate document processing, such as automated report generation, content analysis and knowledge management.

[0151] Taking document processing in the financial field as an example, combined with Figure 4 For detailed explanation:

[0152] In the field of financial analysis, accurately and quickly extracting key information from financial reports is crucial to investment decisions. Take a company's annual financial report as an example. This document contains rich financial data and business analysis, and analysts need to extract and analyze key financial indicators from it.

[0153] 1 Document identification and extraction:

[0154] Input: A PDF file of the annual financial report of a public company.

[0155] Processing: The system first identifies the file format through the document recognition and extraction module, and then extracts the document content to distinguish between structured data (such as tables) and unstructured data (such as text paragraphs).

[0156] 2. Data classification and processing:

[0157] For unstructured data: the system performs full-text cleaning to remove irrelevant formats and noise information, and then performs block processing to divide the document content into multiple text blocks.

[0158] For structured data: the system uses XML attribute parsing technology to parse the table data, and cleans and classifies the data according to the predefined rule base.

[0159] 3 Intelligent parsing and summary generation:

[0160] The system uses a large language model (LLM) to deeply parse unstructured data text blocks, generate a summary of each text block, and capture key information.

[0161] 4 Document tree construction:

[0162] The system constructs a document tree based on finite state machines and XML attribute parsing technology, reflecting the chapter structure and content hierarchy of the document.

[0163] 5Data storage and management:

[0164] Unstructured data: The processed text blocks are converted into vector representations and stored in the Milvus vector database.

[0165] Structured data: After being cleaned and parsed by the rule base, it is stored in the MySQL database to form a structured data table.

[0166] 6Precise recall and query:

[0167] User query requirements: Analysts need to extract the company's key financial indicators such as net profit and operating income growth rate.

[0168] System response: Based on the requirements, the system conducts vector search in the Milvus vector database and recalls text blocks containing relevant indicators; locates chapters such as "Financial Summary" and "Key Financial Indicators" by traversing the document tree; and executes SQL statements in the MySQL database to directly recall accurate answers.

[0169] 7Query and result display:

[0170] The system integrates the recalled data and presents it to analysts through a visual interface, including the text block content recalled from the Milvus database, a visual display of the document tree, and accurate financial data queried from the MySQL database.

[0171] The results include: the company's net profit was 520 million yuan, an increase of 1.5% over last year; the operating income growth rate was 1.6%, showing a steady growth in the company's business.

[0172] Through the above embodiments, the application of the present invention in the field of financial analysis is demonstrated, which greatly improves the efficiency and quality of extracting key information from unstructured financial report documents and provides a powerful decision support tool for financial analysts.

[0173] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0174] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing unstructured data in the field of intelligent document processing, characterized in that: The following steps are involved: Perform file format recognition and content extraction on the documents to be processed; Generate SQL statements from Excel format files through data cleaning and LLM, and store structured data in MySQL data; Divide the document contents extracted from files in formats other than Excel into unstructured and structured data; Full-text cleaning and block processing of unstructured data in the document content; data cleaning and rule base analysis of structured data in the document content; Use a large language model to deeply analyze the extracted document content and generate summaries; Construct a document tree based on the extracted document content based on finite state machine and XML attribute parsing; The unstructured data in the document content is converted into vector representation after full-text cleaning and block processing, and stored in the Milvus vector database; the structured data in the document content is cleaned and parsed by the rule base, and then stored in the MySQL database.

2. The method for processing unstructured data in the field of intelligent document processing according to claim 1, characterized in that: The following steps are also included: According to the needs of user queries, vector searches are performed in the Milvus vector database to recall text blocks, and / or target chapters are recalled by traversing the document tree, and / or SQL statements are executed through the MySQL database to recall accurate answers.

3. The method for processing unstructured data in the field of intelligent document processing according to claim 2, characterized in that: The following steps are also included: Display data recalled from the Milvus vector database and / or MySQL database, and / or provide a visualization interface of the document tree.

4. The method for processing unstructured data in the field of intelligent document processing according to claim 2, characterized in that: According to the user's query requirements, the Milvus vector database is used to search and recall text blocks, including: Convert user query requirements into vectors and perform fine-tuning through vector normalization, feature selection and weighting, and regularization processing; A similarity search is performed in the Milvus vector database to recall text blocks with higher similarity; the similarity is calculated using cosine similarity, Euclidean distance and inner product calculation methods.

5. The method for processing unstructured data in the field of intelligent document processing according to claim 1, characterized in that: Extract content from the document to be processed, including: ResNet-50 and improved U-Net architecture are used to detect the layout area and extract text, titles, tables and pictures; the improved U-Net architecture introduces residual connections based on the U-Net architecture; The CTC loss function is used in combination with the GRU network to capture the long-term dependencies in the document and realize character recognition of the text; Use the pre-trained graph convolutional network (GCN) to identify the row and column information in the table and extract the table data; A multi-task learning framework combining Faster R-CNN and OCR technology is used to identify and extract text information in the seal area.

6. The method for processing unstructured data in the field of intelligent document processing according to claim 1, characterized in that: Full text cleaning of unstructured data in the document content, including: The BERT model is used to understand the context of the unstructured data in the document content. The bidirectional encoder captures the long-distance dependencies in the text and automatically identifies and removes noise words and irrelevant information. Combined with rule-based regular expressions, it cleans up known irrelevant elements including headers, footers, and page separators.

7. The method for processing unstructured data in the field of intelligent document processing according to claim 1, characterized in that: The unstructured data in the document content is processed in blocks, including: An improved Transformer architecture is used to effectively capture the dependencies between the internal parts of the document and perform block processing on the document; the improved Transformer architecture introduces a self-attention mechanism.

8. A system for processing unstructured data in the field of intelligent document processing, characterized in that: include: Document recognition and extraction module, used to identify the file format and extract the content of the document to be processed; Generate SQL statements from Excel format files through data cleaning and LLM, and store structured data in MySQL data; The data classification and processing module is used to classify the document contents extracted from files in formats other than Excel into unstructured and structured data; perform full-text cleaning and block processing on the unstructured data in the document contents; perform data cleaning and rule base analysis on the structured data in the document contents; Intelligent parsing and summary generation module, which uses a large language model to deeply parse the extracted document content and generate summaries; Document tree construction module, used to construct document tree based on finite state machine and XML attribute parsing of extracted document content; The data storage and management module is used to convert the unstructured data in the document content into vector representation after full-text cleaning and block processing, and store it in the Milvus vector database; the structured data in the document content is stored in the MySQL database after data cleaning and rule base parsing.

9. The unstructured data processing system in the field of intelligent document processing according to claim 8, characterized in that: Also includes: The precise recall and query module is used to perform vector search and recall text blocks in the Milvus vector database according to user query requirements, and / or recall target chapters through document tree traversal, and / or execute SQL statements through the MySQL database to recall accurate answers.

10. The unstructured data processing system in the field of intelligent document processing according to claim 9, characterized in that: Also includes: The query and result display module is used to display the data recalled from the Milvus vector database and / or the MySQL database, and / or provide a visualization interface of the document tree.

Images