Document processing methods, systems, electronic devices and storage media

Abstract

This application proposes a document processing method, system, electronic device, and storage medium. The document processing method includes: acquiring a document to be processed; classifying the document to be processed into different file types and generating document classification results; based on the document classification results, calling the corresponding layout type region analysis model to analyze the document to be processed and generating region results for different layout types; parsing the region results for different layout types to obtain region-level structured data; obtaining resource data when there are regions in the region results for different layout types whose content cannot be fully expressed by text; extracting document metadata from the document to be processed and the region-level structured data to obtain document-level structured data; and generating a structured dataset based on the document to be processed, the region-level structured data, the resource data, and the document-level structured data.

Description

Technical Field

[0001] This application relates to the technical field of document processing, and more particularly to a document processing method, system, electronic device, and storage medium. Background Technology

[0002] In the current field of intelligent document processing and perception, with the popularization of digital office work, various document formats such as PDF, Word, presentations, images, and Markdown are widely used. Effective structuring of these multi-source heterogeneous documents is a core prerequisite for achieving efficient document management, accurate information extraction, and supporting subsequent applications such as large-scale model training and intelligent agent interaction.

[0003] Existing document processing methods lack differentiated design for different document types in the overall processing flow, and often adopt a uniform processing mode to deal with all types of documents, resulting in limitations in the accuracy and completeness of structured representation. Summary of the Invention

[0004] This application provides a document processing method, system, electronic device, and storage medium to solve the problems existing in related technologies. The technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a document processing method, including:

[0006] Get the document to be processed;

[0007] The document to be processed is classified into different file types, and the document classification results are generated.

[0008] Based on the document classification results, the corresponding layout type area analysis model is called to analyze the document to be processed and generate area results for different layout types.

[0009] The regional results of different layout types are analyzed to obtain regional-level structured data;

[0010] In cases where there are areas in the results of different page layout types whose content cannot be fully expressed in text, resource data is obtained.

[0011] Document metadata is extracted from the documents and region-level structured data to be processed, resulting in document-level structured data;

[0012] A structured dataset is generated based on the documents to be processed, regional structured data, resource data, and document-level structured data.

[0013] In one embodiment of this application, the document to be processed is classified into different file types, and the document classification result is generated as follows:

[0014] Based on the document to be processed, determine the file extension, file header identifier, and metadata keywords;

[0015] A multi-dimensional feature vector is generated by encoding the file extension, file header identifier, and metadata keywords as a whole using the first specified model.

[0016] The second specified model is used to analyze the homepage elements of the document to be processed and output multidimensional local visual features.

[0017] Based on the document to be processed, identify several high-frequency keywords;

[0018] Multiple high-frequency keywords are encoded using a third-specified model to generate multi-dimensional shallow feature vectors.

[0019] Multidimensional feature vectors, multidimensional local visual features, and multidimensional content shallow feature vectors are concatenated to obtain a multimodal feature matrix;

[0020] Generate a specified feature matrix based on the multimodal feature matrix and the shared projection matrix;

[0021] A multidimensional context-aware feature vector is obtained by calculating the specified feature matrix using the scaled dot product attention algorithm.

[0022] Determine the fused feature vector based on the multi-dimensional context-aware feature vector;

[0023] The fused feature vectors are input into the fully connected layer of the fourth specified model to generate the output probabilities of each category;

[0024] Output the type with the highest probability among all categories to obtain the document classification result.

[0025] In one embodiment of this application, based on the document classification results, the corresponding layout type region analysis model is invoked to analyze the document to be processed, generating region results for different layout types, including:

[0026] Based on the document classification results, the first detection model is determined;

[0027] The weights and sensitivity of the first detection model in detecting each target region are optimized to obtain the second detection model;

[0028] The second detection model is used to detect each target region of the document to be processed, and the coordinates, type and confidence of each target region are generated.

[0029] Based on the type of each target region, its coordinates, and its confidence level, different layout types of region results are generated.

[0030] In one embodiment of this application, target detection is performed on the document to be processed according to the second detection model, and the coordinates of the target region and the confidence level of the target region are generated, including:

[0031] The second detection model is used to perform target detection on the document to be processed, and the first coordinates of the target region and the first confidence score of each target region are generated.

[0032] Based on the first confidence level corresponding to each target region, determine the average confidence level of the target region;

[0033] When the average confidence level of the target region is lower than a preset threshold, the original image of the low-confidence target region and its precise annotation information, as well as the low-confidence samples detected by the second detection model in the past under the same type of document, are obtained.

[0034] The original image of the low-confidence target region, along with the precise annotation information and low-confidence samples detected by the second detection model in the past under the same type of document, are used as training data to train the weight parameters and feature matching logic of the second detection model, thereby generating the third detection model.

[0035] The third detection model is used to perform target detection on the document to be processed, and the coordinates and confidence scores of the target regions are generated.

[0036] In one embodiment of this application, the region results of different layout types are parsed to obtain region-level structured data, including:

[0037] The optical character recognition engine identifies the regional results of different layout types and generates the first recognition result.

[0038] The Hough transform line detection algorithm was used to analyze the regional results of different layout types and obtain the starting coordinates of the horizontal table line, the starting coordinates of the vertical table line, the ending coordinates of the horizontal table line, and the ending coordinates of the vertical table line.

[0039] Determine the row spacing and column spacing of the table based on the starting coordinates of the horizontal table line, the starting coordinates of the vertical table line, the ending coordinates of the horizontal table line, and the ending coordinates of the vertical table line.

[0040] Determine the cell boundary range based on the table's row spacing and column spacing;

[0041] Merged cells are determined by identifying breakpoints in adjacent table lines;

[0042] The optical character recognition engine identifies cell boundaries and merged cells to determine text and numeric information.

[0043] Based on the region results for different layout types, determine the pixel matrix of the image region;

[0044] By analyzing the pixel matrix of the multimodal large model image and the analysis instructions, the object shape information, annotation information, and caption text information are determined.

[0045] Based on the first recognition result, text information, numerical information, object shape information, annotation information, and image caption text information, regional structured data is obtained.

[0046] In one embodiment of this application, document metadata is extracted from the document to be processed and the region-level structured data to obtain document-level structured data, including:

[0047] Read the document metadata from the attributes of the document to be processed to obtain the first metadata data;

[0048] Under the condition that the first meta-information data satisfies the integrity constraint and the relevance constraint, the first structured data is obtained;

[0049] Generate the input dataset based on regional structured data;

[0050] The input dataset and specified prompt words are input into the fourth detection model to obtain the second structured data;

[0051] Generate document-level structured data based on the first and second structured data.

[0052] Secondly, embodiments of this application provide a document processing system, including:

[0053] The first acquisition module is used to acquire the document to be processed.

[0054] The first generation module is used to classify the documents to be processed into different file types and generate document classification results.

[0055] The second generation module is used to call the corresponding layout type area analysis model to analyze the document to be processed based on the document classification results, and generate area results for different layout types.

[0056] The first module is used to parse the regional results of different layout types and obtain regional structured data;

[0057] The second module is used to obtain resource data when there are areas in the area results of different page types whose content cannot be fully expressed by text.

[0058] The third module is used to extract document metadata from the documents to be processed and the regional structured data to obtain document-level structured data.

[0059] The third generation module is used to generate a structured dataset based on the documents to be processed, regional structured data, resource data, and document-level structured data.

[0060] In one embodiment of this application, the first unit is used to determine the file suffix, file header identifier, and metadata keywords based on the document to be processed;

[0061] The second unit is used to encode the file extension, file header identifier and metadata keywords as a whole through the first specified model to generate a multi-dimensional feature vector.

[0062] The second specified model is used to analyze the homepage elements of the document to be processed and output multidimensional local visual features.

[0063] The third unit is used to identify multiple high-frequency keywords based on the document to be processed;

[0064] The fourth unit is used to encode multiple high-frequency keywords through the third specified model to generate multi-dimensional shallow feature vectors of content.

[0065] The fifth unit is used to concatenate multidimensional feature vectors, multidimensional local visual features, and multidimensional content shallow feature vectors to obtain a multimodal feature matrix.

[0066] The sixth unit is used to generate a specified feature matrix based on the multimodal feature matrix and the shared projection matrix;

[0067] The seventh unit is used to calculate a multidimensional context-aware feature vector by using the scaled dot product attention algorithm on a specified feature matrix.

[0068] The eighth unit is used to determine the fused feature vector based on the multi-dimensional context-aware feature vector;

[0069] The ninth unit is used to input the fused feature vector into the fully connected layer of the fourth specified model and generate the output probabilities of each class.

[0070] Unit 10 is used to output the type with the highest probability among all categories, thus obtaining the document classification result.

[0071] Thirdly, embodiments of this application provide an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the above-described document processing method.

[0072] Fourthly, embodiments of this application provide a computer-readable storage medium that stores computer instructions, wherein when the computer instructions are executed on a computer, the methods in any of the above-described embodiments are performed.

[0073] The advantages or beneficial effects of the above technical solutions include at least the following:

[0074] In this embodiment, the document processing method includes: acquiring the document to be processed; classifying the document to be processed into different file types to generate document classification results; based on the document classification results, calling the corresponding layout type region analysis model to analyze the document to be processed and generating region results for different layout types; parsing the region results for different layout types to obtain region-level structured data; obtaining resource data when there are regions in the region results for different layout types whose content cannot be fully expressed by text; extracting document metadata from the document to be processed and the region-level structured data to obtain document-level structured data; and generating a structured dataset based on the document to be processed, the region-level structured data, the resource data, and the document-level structured data. This document processing method decouples the strong binding relationship between document type and processing flow, allowing for more flexible addition of processing algorithms related to special data types. When a new document type appears, only a corresponding processing model needs to be developed for it, without requiring large-scale adjustments to the overall system, greatly improving the system's scalability and adaptability. The targeted region parsing method improves the accuracy of content extraction for different types of regions. Multi-layered structured data and structured datasets are better suited to subsequent data utilization needs, providing a clear and standardized data foundation for various data processing tasks and improving the efficiency and accuracy of data processing. This effectively solves the technical problem of a lack of differentiated design for different document types in the overall processing workflow, often resulting in a uniform processing model for all document types, leading to limitations in the accuracy and completeness of structured representation.

[0075] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of this application will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0076] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0077] Figure 1 This is a flowchart of a document processing method according to an embodiment of this application.

[0078] Figure 2 This is a block diagram of an electronic device according to an embodiment of the present application. Detailed Implementation

[0079] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0080] In the current field of intelligent document processing and perception, with the popularization of digital office work, various document formats such as PDF, Word, presentations, images, and Markdown are widely used. Effective structuring of these multi-source heterogeneous documents is a core prerequisite for achieving efficient document management, accurate information extraction, and supporting subsequent applications such as large-scale model training and intelligent agent interaction.

[0081] Existing technologies have several shortcomings when processing multi-source heterogeneous documents. First, document classification methods are relatively simplistic, relying heavily on file extensions and header information. For documents with special formats or those that have undergone conversion, classification accuracy is insufficient, directly impacting the targetedness and effectiveness of subsequent processing. Second, the overall processing workflow lacks differentiated design for different document types, often employing a uniform processing model for all document types, leading to limitations in the accuracy and completeness of structured representation. For example, in the region identification and parsing stage, due to insufficient consideration of the characteristics of different document types, misjudgments of regions such as titles, tables, images, and text may occur, or inaccurate parsing of region content may result. Furthermore, the design of structured representation lacks systematicity, and data storage methods are inadequate, making it difficult to organically link document metadata and region information. This hinders the structured supply of training data for large models and the flexible scheduling of document resources by agents. Additionally, existing systems have poor scalability; when faced with new and special data types of documents, large-scale adjustments to the overall architecture are often required, making it difficult to quickly adapt to the needs of large-scale model iteration upgrades and agent functional expansion.

[0082] Figure 1 A flowchart illustrating a document processing method according to an embodiment of this application is shown. Figures 1-2 As shown, a document processing method may include:

[0083] S110: Obtain the document to be processed;

[0084] S120: Classify the documents to be processed into different file types and generate document classification results;

[0085] S130: Based on the document classification results, call the corresponding layout type area analysis model to analyze the document to be processed and generate area results for different layout types;

[0086] S140: Analyze the regional results of different layout types to obtain regional-level structured data;

[0087] S150: In cases where there are areas in the results of different page layout types whose content cannot be fully expressed by text, resource data is obtained;

[0088] S160: Extract document metadata from the documents and region-level structured data to be processed to obtain document-level structured data;

[0089] S170: Generate a structured dataset based on the documents to be processed, regional structured data, resource data, and document-level structured data.

[0090] In this embodiment, the document processing method decouples the strong binding relationship between document type and processing flow, enabling more flexible addition of processing algorithms related to special data types. When a new document type appears, only a corresponding processing model needs to be developed, without requiring large-scale adjustments to the overall system, greatly improving the system's scalability and adaptability. Targeted region parsing improves the accuracy of content extraction from different types of regions. Multi-layered structured data and structured datasets better suit subsequent data utilization needs, providing a clear and standardized data foundation for various data processing tasks, improving the efficiency and accuracy of data processing. It effectively solves the technical problem of lacking differentiated design for different document types in the overall processing flow, often using a uniform processing model to deal with all types of documents, leading to limitations in the accuracy and completeness of structured representation.

[0091] In step S110, the document to be processed is obtained;

[0092] In the embodiments of this application, the document to be processed can be of different types such as PDF, Word, presentation, image, Markdown, etc., and may also include text information, image information, numerical information or identifier information, etc. The document to be processed is stored in the original database, and the document to be processed is indexed as the original document. The corresponding symbols are configured to represent the meaning of each data, and a multi-level reference relationship is configured with subsequent document-level structured data and subsequent resource data symbols to facilitate reference and association by other subsequent data.

[0093] In step S120, the documents to be processed are classified into different file types to generate document classification results.

[0094] In this embodiment, the documents to be processed are classified according to different document types such as PDF, Word, presentation, image, and Markdown. The classification method can combine the judgment of file extensions and file header information with visual model recognition. A document type recognition rule base is established and updated regularly, while a specialized visual classification model is trained to assist in classification by analyzing the visual features of the documents (such as page layout, font style, element arrangement, etc.). For documents with unclear or ambiguous information, the visual model recognition results are given priority.

[0095] The input documents are classified using a combination of file extension and header information-based judgment and visual model recognition. It relies on a document type recognition rule base and integrates a pre-trained visual classification model to assist in classification by analyzing the document's visual features.

[0096] After classifying the documents to be processed by file type, the document to be classified is determined to be one of the following document categories: PDF, Word, presentation, image, Markdown, etc., and the document category is determined as the document classification result.

[0097] In step S130, based on the document classification results, the corresponding layout type region analysis model is called to analyze the document to be processed and generate region results for different layout types.

[0098] In this embodiment, based on the document classification results, different document categories invoke the corresponding layout type region analysis model. The layout type region analysis model includes functions such as visual model recognition and direct program parsing. The goal of analyzing the document to be processed by invoking the corresponding layout type region analysis model based on the document classification results is to accurately distinguish different types of regions in the document, such as titles, tables, images, and text. By invoking the corresponding layout type region analysis model to perform layout analysis on the document, different types of regions are distinguished.

[0099] In this embodiment, the layout type region analysis model can be a large AI model, configured with rules to accurately distinguish different types of regions in the document, such as titles, tables, images, and text. The layout type region analysis model is obtained through training. Different layout type region analysis models are generated for different document types. The corresponding model can be called according to the document type result. The called layout type region analysis model analyzes the document to be processed and can generate region results of different layout types in the document.

[0100] In step S140, the regional results of different layout types are parsed to obtain regional-level structured data.

[0101] In this embodiment, by parsing the different regions in the results of different layout types, which are respectively titles, tables, images, and text, and the corresponding positions, such as position coordinates, the results of different layout types also include corresponding positions. After parsing the results of different layout types, regional structured data containing region identifiers, region types, positioning information, and parsed content is constructed, regional structured data is stored in a regional structured database, and references to subsequent document-level structured data are added.

[0102] This involves indexing the obtained regional-level structured data, setting corresponding symbols to represent the meaning of each data point, and configuring multi-level reference relationships with subsequent document-level structured data and subsequent resource data symbols.

[0103] For the documents being processed, whether they are Word, PDF, or PPT (in terms of format), or academic papers, patents, notices, etc. (in terms of content), they are usually composed of multiple regions. Taking a PDF academic paper as an example, each page can be divided into different types c, such as title, paragraphs, illustrations, tables, etc. After detecting each region, a region identifier i is assigned to each region as a unique identifier. Each region is usually rectangular and has positioning information (x, y, w, h). Furthermore, each region undergoes content parsing to obtain the text representation t of the content. For plain text regions, such as paragraphs, the region-level structured data is {i, c, (x, y, w, h), t}.

[0104] Specifically, for the results of different page layouts, corresponding parsing methods are used for content parsing. For example, OCR is used to process text and number content, table structure and content recognition, and image text descriptions can be processed using AI large models with pre-defined rules.

[0105] In step S150, if there are areas in the area results of different layout types whose content cannot be fully expressed by text, resource data is obtained;

[0106] In the embodiments of this application, regions whose content cannot be fully expressed in text are identified as resource data. These regions are stored in a resource database as resource data, and bidirectional references are added between the region-level structured data and the resource data. The obtained resource data is indexed, with corresponding symbols set to represent the meaning of each data point, and bidirectional references are established between these symbols and the symbols in the region-level structured data.

[0107] For regions whose content cannot be fully expressed in text, such as images, the image is treated as resource data and stored separately in a resource database. The image is then assigned an identifier z, and the region is represented as {i, c, (x, y, w, h), t, z}. A relation i is added to the image resource in the resource database as a reference to the region.

[0108] In step S160, document metadata is extracted from the document to be processed and the region-level structured data to obtain document-level structured data.

[0109] In this embodiment, metadata is extracted from the document to be processed. The extracted metadata includes, but is not limited to, the main title, author, keywords, abstract, and date. This data can come directly from the metadata carried by the document itself, or it can be integrated and extracted from the document's regional structured data using algorithms such as large-scale models. After extracting this metadata, document-level structured data containing document identifiers, document metadata, and regional identifiers organized by page numbers is constructed and stored in a document-level structured database. Furthermore, the document-level structured data is linked to the document to be processed.

[0110] The obtained document-level data is indexed, and corresponding symbols are set to represent the meaning of each data point.

[0111] In step S170, a structured dataset is generated based on the document to be processed, region-level structured data, resource data, and document-level structured data. In this embodiment, through the established multi-level reference relationships, forward retrieval from document location to specific region and from region association to corresponding resource can be achieved, as well as reverse tracing from resource back to its region and from region association to its document.

[0112] This structured dataset has established multi-level referencing relationships among the documents to be processed, region-level structured data, resource data, and document-level structured data.

[0113] Starting with a document, referencing it allows access to the original document. Alternatively, all areas can be organized by page number using area identifiers, allowing you to locate specific area data. If the area data contains non-text resources, those resources are stored in a resource database and assigned a resource identifier, which can also be used to find the resource.

[0114] If starting with resource information (such as images), these reference relationships can also be used to find regional-level structured data or higher-level document-level structured data.

[0115] This multi-level referencing relationship is suitable for retrieval scenarios. In the current technological ecosystem, LLM RAG can be used to assist in finding the context more efficiently. The simplest way to establish this referencing relationship is to generate globally unique identifiers.

[0116] LLM RAG Application Examples

[0117] 1. Implementation Objectives

[0118] In the enterprise knowledge base, by searching for "installation diagram of a certain type of equipment", the region where the image is located can be accurately located with the help of regional structured data, and the complete document and corresponding text information can be obtained simultaneously to support LLM in understanding "the relationship between the image and the installation steps".

[0119] 2. Key Steps

[0120] (1) Search triggers and demand breakdown

[0121] A user inputs the query "Find the installation diagram for model XX equipment and explain the corresponding installation steps" into the LLM. The LLM interprets the requirement as "Search keywords: model XX, installation diagram; related requirements: the document text corresponding to the diagram", thus triggering the search command.

[0122] (2) Regional-level structured data matching and positioning

[0123] The system invokes the "Regional-level Structured Database," matches regional-level structured data based on keywords, and quickly locates the regional-level structured data of the target region by using "Regional Type = Installation Diagram" and "Associated Document Meta Information Contains XX Model."

[0124] (3) Cross-level representation and bidirectional traceability

[0125] By leveraging the bidirectional referencing relationship between regional structured data and resource data, the corresponding installation schematic image files can be directly retrieved from the "resource database" and fed back to the LLM for visual association.

[0126] By using the document-level structured data index associated with the regional-level structured data, the corresponding document-level structured data is retrieved from the "document-level structured database," including the document name (such as "Installation Guide for XX Model Equipment V2.1") and the original document storage path. At the same time, based on the mapping relationship between "page number - regional-level structured data" in the document-level structured data, the text area data of the page number where the image is located is extracted to obtain the corresponding text content: "Step 3: Align the equipment bracket with the pre-embedded hole in the wall, adjust the horizontal angle according to the corresponding installation diagram, and then fix the bolts."

[0127] (4) LLM output results

[0128] LLM combines image resource data with the text content to generate an accurate answer: "The installation diagram of the XX model equipment is located on the corresponding page of the 'XX Model Equipment Installation Guide V2.1'. The corresponding installation steps are: align the equipment bracket with the pre-embedded hole in the wall, adjust the horizontal angle according to the diagram, and then fix the bolts." It also includes an image preview and a document download link, and supports direct jump to the page number of the image in the document.

[0129] (vi) Image retrieval examples

[0130] 1. Core Objectives

[0131] Users upload "product performance comparison charts" from paper reports, and with the help of structured datasets, the entire chain of "image → region → document" can be traced to obtain the original electronic document corresponding to the chart and the accompanying analysis conclusions.

[0132] 2. Key Steps

[0133] (1) Retrieval trigger and visual feature extraction

[0134] When a user uploads a "product performance comparison chart", the system initiates "image source tracing retrieval", automatically extracts the visual features of the image, and triggers a matching process with the "resource database".

[0135] (2) Correlation between resource-level and regional-level representations

[0136] The visual features of the uploaded image are compared with the chart resources stored in the "Resource Database". The resource data of the target chart is located by high feature similarity. Then, the corresponding regional representation is retrieved through the bidirectional reference relationship of "resource data → regional structured data".

[0137] (3) Tracing from regional-level structured data to document-level structured data

[0138] By using document-level structured data identifiers associated with regional-level structured data, the corresponding document-level structured data is retrieved, including, for example, the document name (such as "2024Q1 Product Competitiveness Analysis Report"), document author, update time, and original document storage path; at the same time, based on the text area index associated with regional-level structured data, the corresponding text content is extracted: "As can be seen from the corresponding product performance comparison chart, product XX leads the competitors in battery life, while there is a certain difference in cost. It is recommended to optimize the supply chain to balance costs in the future."

[0139] (4) Output of search results

[0140] Returning to the user: "Matching chart information (page number of the document, region type) + full document download link + correlation analysis conclusion (text content)", supporting direct jump to the page number of the chart in the document for easy and quick reference to the context.

[0141] The document processing method in this embodiment decouples the strong binding relationship between document type and processing flow, and can more flexibly add processing algorithms related to special data types. When a new document type appears, only a corresponding processing model needs to be developed for it, without the need for large-scale adjustments to the overall system, which greatly improves the scalability and adaptability of the system.

[0142] By employing targeted regional parsing methods, the accuracy of content extraction from different types of regions has been improved. For example, the accurate identification of table structures and content, and the reasonableness of image and text descriptions, have provided high-quality raw data for regional structured data.

[0143] Multi-level structured datasets and databases are better suited to subsequent data utilization needs, providing a clear and standardized data foundation for various data processing tasks and improving the efficiency and accuracy of data processing.

[0144] In one embodiment of this application, the document to be processed is classified into different file types, and the document classification result is generated as follows:

[0145] Based on the document to be processed, determine the file extension, file header identifier, and metadata keywords;

[0146] A multi-dimensional feature vector is generated by encoding the file extension, file header identifier, and metadata keywords as a whole using the first specified model.

[0147] The second specified model is used to analyze the homepage elements of the document to be processed and output multidimensional local visual features.

[0148] Based on the document to be processed, identify several high-frequency keywords;

[0149] Multiple high-frequency keywords are encoded using a third-specified model to generate multi-dimensional shallow feature vectors.

[0150] Multidimensional feature vectors, multidimensional local visual features, and multidimensional content shallow feature vectors are concatenated to obtain a multimodal feature matrix;

[0151] Generate a specified feature matrix based on the multimodal feature matrix and the shared projection matrix;

[0152] A multidimensional context-aware feature vector is obtained by calculating the specified feature matrix using the scaled dot product attention algorithm.

[0153] Determine the fused feature vector based on the multi-dimensional context-aware feature vector;

[0154] The fused feature vectors are input into the fully connected layer of the fourth specified model to generate the output probabilities of each category;

[0155] Output the type with the highest probability among all categories to obtain the document classification result.

[0156] In the embodiments of this application, file suffixes are collected by extracting multi-dimensional features and extracting file meta-features. File header identifier and metadata keywords The above file extensions are processed using a first specified model (e.g., a Qwen-Embedding model that supports MRL). File header identifier and metadata keywords Perform overall encoding to generate multi-dimensional feature vectors (e.g., 768-dimensional feature vectors). For visual feature extraction, the document's homepage elements are analyzed using a second specified model (e.g., the ViT-Base model), outputting multi-dimensional local visual features (e.g., 768-dimensional local visual features). For shallow feature extraction, based on the number of pages in the document to be processed, a sample page set is determined (≤5 pages: all selected; >5 pages: first page, last page, and remaining pages are sampled evenly, totaling 3 pages). The top 20 high-frequency keywords are extracted using the Tesseract OCR engine and encoded by a third-party specified model (e.g., Qwen-Embedding model) to generate a multi-dimensional shallow feature vector (e.g., a 768-dimensional shallow feature vector). ).

[0157] Dynamic weight fusion classification is used to construct a cross-modal attention calculation module, which integrates multi-dimensional feature vectors, multi-dimensional local visual features, and multi-dimensional shallow content feature vectors (768-dimensional feature vectors). 768-dimensional local visual features 768-dimensional shallow feature vectors The multimodal feature matrix is ​​formed by concatenating rows. Then based on the multimodal feature matrix and shared projection matrix Generate a specified feature matrix , , (All are) The scaled dot product attention algorithm is used (the formula is calculated as follows). The specified feature matrix is ​​used for calculation to obtain the multidimensional context-aware feature vector. That is, concatenating multi-dimensional context-aware feature vectors The fused feature vector is obtained from the row vectors of each row. Finally, the feature vectors will be fused. Input to a fully connected layer, pass through the softmax function of the fully connected layer, and output the probabilities of each class. The system outputs the type with the highest probability among all categories, thus obtaining the document classification result.

[0158] By fusing multimodal features, accurate classification of heterogeneous documents from multiple sources, such as PDFs, Word documents, and images, is achieved, solving the problem of classification accuracy in scenarios with ambiguous file information.

[0159] The document processing method in this embodiment adopts a document classification approach that combines file information with a visual model, which improves the classification accuracy of documents with special formats or after conversion, reduces classification errors caused by unclear file information, and provides a more reliable foundation for subsequent processing steps.

[0160] In one embodiment of this application, based on the document classification results, the corresponding layout type region analysis model is invoked to analyze the document to be processed, generating region results for different layout types, including:

[0161] Based on the document classification results, the first detection model is determined;

[0162] The weights and sensitivity of the first detection model in detecting each target region are optimized to obtain the second detection model;

[0163] The second detection model is used to detect each target region of the document to be processed, and the coordinates, type and confidence of each target region are generated.

[0164] Based on the type of each target region, its coordinates, and its confidence level, different layout types of region results are generated.

[0165] In the embodiments of this application, the document classification results determine the preprocessing strategy. For scanned PDFs, text edge enhancement and chart feature enhancement operations can be optimized, and the image tilt can be corrected before output. For image types (photos / screenshots), resolution and contrast can be adjusted, environmental interference such as backlight or shadows can be eliminated, and a preprocessed image can be output.

[0166] The first detection model (e.g., YOLOv8 model) is used to process the document to be processed after preprocessing the image or to process the document directly. Different types of first detection models are loaded according to the document classification results. The detection weights and sensitivities of target regions such as text, tables, and illustrations in the first detection model are optimized. The optimization method can be to train by setting a training set or to configure rules to obtain an optimized second detection model. The optimized second detection model is used to detect target regions such as text, tables, and illustrations. After the target region detection is completed, the coordinates of the target region, the type of the target region, and the confidence of the target region are output.

[0167] Based on the type of each target region, the coordinates of the target region, and the confidence level of the target region, region results for different layout types are generated.

[0168] The document processing method in this embodiment improves the accuracy of content extraction for different types of regions through targeted region parsing, such as accurate identification of table structure and content, and reasonable image text description, providing high-quality document data to be processed for structured datasets.

[0169] Based on the document classification results, the system adaptively performs layout area detection and filtering, accurately distinguishes text, tables, illustrations, and other areas, and outputs standardized results.

[0170] In one embodiment of this application, the document to be processed is subjected to target region detection according to the second detection model, and the coordinates of each target region, the type of each target region, and the confidence level corresponding to each target region are generated, including:

[0171] The second detection model is used to perform target detection on the document to be processed, and the first confidence score corresponding to each target region is generated.

[0172] Based on the first confidence level corresponding to each target region, determine the average confidence level of the target region;

[0173] When the average confidence level of the target region is lower than a preset threshold, the original image of the low-confidence target region and its precise annotation information, as well as the low-confidence samples detected by the second detection model in the past under the same type of document, are obtained.

[0174] The original image of the low-confidence target region, along with the precise annotation information and low-confidence samples detected by the second detection model in the past under the same type of document, are used as training data to train the weight parameters and feature matching logic of the second detection model, thereby generating the third detection model.

[0175] The third detection model is used to perform target detection on the document to be processed, generating the type of each target region, the coordinates of each target region, and the confidence level of each target region.

[0176] In the embodiments of this application, each document category has its corresponding second detection model (e.g., YOLOv8 model), which tracks the first confidence level of the target region output by the second detection model in real time. When the average confidence level of each target region (text, table, illustration, etc.) output by the second detection model for the document type to be processed is lower than a preset threshold (e.g., 85%), the second training process of the second detection model is automatically triggered. The training data prioritizes collecting the original images and precise annotation information of the low-confidence regions of each target region (text, table, illustration, etc.) output by the document type to be processed, supplementing the low-confidence samples detected by the second detection model in the past under the same type of document. By training, the weight parameters and feature matching logic of the second detection model are adjusted to obtain the third detection model, which enhances its accuracy in recognizing the layout region of the document type to be processed and the same type of document. After training, the third detection model replaces the detection model and is applied to the layout analysis task of the same type of document. Finally, the third detection model is used to perform target detection on the document to be processed, generating the type of each target region, the coordinates of each target region, and the confidence level of each target region.

[0177] In one embodiment of this application, the region results of different layout types are parsed to obtain region-level structured data, including:

[0178] The optical character recognition engine identifies the regional results of different layout types and generates the first recognition result.

[0179] The Hough transform line detection algorithm was used to analyze the regional results of different layout types and obtain the starting coordinates of the horizontal table line, the starting coordinates of the vertical table line, the ending coordinates of the horizontal table line, and the ending coordinates of the vertical table line.

[0180] Determine the row spacing and column spacing of the table based on the starting coordinates of the horizontal table line, the starting coordinates of the vertical table line, the ending coordinates of the horizontal table line, and the ending coordinates of the vertical table line.

[0181] Determine the cell boundary range based on the table's row spacing and column spacing;

[0182] Merged cells are determined by identifying breakpoints in adjacent table lines;

[0183] The optical character recognition engine identifies cell boundaries and merged cells to determine text and numeric information.

[0184] Based on the region results for different layout types, determine the pixel matrix of the image region;

[0185] By analyzing the pixel matrix of the multimodal large model image and the analysis instructions, the object shape information, annotation information, and caption text information are determined.

[0186] Based on the first recognition result, text information, numerical information, object shape information, annotation information, and image caption text information, regional structured data is obtained.

[0187] In the embodiments of this application, for the regional results of different layout types, corresponding parsing schemes are used to extract structured content, providing data support for the construction of regional structured data.

[0188] Text region parsing: First, initialize the optical character recognition engine (Tesseract OCR engine), load the Chinese + English bilingual package and the configuration file that optimizes the recognition parameters of printed and small fonts, so that the optical character recognition engine can have the recognition capability of more categories and higher accuracy.

[0189] The region images in the results of different layout types are subjected to grayscale conversion, Gaussian filtering for noise reduction, and binarization to improve the contrast between text and background. Finally, the optical character recognition engine reads the processed image and outputs the first recognition result.

[0190] Parse the table area: Perform edge enhancement on the table area images in the area results of different layout types, and call the Hough transform line detection algorithm to output the starting coordinates of the horizontal table line, the starting coordinates of the vertical table line, the ending coordinates of the horizontal table line, and the ending coordinates of the vertical table line.

[0191] Based on the starting coordinates of the horizontal and vertical table lines, the row spacing and column spacing of the table are determined. The cell boundary range is determined by the row spacing and column spacing. By identifying the breakpoints of adjacent table lines, merged cells are determined. Finally, the optical character recognition engine (Tesseract OCR engine) is called for each cell to extract text and numeric information.

[0192] Image region parsing: For image regions in the results of different layout types, determine the pixel matrix. Use the image region pixel matrix as visual input and construct parsing instructions (e.g., "Describe the image theme, core elements and type, extract the caption text and integrate it"). Then call the multimodal large model (e.g., Qwen-VL, GPT-4V, etc.) to simultaneously parse the image region pixel matrix and parsing instructions, and identify object shape information, annotation information and caption text information without the need for additional OCR tools.

[0193] Finally, the first recognition result, text information, numerical information, object shape information, annotation information, and image caption text information are combined to obtain regional structured data.

[0194] The file processing method in this embodiment improves the accuracy of content extraction for different types of regions through targeted region parsing, such as accurate identification of table structure and content, and reasonable image text description, providing high-quality raw data for structured representation.

[0195] In one embodiment of this application, document metadata is extracted from the document to be processed and the region-level structured data to obtain document-level structured data, including:

[0196] Read the document metadata from the attributes of the document to be processed to obtain the first metadata data;

[0197] Under the condition that the first meta-information data satisfies the integrity constraint and the relevance constraint, the first structured data is obtained;

[0198] Generate the input dataset based on regional structured data;

[0199] The input dataset and specified prompt words are input into the fourth detection model to obtain the second structured data;

[0200] Generate document-level structured data based on the first and second structured data.

[0201] In this embodiment, the document's built-in metadata detection and extraction are performed as follows: Using tools such as PyPDF2 (for PDF format) and python-docx (for DOCX format), the first metadata data, including "Title," "Author," "Creation Date," "Keywords," and "Description," are read from the document attributes. Then, it is determined whether the first metadata data meets the integrity constraints (core fields such as title and author are not empty) and the relevance constraints (content matches the document topic). If the verification passes, that is, if the first metadata data meets the integrity and relevance constraints, the first metadata data is organized into the first structured data.

[0202] Large model meta-information extraction: First, filter “title candidate paragraphs (largest font, top text area), abstract candidate paragraphs (introduction / summary text), and keyword candidate words (high-frequency professional terms in the whole text)” from the regional structured data and integrate them into the input dataset;

[0203] Then design prompts using the "Instruction + Input Data + Output Format" format (example: "Extract title (summarize the core), author (mark 'Unknown' if none), 3-5 keywords, 100-200 word abstract, key dates (mark 'Not mentioned' if none), output in JSON").

[0204] Finally, the prompt words and input dataset are input into the fourth detection model (e.g., Qwen3 model) to generate initial structured data. The initial structured data is then subjected to secondary verification (verification can be based on whether the summary covers the core content and whether the keywords match the domain). Specifically, it is determined whether the summary of the initial structured data can cover the core content and whether the keywords of the initial structured data can match the domain. If the summary of the initial structured data can cover the core content and the keywords of the initial structured data can match the domain, the initial structured data is then organized into second structured data.

[0205] Finally, the first and second structured data are merged to obtain document-level structured data.

[0206] By employing a dual-path approach of detecting metadata inherent in documents and extracting it from large models, complete and accurate document metadata is obtained, providing support for the construction of document-level structured data.

[0207] The document processing method proposed in this application solves the problem of structuring multi-source heterogeneous documents. Through layered processing and targeted analysis, it achieves efficient and accurate structured representation and clarifies the specific implementation and collaborative logic of each step.

[0208] Secondly, embodiments of this application provide a document processing system, including:

[0209] The first acquisition module is used to acquire the document to be processed.

[0210] The first generation module is used to classify the documents to be processed into different file types and generate document classification results.

[0211] The second generation module is used to call the corresponding layout type area analysis model to analyze the document to be processed based on the document classification results, and generate area results for different layout types.

[0212] The first module is used to parse the regional results of different layout types and obtain regional structured data;

[0213] The second module is used to obtain resource data when there are areas in the area results of different page types whose content cannot be fully expressed by text.

[0214] The third module is used to extract document metadata from the documents to be processed and the regional structured data to obtain document-level structured data.

[0215] The third generation module is used to generate a structured dataset based on the documents to be processed, regional structured data, resource data, and document-level structured data.

[0216] In this embodiment, the document processing system decouples the strong binding relationship between document type and processing flow, enabling more flexible addition of processing algorithms related to special data types. When a new document type appears, only a corresponding processing model needs to be developed, without requiring large-scale adjustments to the overall system, greatly improving the system's scalability and adaptability. Targeted region parsing methods improve the accuracy of content extraction from different types of regions. Multi-layered structured data and structured datasets better suit subsequent data utilization needs, providing a clear and standardized data foundation for various data processing tasks, improving the efficiency and accuracy of data processing. It effectively solves the technical problem of lacking differentiated design for different document types in the overall processing flow, often using a uniform processing mode to deal with all types of documents, resulting in limitations in the accuracy and completeness of structured representation.

[0217] In one embodiment of this application, the first unit is used to determine the file suffix, file header identifier, and metadata keywords based on the document to be processed;

[0218] The second unit is used to encode the file extension, file header identifier and metadata keywords as a whole through the first specified model to generate a multi-dimensional feature vector.

[0219] The second specified model is used to analyze the homepage elements of the document to be processed and output multidimensional local visual features.

[0220] The third unit is used to identify multiple high-frequency keywords based on the document to be processed;

[0221] The fourth unit is used to encode multiple high-frequency keywords through the third specified model to generate multi-dimensional shallow feature vectors of content.

[0222] The fifth unit is used to concatenate multidimensional feature vectors, multidimensional local visual features, and multidimensional content shallow feature vectors to obtain a multimodal feature matrix.

[0223] The sixth unit is used to generate a specified feature matrix based on the multimodal feature matrix and the shared projection matrix;

[0224] The seventh unit is used to calculate a multidimensional context-aware feature vector by using the scaled dot product attention algorithm on a specified feature matrix.

[0225] The eighth unit is used to determine the fused feature vector based on the multi-dimensional context-aware feature vector;

[0226] The ninth unit is used to input the fused feature vector into the fully connected layer of the fourth specified model and generate the output probabilities of each class.

[0227] Unit 10 is used to output the type with the highest probability among all categories, thus obtaining the document classification result.

[0228] In one embodiment of this application, based on the document classification results, the corresponding layout type region analysis model is invoked to analyze the document to be processed, generating region results for different layout types, including:

[0229] Based on the document classification results, the first detection model is determined;

[0230] The weights and sensitivity of the first detection model in detecting each target region are optimized to obtain the second detection model;

[0231] The second detection model is used to detect each target region of the document to be processed, and the coordinates, type and confidence of each target region are generated.

[0232] Based on the type of each target region, its coordinates, and its confidence level, different layout types of region results are generated.

[0233] In one embodiment of this application, target detection is performed on the document to be processed according to the second detection model, and the coordinates of the target region and the confidence level of the target region are generated, including:

[0234] The second detection model is used to perform target detection on the document to be processed, and the first coordinates of the target region and the first confidence score of each target region are generated.

[0235] Based on the first confidence level corresponding to each target region, determine the average confidence level of the target region;

[0236] When the average confidence level of the target region is lower than a preset threshold, the original image of the low-confidence target region and its precise annotation information, as well as the low-confidence samples detected by the second detection model in the past under the same type of document, are obtained.

[0237] The original image of the low-confidence target region, along with the precise annotation information and low-confidence samples detected by the second detection model in the past under the same type of document, are used as training data to train the weight parameters and feature matching logic of the second detection model, thereby generating the third detection model.

[0238] The third detection model is used to perform target detection on the document to be processed, and the coordinates and confidence scores of the target regions are generated.

[0239] In one embodiment of this application, the region results of different layout types are parsed to obtain region-level structured data, including:

[0240] The optical character recognition engine identifies the regional results of different layout types and generates the first recognition result.

[0241] The Hough transform line detection algorithm was used to analyze the regional results of different layout types and obtain the starting coordinates of the horizontal table line, the starting coordinates of the vertical table line, the ending coordinates of the horizontal table line, and the ending coordinates of the vertical table line.

[0242] Determine the row spacing and column spacing of the table based on the starting coordinates of the horizontal table line, the starting coordinates of the vertical table line, the ending coordinates of the horizontal table line, and the ending coordinates of the vertical table line.

[0243] Determine the cell boundary range based on the table's row spacing and column spacing;

[0244] Merged cells are determined by identifying breakpoints in adjacent table lines;

[0245] The optical character recognition engine identifies cell boundaries and merged cells to determine text and numeric information.

[0246] Based on the region results for different layout types, determine the pixel matrix of the image region;

[0247] By analyzing the pixel matrix of the multimodal large model image and the analysis instructions, the object shape information, annotation information, and caption text information are determined.

[0248] Based on the first recognition result, text information, numerical information, object shape information, annotation information, and image caption text information, regional structured data is obtained.

[0249] In one embodiment of this application, document metadata is extracted from the document to be processed and the region-level structured data to obtain document-level structured data, including:

[0250] Read the document metadata from the attributes of the document to be processed to obtain the first metadata data;

[0251] Under the condition that the first meta-information data satisfies the integrity constraint and the relevance constraint, the first structured data is obtained;

[0252] Generate the input dataset based on regional structured data;

[0253] The input dataset and specified prompt words are input into the fourth detection model to obtain the second structured data;

[0254] Generate document-level structured data based on the first and second structured data.

[0255] The functions of each module in each device in the embodiments of this application can be found in the corresponding descriptions in the above methods, and will not be repeated here.

[0256] Figure 2 A structural block diagram of an electronic device according to an embodiment of this application is shown. Figure 2 As shown, the electronic device includes a memory 410 and a processor 420. The memory 410 stores instructions that can be executed on the processor 420. When the processor 420 executes the instructions, it implements the document processing method in the above embodiments. The number of memories 410 and processors 420 can be one or more. This electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0257] The electronic device may also include a communication interface 430 for communicating with external devices and exchanging data. The devices are interconnected using different buses and can be mounted on a common motherboard or otherwise as needed. The processor 420 can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). The bus can be divided into address buses, data buses, control buses, etc. For ease of illustration, Figure 2 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0258] Optionally, in a specific implementation, if the memory 410, processor 420 and communication interface 430 are integrated on a single chip, the memory 410, processor 420 and communication interface 430 can communicate with each other through an internal interface.

[0259] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0260] This application provides a computer-readable storage medium (such as the memory 410 described above) that stores computer instructions, which, when executed by a processor, implement the method provided in this application.

[0261] Optionally, memory 410 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device, etc. Furthermore, memory 410 may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 410 may optionally include memory remotely located relative to processor 420, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0262] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0263] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0264] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more (two or more) executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0265] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0266] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0267] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0268] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A document processing method, characterized in that, include: Get the document to be processed; The documents to be processed are classified into different file types to generate document classification results; Based on the document classification results, the corresponding layout type region analysis model is called to analyze the document to be processed, and region results for different layout types are generated; The regional results of the different layout types are parsed to obtain regional-level structured data; In cases where there are regions in the area results of the different layout types whose content cannot be fully expressed by text, resource data is obtained; Document metadata is extracted from the document to be processed and the region-level structured data to obtain document-level structured data; A structured dataset is generated based on the document to be processed, the region-level structured data, the resource data, and the document-level structured data; The step of analyzing the document to be processed by calling the corresponding layout type region analysis model based on the document classification result and generating region results for different layout types includes: Based on the document classification results, a first detection model is determined; The weights and sensitivity of the first detection model in detecting each target region are optimized to obtain the second detection model; The document to be processed is subjected to target region detection according to the second detection model, and the coordinates of each target region, the type of each target region and the confidence level of each target region are generated. Based on the type of each target region, the coordinates of the target region, and the confidence level of the target region, generate region results for different layout types; The step of detecting target regions in the document to be processed according to the second detection model, and generating the coordinates of each target region, the type of each target region, and the confidence level corresponding to each target region includes: The document to be processed is subjected to target detection according to the second detection model, and a first confidence score is generated for each target region. Based on the first confidence level corresponding to each target region, determine the average confidence level of the target region; If the average confidence level of the target region is lower than a preset threshold, the original image of the low-confidence target region and its precise annotation information, as well as low-confidence samples detected historically by the second detection model under the same type of document, are obtained. The original image of the low-confidence target region, along with its precise annotation information and low-confidence samples detected historically by the second detection model under the same type of document, are used as training data to train the weight parameters and feature matching logic of the second detection model, thereby generating a third detection model. The document to be processed is subjected to target detection based on the third detection model, and the type, coordinates and confidence scores of each target region are generated.

2. The method according to claim 1, characterized in that, The step of classifying the document to be processed into different file types and generating document classification results includes: Based on the document to be processed, determine the file extension, file header identifier, and metadata keywords; A multidimensional feature vector is generated by encoding the file suffix, the file header identifier, and the metadata keywords as a whole using a first specified model. The first page elements of the document to be processed are analyzed using a second specified model, and multidimensional local visual features are output. Based on the document to be processed, identify several high-frequency keywords; The multiple high-frequency keywords are encoded using a third specified model to generate a multi-dimensional content shallow feature vector; The multidimensional feature vector, the multidimensional local visual features, and the multidimensional content shallow feature vector are concatenated to obtain a multimodal feature matrix; Based on the multimodal feature matrix and the shared projection matrix, a specified feature matrix is ​​generated; The specified feature matrix is ​​calculated using a scaled dot product attention algorithm to obtain a multidimensional context-aware feature vector; Based on the multi-dimensional context-aware feature vector, determine the fused feature vector; The fused feature vector is input into the fully connected layer of the fourth specified model to generate the output probabilities of each category; Output the type with the highest probability among the various categories to obtain the document classification result.

3. The method according to claim 1, characterized in that, The parsing of the regional results for the different layout types to obtain regional-level structured data includes: The optical character recognition engine identifies the regional results of the different layout types to generate a first recognition result; The Hough transform line detection algorithm is used to analyze the regional results of the different layout types to obtain the starting coordinates of the horizontal table line, the starting coordinates of the vertical table line, the ending coordinates of the horizontal table line, and the ending coordinates of the vertical table line. The row spacing and column spacing of the table are determined based on the starting coordinates of the horizontal table line, the starting coordinates of the vertical table line, the ending coordinates of the horizontal table line, and the ending coordinates of the vertical table line. The cell boundary range is determined based on the row spacing and column spacing of the table; Merged cells are determined by identifying breakpoints in adjacent table lines; The cell boundary range and the merged cells are identified by an optical character recognition engine to determine text and numerical information; Based on the region results for the different layout types, determine the image region pixel matrix; By analyzing the pixel matrix of the multimodal large model image and the analysis instructions, the object shape information, annotation information, and caption text information are determined. Based on the first recognition result, the text information, the numerical information, the object shape information, the annotation information, and the image caption text information, regional structured data is obtained.

4. The method according to claim 3, characterized in that, The step of extracting document metadata from the document to be processed and the region-level structured data to obtain document-level structured data includes: Read the document meta-information from the attributes of the document to be processed to obtain the first meta-information data; Under the condition that the first metadata satisfies the integrity constraint and the correlation constraint, the first structured data is obtained; Based on the aforementioned regional structured data, generate the input dataset; The input dataset and the specified prompt words are input into the fourth detection model to obtain the second structured data; Based on the first structured data and the second structured data, document-level structured data is generated.

5. A document processing system, characterized in that, include: The first acquisition module is used to acquire the document to be processed. The first generation module is used to classify the document to be processed into different file types and generate document classification results. The second generation module is used to analyze the document to be processed by calling the corresponding layout type region analysis model according to the document classification results, and generate region results for different layout types. The first module is used to parse the regional results of the different layout types to obtain regional structured data; The second obtaining module is used to obtain resource data when there are areas in the area results of the different page types whose content cannot be fully expressed by text. The third module is used to extract document metadata from the document to be processed and the region-level structured data to obtain document-level structured data. The third generation module is used to generate a structured dataset based on the document to be processed, the region-level structured data, the resource data, and the document-level structured data. The step of analyzing the document to be processed by calling the corresponding layout type region analysis model based on the document classification result and generating region results for different layout types includes: Based on the document classification results, a first detection model is determined; The weights and sensitivity of the first detection model in detecting each target region are optimized to obtain the second detection model; The document to be processed is subjected to target region detection according to the second detection model, and the coordinates of each target region, the type of each target region and the confidence level of each target region are generated. Based on the type of each target region, the coordinates of the target region, and the confidence level of the target region, generate region results for different layout types; The step of detecting target regions in the document to be processed according to the second detection model, and generating the coordinates of each target region, the type of each target region, and the confidence level corresponding to each target region includes: The document to be processed is subjected to target detection according to the second detection model, and a first confidence score is generated for each target region. Based on the first confidence level corresponding to each target region, determine the average confidence level of the target region; If the average confidence level of the target region is lower than a preset threshold, the original image of the low-confidence target region and its precise annotation information, as well as low-confidence samples detected historically by the second detection model under the same type of document, are obtained. The original image of the low-confidence target region, along with its precise annotation information and low-confidence samples detected historically by the second detection model under the same type of document, are used as training data to train the weight parameters and feature matching logic of the second detection model, thereby generating a third detection model. The document to be processed is subjected to target detection based on the third detection model, and the type, coordinates and confidence scores of each target region are generated.

6. The system according to claim 5, characterized in that, The first generation module includes: The first unit is used to determine the file extension, file header identifier, and metadata keywords based on the document to be processed; The second unit is used to encode the file suffix, the file header identifier, and the metadata keywords as a whole using the first specified model to generate a multi-dimensional feature vector. The first page elements of the document to be processed are analyzed using a second specified model, and multidimensional local visual features are output. The third unit is used to determine multiple high-frequency keywords based on the document to be processed; The fourth unit is used to encode the multiple high-frequency keywords through the third specified model to generate a multi-dimensional content shallow feature vector; The fifth unit is used to concatenate the multidimensional feature vector, the multidimensional local visual features, and the multidimensional content shallow feature vector to obtain a multimodal feature matrix. The sixth unit is used to generate a specified feature matrix based on the multimodal feature matrix and the shared projection matrix; The seventh unit is used to calculate the specified feature matrix using a scaled dot product attention algorithm to obtain a multidimensional context-aware feature vector; The eighth unit is used to determine the fused feature vector based on the multi-dimensional context-aware feature vector; The ninth unit is used to input the fused feature vector into the fully connected layer of the fourth specified model to generate output probabilities for each category; The tenth unit is used to output the type with the highest probability among the probabilities of each category, thus obtaining the document classification result.

7. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-4.

Images