Document analysis method and device and electronic equipment

Abstract

The invention discloses a document analysis method and device and electronic equipment. The method comprises the following steps: acquiring a document input by a user, and extracting metadata of the document; under the condition that the metadata indicates that the document is a pure document, text content of the pure document is extracted, a pure document analysis result is obtained, and the pure document is a document only containing text information; under the condition that the metadata indicates that the document is a heterogeneous document, the heterogeneous document is analyzed, a heterogeneous document analysis result is obtained, the heterogeneous document is a complex document containing text information, table information and image information, and analysis processing comprises format analysis, overlap filtering, layout rearrangement and content reconstruction processing on the heterogeneous document. According to the method and the device, the technical problem that the efficiency and the quality of enterprise-level knowledge base construction are limited due to the fact that a document analysis method in related technologies is limited in document content analysis capability and is difficult to adapt to documents of different formats and complex structures is solved.

Description

Technical Field

[0001] The present application relates to the field of document processing technology, and in particular to a document parsing method, device and electronic device. Background Art

[0002] In the practical scenarios of enterprise-level knowledge management and information retrieval, AI-driven search enhancement generation applications are gaining increasing attention. The core of AI-driven search enhancement generation applications is to integrate and analyze different document resources from within the enterprise, including but not limited to Txt text, Email, PPT presentation, Word document and PDF file, so as to facilitate subsequent information retrieval and knowledge generation. Since the format characteristics of these documents vary, intelligent analysis of their contents has become one of the key challenges in building an efficient knowledge base.

[0003] In the related technologies, strategies based on document parsers can extract information, but they have limited understanding of format diversity and content context, especially when dealing with complex charts. Although natural language processing is good at text analysis, it is difficult to properly handle non-text elements, such as images and tables, and may be inaccurate in the face of complex semantic relationships. Although the rule engine method is highly targeted, the rule definition is complex, the adaptability to new document types is poor, and the effect on unstructured or complex layout content is poor, which increases the system maintenance cost. These methods are insufficient in flexibility, compatibility and efficiency, and it is difficult to meet the needs of enterprise-level knowledge bases for intelligent parsing of heterogeneous documents.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a document parsing method, device and electronic device to at least solve the technical problem that the document parsing method in the related art has limited parsing ability for document content and is difficult to adapt to documents of different formats and complex structures, which limits the efficiency and quality of building an enterprise-level knowledge base.

[0006] According to one aspect of an embodiment of the present application, a document parsing method is provided, comprising: obtaining a document input by a user and extracting metadata of the document; when the metadata indicates that the document is a pure document, extracting text content of the pure document to obtain a pure document parsing result, wherein the pure document is a document containing only text information; when the metadata indicates that the document is a heterogeneous document, parsing the heterogeneous document to obtain a heterogeneous document parsing result, wherein the heterogeneous document is a complex document containing text information, table information and image information, and the parsing process includes performing layout analysis, overlap filtering, layout rearrangement and content reconstruction on the heterogeneous document, the layout analysis is used to identify the type and position of different blocks in the heterogeneous document, the overlap filtering is used to remove overlapping content between different blocks in the heterogeneous document, the layout rearrangement is used to rearrange the page layout of the heterogeneous document, and the content reconstruction is used to perform format conversion and standardization on the heterogeneous document.

[0007] Optionally, after obtaining the document input by the user, the method also includes: obtaining the document suffix and file header type identifier of the document; determining the document type of the document based on the document suffix and file header type identifier; and converting the document into the preset document type when the document type is inconsistent with the preset document type.

[0008] Optionally, layout analysis is used to identify the types and positions of different blocks in heterogeneous documents, including: determining the number of pages of the heterogeneous document, and converting the heterogeneous document into multiple image data based on the number of pages; converting the image data into a preset data format, wherein the preset data format is a data format readable by a layout analysis model; processing the heterogeneous document through the layout analysis model to obtain a layout analysis result, wherein the layout analysis result includes at least classification labels, block contents and bounding box coordinate information of different blocks in the heterogeneous document.

[0009] Optionally, overlapping filtering is used to remove overlapping content between different blocks in the heterogeneous document, including: determining overlapping content of the heterogeneous document according to bounding box coordinate information; and removing overlapping content according to preset filtering conditions.

[0010] Optionally, layout rearrangement is used to rearrange the page layout of heterogeneous documents, including: determining the layout order of blocks according to the bounding box coordinate information; arranging the blocks according to the layout order, and filling the block contents into the corresponding blocks to obtain a layout result.

[0011] Optionally, content reconstruction is used to perform format conversion and standardization processing on heterogeneous documents, including: converting text information, table information and image information in the layout result into a preset language format; and standardizing the layout result after format conversion.

[0012] Optionally, the method further includes: when the metadata indicates that the document is an encrypted document or garbled, ending the parsing process and returning a prompt message.

[0013] According to another aspect of an embodiment of the present application, a document parsing device is also provided, including: an acquisition module, used to acquire a document input by a user and extract metadata of the document; a first parsing module, used to extract text content of a pure document when the metadata indicates that the document is a pure document, and obtain a pure document parsing result, wherein a pure document is a document containing only text information; a second parsing module, used to parse the heterogeneous document when the metadata indicates that the document is a heterogeneous document, and obtain a heterogeneous document parsing result, wherein a heterogeneous document is a complex document containing text information, table information and image information, and the parsing process includes performing layout analysis, overlap filtering, layout rearrangement and content reconstruction on the heterogeneous document, the layout analysis is used to identify the type and position of different blocks in the heterogeneous document, the overlap filtering is used to remove overlapping content between different blocks in the heterogeneous document, the layout rearrangement is used to rearrange the page layout of the heterogeneous document, and the content reconstruction is used to perform format conversion and standardization on the heterogeneous document.

[0014] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned document parsing method.

[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned document parsing method by running the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is also provided, including computer instructions, which implement the above-mentioned document parsing method when executed by a processor.

[0017] In an embodiment of the present application, a document input by a user is obtained and metadata of the document is extracted; when the metadata indicates that the document is a pure document, the text content of the pure document is extracted to obtain a pure document parsing result, wherein a pure document is a document containing only text information; when the metadata indicates that the document is a heterogeneous document, the heterogeneous document is parsed to obtain a heterogeneous document parsing result, wherein a heterogeneous document is a complex document containing text information, table information and image information, and the parsing process includes performing layout analysis, overlap filtering, layout rearrangement and content reconstruction on the heterogeneous document, and the layout analysis is used to identify different blocks in the heterogeneous document. Type and position, overlapping filtering is used to remove overlapping content between different blocks in heterogeneous documents, layout rearrangement is used to rearrange the page layout of heterogeneous documents, and content reconstruction is used to convert the format and standardize the heterogeneous documents, achieving the purpose of deep analysis and standardization of heterogeneous document content, thereby achieving the technical effect of improving parsing efficiency, enhancing information extraction accuracy, and improving the processability and readability of document content, and further solving the technical problem that the document parsing method in the relevant technology has limited parsing ability for document content and is difficult to adapt to documents of different formats and complex structures, which limits the efficiency and quality of building an enterprise-level knowledge base. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a hardware structure diagram of a computer terminal for implementing a document parsing method according to an embodiment of the present application;

[0020] Figure 2 is a flowchart of a document parsing method according to an embodiment of the present application;

[0021] Figure 3 is a flowchart of another document parsing method according to an embodiment of the present application;

[0022] Figure 4 It is a structural diagram of a document parsing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] First, some nouns or terms that appear in the process of explaining the embodiments of the present application are subject to the following explanations:

[0026] RAG (Retrieval-Augmented Generation) is a technology that combines information retrieval and natural language generation. It can use the retrieval system to extract relevant information from large-scale documents or databases when generating text, so as to enhance the accuracy and contextual relevance of the generated content.

[0027] NLP (Natural Language Processing): A branch of artificial intelligence that focuses on enabling computers to understand, parse, and generate the ability to interact with human language. It covers speech recognition, text analysis, sentiment analysis, machine translation, and more.

[0028] Dynamic Policy Library: In this application, it refers to a system or component that can dynamically select the most appropriate parsing strategy based on the type, format and content of the document. The dynamic policy library enables the parsing system to flexibly respond to different types of documents and improve parsing efficiency and accuracy.

[0029] Layout analysis: A key technology in document parsing that is used to identify the layout and structure of elements (such as text, images, tables) on a document page in order to more accurately extract and reorganize these elements.

[0030] Markdown format: A lightweight markup language that is easy to read and write, and can be easily converted to other formats such as HTML and PDF. In document parsing, using Markdown format as a standardized output helps to uniformly manage and process content.

[0031] In order to solve the problem of poor document parsing efficiency in the related art, the present application embodiment provides a document parsing method, which can be run on Figure 1 In the computer terminal shown, the computer terminal is described below.

[0032] The document parsing method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a document parsing method. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0033] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0034] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the document parsing method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned document parsing method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0035] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0036] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0037] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0038] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a document parsing method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] Figure 2 is a flowchart of a document parsing method according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0040] Step S202: Obtain the document input by the user and extract metadata of the document.

[0041] In the above step S202, it is first necessary to receive documents uploaded by users or read by the system, including but not limited to emails, Word documents, PDF documents, presentations or other format documents. Subsequently, the basic type of the document can be determined by identifying the suffix name or file header identifier of the document. Next, the document is deeply analyzed to extract its metadata, including but not limited to document title, author, creation date, modification date, document summary and other information, so as to accurately judge the content characteristics and structure of the document, and provide a basis for the subsequent selection of parsing strategies.

[0042] Step S204, when the metadata indicates that the document is a pure document, extract the text content of the pure document to obtain a pure document parsing result, wherein the pure document is a document containing only text information.

[0043] In the above step S204, when the system confirms through metadata that the document is a pure document, that is, the document consists of text information only, without charts, images or complex layouts, the pure text parsing process will be directly started. For example, each line of text in the pure document is read and basic cleaning processing is performed, such as removing unnecessary spaces, line breaks, punctuation marks, etc., to ensure that the output text content is clear and coherent. Subsequently, the processed text content is output in a certain format, such as presented in a structured data format such as JSON, XML or CSV, so that subsequent processing can easily read and use the text information to form a pure document parsing result.

[0044] Step S206, when the metadata indicates that the document is a heterogeneous document, the heterogeneous document is parsed to obtain a heterogeneous document parsing result, wherein the heterogeneous document is a complex document containing text information, table information and image information, and the parsing process includes performing layout analysis, overlap filtering, layout rearrangement and content reconstruction on the heterogeneous document. The layout analysis is used to identify the type and position of different blocks in the heterogeneous document, the overlap filtering is used to remove overlapping content between different blocks in the heterogeneous document, the layout rearrangement is used to rearrange the page layout of the heterogeneous document, and the content reconstruction is used to perform format conversion and standardization on the heterogeneous document.

[0045] In the above step S206, the system will start a more comprehensive parsing process for complex heterogeneous documents containing text, tables and images. For example, the type and position of each block in the document are precisely identified by using layout analysis technology, laying the foundation for the analysis of complex content. Secondly, redundant information between blocks is effectively eliminated through overlapping filtering to purify document data. Finally, the logic of the document layout and the standardization of the content are ensured through layout rearrangement and content reconstruction, and non-text elements such as images and tables are converted into a structured format for subsequent processing. This series of operations ultimately achieves in-depth analysis of heterogeneous documents, generates analysis results with clear structure and complete information, and greatly improves the efficiency and accuracy of complex document processing.

[0046] Through the above steps S202 to S206, the purpose of deep analysis and standardization of heterogeneous document content is achieved, thereby realizing the technical effects of improving analysis efficiency, enhancing information extraction accuracy, and improving the processability and readability of document content, thereby solving the technical problem that the document parsing methods in related technologies have limited parsing capabilities for document content and are difficult to adapt to documents of different formats and complex structures, thus limiting the efficiency and quality of building an enterprise-level knowledge base.

[0047] Figure 3 FIG. 1 is a flowchart of another document parsing method according to an embodiment of the present application. Figure 3 As shown in FIG. 1 , the whole process of document parsing is shown in more detail, including: first, receiving documents to be parsed in different formats input by users or systems, identifying the type of documents by analyzing the document suffix or file header type identifier, and selecting appropriate methods to convert them into documents of the same type according to different document types, such as converting them into PDF type documents, and then reading the metadata of the documents to be parsed, and dividing them into pure (text) documents, garbled or encrypted documents, and heterogeneous documents according to the document content. Secondly, for pure (text) documents, the text content is directly extracted; for garbled or encrypted documents, the process is terminated and the end reason is returned. For heterogeneous documents, the system starts a more complex processing flow, including slicing and scaling them into image data, converting them into Numpy format after scaling, and then using the LayoutLMv3 model for layout analysis, identifying and locating the types and positions of different blocks, then removing redundant information between blocks through overlapping filtering, and rearranging the layout to adapt to the complex page structure. Finally, the image is converted into data format, chart objects are extracted, and noise is removed through content reconstruction processing. Finally, the system integrates the parsed document content into structured data and outputs standardized document content.

[0048] The following combination Figure 2 and Figure 3 The above steps S202 to S206 are described in detail.

[0049] In the above step S202, after obtaining the document input by the user, it also includes: obtaining the document suffix and file header type identifier of the document; determining the document type of the document based on the document suffix and file header type identifier; and converting the document into the preset document type when the document type is inconsistent with the preset document type.

[0050] In an embodiment of the present application, after receiving the document uploaded by the user, the system will determine the specific type of the document by analyzing the document's suffix name and file header type identifier. Because different document formats, such as text files (Txt), emails (Email), presentations (PPT), Word documents, and PDF files, have different internal structures and storage methods, they are directly related to the selection and processing efficiency of subsequent parsing methods. If the identified document type does not match the preset document type (PDF is the preset standard type in this application), the system will automatically convert the document to PDF format, which is to ensure that all documents can be processed in a unified format, thereby simplifying the parsing process, improving the versatility of the parsing strategy, and the consistency of the parsing results. Through this preprocessing step, the system can better adapt to subsequent deep processing methods such as layout analysis, layout filtering, and content reconstruction, and ultimately achieve efficient parsing and standardized output of heterogeneous documents.

[0051] In the above step S206, the layout analysis is used to identify the types and positions of different blocks in the heterogeneous document, including: determining the number of pages of the heterogeneous document, and converting the heterogeneous document into multiple image data based on the number of pages; converting the image data into a preset data format, wherein the preset data format is a data format readable by the layout analysis model; processing the heterogeneous document through the layout analysis model to obtain a layout analysis result, wherein the layout analysis result includes at least the classification labels, block contents and bounding box coordinate information of different blocks in the heterogeneous document.

[0052] In the embodiment of the present application, layout analysis is a key step in the in-depth analysis of heterogeneous documents. It is used to identify the types and specific locations of different blocks in the document, providing a basis for subsequent content filtering and reconstruction. The specific process can be as follows:

[0053] First, heterogeneous documents are converted into image data based on page slices. Specifically, the structure of the heterogeneous document is analyzed to determine the total number of pages it contains, and then the document information in all pages is converted into image data. This process also involves adjusting the image resolution to ensure that the image quality is suitable for subsequent deep learning model processing. The purpose of image processing is to convert the document content into a more intuitive and easy-to-analyze format to facilitate layout recognition by the layout analysis model.

[0054] Secondly, the generated image data is further converted into a preset data format, such as NumPy array, which is a format that deep learning models such as LayoutLMv3 can read and process. Formatting is a necessary step to ensure that the model can correctly process the input data, which may include data preprocessing, such as normalization, resizing, etc., to meet the input requirements of the model.

[0055] Next, the pre-trained LayoutLMv3 model is used to process the imaged and formatted documents. LayoutLMv3 is a deep learning model based on the Transformer architecture, which is specially designed to understand the layout and content of documents. Through model processing, the classification labels of different blocks in the document (for example, text, title, image, table, etc.), block content, and bounding box coordinate information of each block can be obtained, which provides a key basis for identifying the structural layout and content type of the document. Specifically, the classification label is used to help the system understand the function of each block, such as title, text, chart, etc.; the block content refers to the specific text or image information in the block; the bounding box coordinates accurately describe the position of each block in the page, which is crucial for subsequent processing such as layout rearrangement and content reconstruction.

[0056] Optionally, overlapping filtering is used to remove overlapping content between different blocks in the heterogeneous document, including: determining overlapping content of the heterogeneous document according to bounding box coordinate information; and removing overlapping content according to preset filtering conditions.

[0057] In an embodiment of the present application, the overlapping parts between blocks in heterogeneous documents can be accurately identified based on the bounding box coordinate information obtained by layout analysis. This step determines which blocks overlap each other on the page by analyzing the coordinates, thereby determining the overlapping content. Based on preset filtering conditions, such as the priority, type or content importance of the block, the system can intelligently decide which content to retain and which overlapping information to remove, ensuring the independence and accuracy of the content of each block, so as to avoid information redundancy and improve the clarity and accuracy of the parsing results. Through this process, overlapping filtering effectively improves the data quality after layout analysis, ensuring the accurate extraction of document content and the efficiency of subsequent processing.

[0058] Optionally, layout rearrangement is used to rearrange the page layout of heterogeneous documents, including: determining the layout order of blocks according to the bounding box coordinate information; arranging the blocks according to the layout order, and filling the block contents into the corresponding blocks to obtain a layout result.

[0059] In the embodiment of the present application, the layout rearrangement is also based on the deep processing of the layout analysis results. The bounding box coordinate information is also used to determine the relative position and logical order of each block in the document, and then these blocks are reorganized to eliminate confusion and overlap in the layout, and improve the readability and logical coherence of the document content. The system calculates and analyzes the coordinate information to clarify the correct arrangement order between the blocks, and then arranges the blocks in this order, and accurately fills the block content into the corresponding block position, generating a neater and more organized layout result. This process not only solves the interference of complex layout on content recognition, but also ensures the high-quality output of structured data, greatly improving the efficiency and accuracy of heterogeneous document parsing.

[0060] Optionally, content reconstruction is used to perform format conversion and standardization processing on heterogeneous documents, including: converting text information, table information and image information in the layout result into a preset language format; and standardizing the layout result after format conversion.

[0061] In the embodiment of the present application, for content reconstruction, the Markdown, a flexible and widely supported markup language format, is innovatively adopted to uniformly convert and standardize the text information, tables and image information after parsing heterogeneous documents. First, the parsed text information is converted into Markdown text to facilitate structured display and further processing. For table information, the system further identifies the table content in the image, converts and reconstructs it into a table in Markdown format through OCR technology to ensure the structuring and integrity of the data. In addition, the system uses regular expressions to filter document elements with no knowledge value, such as directories, headers and footers, etc., and formats key information such as titles to achieve a multi-level title structure, enhancing the hierarchy and readability of the document content. Through this series of format conversion and standardization, the present application achieves high-quality reconstruction of heterogeneous document content, provides a unified and standardized document data format for subsequent data analysis, knowledge management and RAG applications, and greatly improves processing efficiency and information availability.

[0062] Optionally, the method further includes: when the metadata indicates that the document is an encrypted document or garbled, ending the parsing process and returning a prompt message.

[0063] In the embodiment of the present application, if the metadata check reveals that the document has been encrypted or has garbled characters, the system will immediately terminate the subsequent parsing process and return a clear prompt message to the user, explaining why the document cannot be parsed, which may be garbled characters caused by encryption status or data corruption. This design is not only an effective management of resources, avoiding wasting computing resources on documents that cannot be parsed, but also provides user-friendly feedback to help users quickly locate problems and decide whether to provide document passwords, repair documents, or select other documents for parsing.

[0064] In general, this application first determines whether the document is suitable for parsing through document type identification and conversion, combined with metadata analysis, and then uses format analysis, content filtering and layout rearrangement technology based on the characteristics of heterogeneous documents, especially using the LayoutLMv3 model to process mixed chart documents, ensuring the accurate extraction of complex layout information. Finally, the document information is converted into Markdown format through content reconstruction, and the table image is OCR recognized and converted into a structured table. At the same time, regular expressions are used to remove worthless information and format the title, realizing in-depth analysis and standardization of the document content. Among them, the heterogeneous parsing algorithms involved are shown in Table 1:

[0065] Table 1 A dynamic strategy for heterogeneous document parsing

[0066]

[0067] This algorithm innovatively combines dynamic strategy selection, deep learning technology and content standardization output, effectively solving the problem of heterogeneous document parsing and providing advanced technical means for efficient data processing in building enterprise-level knowledge bases and document management systems.

[0068] In the embodiments of the present application, through the combination of dynamic strategies and deep content parsing technology, efficient and accurate parsing of heterogeneous documents is achieved, which provides strong technical support for the construction of enterprise-level knowledge bases and intelligent processing of document management systems, and significantly improves the application efficiency and flexibility of existing technologies in the field of document parsing. Specifically, by building an intelligent dynamic strategy library, the characteristics of documents in different formats are automatically identified and adapted to select the optimal parsing strategy. In particular, when processing heterogeneous documents containing complex charts and mixed layouts, technologies such as layout analysis, overlapping filtering and layout rearrangement are used to effectively improve the accuracy of content recognition and the output quality of structured data. In addition, through intelligent analysis of metadata, the system can determine whether the document is encrypted or damaged, terminate invalid parsing in a timely manner and feedback to the user, enhancing the robustness of the method and user experience.

[0069] According to an embodiment of the present application, a document parsing device is provided. It should be noted that the document parsing device of the embodiment of the present application can be used to execute the document parsing method provided by the embodiment of the present application. The document parsing device provided by the embodiment of the present application is introduced below.

[0070] Figure 4 is a structural diagram of a document parsing device provided according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0071] The acquisition module 40 is used to acquire the document input by the user and extract the metadata of the document;

[0072] A first parsing module 42 is used to extract text content of a pure document to obtain a pure document parsing result when the metadata indicates that the document is a pure document, wherein a pure document is a document containing only text information;

[0073] The second parsing module 44 is used to parse the heterogeneous document when the metadata indicates that the document is a heterogeneous document, and obtain the heterogeneous document parsing result, wherein the heterogeneous document is a complex document containing text information, table information and image information, and the parsing process includes performing layout analysis, overlap filtering, layout rearrangement and content reconstruction on the heterogeneous document. The layout analysis is used to identify the type and position of different blocks in the heterogeneous document, the overlap filtering is used to remove the overlapping content between different blocks in the heterogeneous document, the layout rearrangement is used to rearrange the page layout of the heterogeneous document, and the content reconstruction is used to perform format conversion and standardization on the heterogeneous document.

[0074] Through the acquisition module 40, the first parsing module 42 and the second parsing module 44 in the above-mentioned document parsing device, the purpose of deep parsing and standardizing heterogeneous document contents is achieved, thereby realizing the technical effects of improving parsing efficiency, enhancing information extraction accuracy, and improving the processability and readability of document contents, thereby solving the technical problem that the document parsing methods in related technologies have limited parsing capabilities for document contents and are difficult to adapt to documents of different formats and complex structures, thus limiting the efficiency and quality of building an enterprise-level knowledge base.

[0075] In the document parsing device provided in the embodiment of the present application, the acquisition module is also used to obtain the document suffix and file header type identifier of the document; determine the document type of the document based on the document suffix and file header type identifier; and convert the document into the preset document type when the document type is inconsistent with the preset document type.

[0076] In the document parsing device provided in the embodiment of the present application, the second parsing module is also used to determine the number of pages of the heterogeneous document, and convert the heterogeneous document into multiple image data based on the number of pages; convert the image data into a preset data format, wherein the preset data format is a data format readable by the layout analysis model; process the heterogeneous document through the layout analysis model to obtain a layout analysis result, wherein the layout analysis result includes at least classification labels, block contents and bounding box coordinate information of different blocks in the heterogeneous document.

[0077] In the document parsing device provided in the embodiment of the present application, the second parsing module is further used to determine the overlapping content of the heterogeneous documents based on the bounding box coordinate information; and remove the overlapping content based on the preset filtering conditions.

[0078] In the document parsing device provided in the embodiment of the present application, the second parsing module is also used to determine the layout order of the blocks based on the bounding box coordinate information; arrange the blocks according to the layout order, and fill the block contents into the corresponding blocks to obtain the layout result.

[0079] In the document parsing device provided in the embodiment of the present application, the second parsing module is also used to convert the text information, table information and image information in the layout result into a preset language format; and perform standardization processing on the layout result after the format conversion.

[0080] The document parsing device provided in the embodiment of the present application also includes a return module 46, which is used to end the parsing process and return prompt information when the metadata indicates that the document is an encrypted document or garbled.

[0081] An embodiment of the present application also provides an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned document parsing method.

[0082] It should be noted that the above electronic equipment is used to execute Figure 2 The document parsing method shown, therefore the relevant explanations and instructions in the above document parsing method are also applicable to the electronic device and will not be repeated here.

[0083] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored computer program, wherein a device where the non-volatile storage medium is located executes the above-mentioned document parsing method by running the computer program.

[0084] It should be noted that the above non-volatile storage medium is used to execute Figure 2 The document parsing method shown, therefore the relevant explanations in the above document parsing method are also applicable to the non-volatile storage medium, and will not be repeated here.

[0085] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the above-mentioned document parsing method when executed by a processor.

[0086] It should be noted that the above-mentioned computer program product is used to execute Figure 2 The document parsing method shown, therefore the relevant explanations and instructions in the above document parsing method are also applicable to the computer program product and will not be repeated here.

[0087] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0088] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0090] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0091] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0092] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0093] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A document parsing method, characterized in that: include: Obtaining a document input by a user and extracting metadata of the document; In the case where the metadata indicates that the document is a plain document, extracting text content of the plain document to obtain a plain document parsing result, wherein the plain document is a document containing only text information; In the case where the metadata indicates that the document is a heterogeneous document, the heterogeneous document is parsed to obtain a heterogeneous document parsing result, wherein the heterogeneous document is a complex document containing text information, table information and image information, and the parsing includes performing layout analysis, overlap filtering, layout rearrangement and content reconstruction on the heterogeneous document, the layout analysis is used to identify the type and position of different blocks in the heterogeneous document, the overlap filtering is used to remove overlapping content between different blocks in the heterogeneous document, the layout rearrangement is used to rearrange the page layout of the heterogeneous document, and the content reconstruction is used to perform format conversion and standardization on the heterogeneous document.

2. The method according to claim 1, characterized in that After obtaining the document input by the user, the method further includes: Obtaining a document suffix and a file header type identifier of the document; Determining the document type of the document according to the document suffix and the file header type identifier; When the document type is inconsistent with the preset document type, the document is converted into the preset document type.

3. The method according to claim 1, characterized in that The layout analysis is used to identify the types and positions of different blocks in the heterogeneous document, including: Determining the number of pages of the heterogeneous document, and converting the heterogeneous document into a plurality of image data according to the number of pages; Converting the image data into a preset data format, wherein the preset data format is a data format readable by a layout analysis model; The heterogeneous document is processed by the layout analysis model to obtain a layout analysis result, wherein the layout analysis result at least includes classification labels, block contents and bounding box coordinate information of different blocks in the heterogeneous document.

4. The method according to claim 3, characterized in that The overlapping filtering is used to remove overlapping contents between different blocks in the heterogeneous document, and includes: Determining overlapping content of the heterogeneous documents according to the bounding box coordinate information; The overlapping content is removed according to a preset filtering condition.

5. The method according to claim 3, characterized in that: The layout rearrangement is used to rearrange the layout of the heterogeneous documents, including: Determining a layout order of the blocks according to the bounding box coordinate information; The blocks are arranged according to the layout order, and the block contents are filled into corresponding blocks to obtain a layout result.

6. The method according to claim 5, characterized in that The content reconstruction is used to perform format conversion and standardization processing on the heterogeneous documents, including: Converting text information, table information and image information in the layout result into a preset language format; Standardize the layout results after format conversion.

7. The method according to claim 1, characterized in that The method further comprises: When the metadata indicates that the document is an encrypted document or garbled, the parsing process is terminated and a prompt message is returned.

8. A document parsing device, characterized in that: include: An acquisition module, used to acquire a document input by a user and extract metadata of the document; A first parsing module, configured to extract text content of the pure document to obtain a pure document parsing result when the metadata indicates that the document is a pure document, wherein the pure document is a document containing only text information; The second parsing module is used to parse the heterogeneous document to obtain the heterogeneous document parsing result when the metadata indicates that the document is a heterogeneous document, wherein the heterogeneous document is a complex document containing text information, table information and image information, and the parsing process includes performing layout analysis, overlap filtering, layout rearrangement and content reconstruction on the heterogeneous document, the layout analysis is used to identify the type and position of different blocks in the heterogeneous document, the overlap filtering is used to remove overlapping content between different blocks in the heterogeneous document, the layout rearrangement is used to rearrange the page layout of the heterogeneous document, and the content reconstruction is used to perform format conversion and standardization on the heterogeneous document.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory and is used to execute the document parsing method described in any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the document parsing method according to any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the document parsing method described in any one of claims 1 to 7 is implemented.

Images