Document splitting method and system, electronic equipment and readable storage medium

Abstract

The invention provides a document splitting method and system, electronic equipment and a readable storage medium, and belongs to the technical field of artificial intelligence. The method comprises the steps that multiple types of documents are obtained, and the formats of the multiple types of documents are different; analyzing each type of documents according to the target analysis strategy, and determining document contents of the type of documents; splitting the document content of each type of documents based on the target splitting strategy, and determining the split content of the type of documents; and outputting the split content of each type of documents in a structured manner. According to the document splitting method and system, the electronic equipment and the readable storage medium provided by the invention, the accuracy and efficiency of document splitting can be improved.

Description

Technical Field

[0001] The present disclosure belongs to the field of artificial intelligence technology, and more specifically, relates to a document splitting method and system, an electronic device, and a readable storage medium. Background Art

[0002] With the explosive growth of digital information, the management and processing of long documents has become a major challenge in many fields, especially in complex scenarios such as legal, financial, scientific research, and technical documents, where the length, structural complexity, and content diversity of documents have increased significantly. Traditional document processing methods mostly rely on rule-based or template-based splitting methods, but these methods are inefficient and inaccurate for long documents with low structuring and complex semantics. Especially in the scenario of unstructured documents, the splitting results are often difficult to meet the needs of subsequent analysis or processing. Therefore, the existing technology has the limitations of low efficiency, insufficient accuracy, and difficulty in adapting to complex structures and semantics when processing long documents. Summary of the invention

[0003] The purpose of the present disclosure is to provide a document splitting method and system, an electronic device, and a readable storage medium to improve the accuracy and efficiency of document splitting.

[0004] A first aspect of the embodiments of the present disclosure provides a document splitting method, comprising: Obtain multiple types of documents, wherein the formats of the multiple types of documents are different; Parse each type of document based on the target parsing strategy to determine the document content of the document; Splitting the document content of each type of document based on the target splitting strategy, and determining the split content of the document of this type; Structured output of the split content of each type of document.

[0005] A second aspect of the embodiments of the present disclosure provides a document splitting system, including: A document acquisition module is used to acquire multiple types of documents, wherein the multiple types of documents have different formats; A parsing module, used to parse each type of document based on a target parsing strategy to determine the document content of the document; A splitting module is used to split the document content of each type of document based on the target splitting strategy and determine the split content of the document of this type; The structured output module is used to structure the output of the split contents of each type of document.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned document splitting method when executing the computer program.

[0007] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned document splitting method are implemented.

[0008] The document splitting method and system, electronic device, and readable storage medium provided by the embodiments of the present disclosure have the following beneficial effects: On the one hand, the present disclosure can enhance the accuracy of document splitting. Based on the deep semantic understanding of the large model, the accurate grasp of the semantic structure of the document is achieved, and the text blocks can be reasonably divided according to the contextual relationship, avoiding the errors of splitting based on fixed templates or formats. Each split block has strong semantic integrity and independence, providing accurate input for subsequent text retrieval and generation, ensuring that information will not be lost or ambiguous due to unreasonable splitting.

[0009] On the other hand, the present disclosure can improve document processing efficiency. The present disclosure uses the semantic understanding ability of a large-scale pre-trained model to automatically and efficiently split long documents, reducing the need to rely on complex rules or manual annotation in traditional document processing methods. Compared with the existing technology, the processing time is significantly shortened, especially in scenarios where large quantities of documents need to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A schematic diagram of a flow chart of a document splitting method provided in an embodiment of the present disclosure; Figure 2 A schematic diagram of a specific process of splitting a docx format document provided in an embodiment of the present disclosure; Figure 3 A schematic diagram of a specific process of a PDF format document splitting method provided by an embodiment of the present disclosure; Figure 4 A structural block diagram of a document splitting system provided by an embodiment of the present disclosure; Figure 5 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0012] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0013] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0014] Please refer to Figure 1 , Figure 1 A schematic diagram of a document splitting method provided in an embodiment of the present disclosure, the method comprising: S101: Acquire multiple types of documents, where the formats of the multiple types of documents are different.

[0015] In this embodiment, before obtaining multiple types of documents, the following steps are also included: Upload each type of document; Each category of documents is summarized to determine multiple categories of documents.

[0016] Multi-type documents are different types of documents, that is, they have different formats. Multi-type documents can be docx documents, pdf documents, txt documents, markdown documents, and doc documents. Documents of different formats differ in editing methods and presentation forms. For example, docx format documents are an open XML format used by Microsoft Word software, which can well preserve information on various elements such as text layout, pictures, and charts; pdf format documents (portable document format) focus on maintaining the consistency of document display on different devices and software, and are often used for fixed-layout document publishing, etc.; txt format documents are plain text formats, containing only simple text content, with almost no complex layout and formatting; markdown format documents are a lightweight markup language format that uses simple markup symbols to typeset text, and are often used for writing, document recording, etc.; doc format documents are binary document formats commonly used by early Microsoft Word software. Because documents of different formats have different characteristics, the subsequent parsing and splitting processing methods will also be different accordingly.

[0017] S102: Parse each type of document based on the target parsing strategy to determine the document content of the document.

[0018] In this embodiment, parsing each type of document is performed by deeply parsing documents of different formats through a dedicated parser or library.

[0019] The target parsing strategy is a set of rules, methods and processes that are pre-established for documents of different formats. Its purpose is to accurately and effectively extract the content information contained in documents of the corresponding format.

[0020] If each type of document is in docx format: use mature open source libraries (such as Python-docx, Pandoc) to parse the document structure using the following formula: .

[0021] in, Represents the document itself, Represents text, Indicates the picture, Represents the formula, Represents a table, maintaining the hierarchy and logical structure of the document. Also the abbreviation of docx.

[0022] If each type of document is in PDF format: use PDF to Image technology to convert the PDF format document into an image, and combine it with the visual detection model to further analyze the content in the image.

[0023] If each type of document is in txt format: Due to its simple structure, the text content can be semantically analyzed and split directly through natural language processing technology without the need for complex layout analysis.

[0024] If each type of document is in markdown format: first identify various tags in the markdown document, such as # for titles, - for lists, ``` for code blocks, etc. Based on these tags, the system will initially split the document into chapters, paragraphs, and other structures, extract images, hyperlinks, and other content, and then combine the large model for semantic division and splitting to maintain the integrity and logical consistency of the page content.

[0025] If each type of document is in doc format: you need to use LibreOffice to convert the doc format document into docx format document for further processing.

[0026] For complex document formats (such as PDF), visual detection models can be used for content annotation to accurately annotate content blocks in the document. The visual detection model recognizes different objects and areas in the image through pre-training, and annotates the pages of PDF format documents in blocks, mainly including: Text blocks: The visual detection model separates text areas from other non-text areas (such as images and tables) to ensure efficient text extraction. The text areas are separated from non-text areas through the following marking function:

[0027] in, Represents a block of content that is identified as "text", i.e. a text block, Indicates each block of content to be identified as "text". Represents the set of all content blocks identified as "text". Indicates that the identified content block must belong to , while satisfying , indicating that the recognized content must be of text type.

[0028] Images and tables: For image and table areas, the visual inspection model can not only identify the location, but also extract image descriptions and table borders, cells and other details.

[0029] Formula: The visual inspection model is specially optimized for the detection of complex formulas, distinguishing between text, images and formulas, and accurately marking them in an editable typesetting format, namely LaTeX format.

[0030] The introduction of visual detection models makes content annotation more accurate, especially when processing unstructured documents (such as scanned PDFs), ensuring the distinction and location of different content blocks in each page.

[0031] Layout analysis technology analyzes the page structure and layout information of the document to achieve intelligent division of different areas. Based on the annotation information generated by the visual detection model, the document page can be further analyzed in detail: Region aggregation: The system can aggregate the same layout area (such as header, body, footer, etc.) according to the layout characteristics of the document. Ensure that the split document can maintain the original layout logic. The region aggregation formula can be as follows: .

[0032] in, Represents a collection of overall areas, that is, each layout area The union of Represents each layout area (such as header, body, footer, etc.), ensuring that the split document maintains the original layout logic.

[0033] Multi-level analysis: Layout analysis not only identifies the physical location of text blocks, but also combines the semantic information of the content to automatically determine hierarchical information such as chapter titles and paragraphs, providing basic support for subsequent text segmentation.

[0034] Table and image layout recognition: Layout analysis can accurately identify the row and column structure of a table, and automatically convert the table into Hypertext Markup Language (html) format by combining it with the visual big model; when annotating images, its relative position in the document is maintained.

[0035]

[0036] in, Indicates the converted HTML format table. Represents parsed table data.

[0037] Based on visual detection and layout analysis, the visual big model can be further used for high-level content extraction. The visual big model deeply analyzes complex objects in the document, especially pictures, tables and formulas through prompt engineering: Image content extraction: The visual big model can automatically identify important elements in the image and generate descriptive information, such as the content overview of the image, the relationship between the image and the surrounding text, etc.

[0038] Formula content extraction: The visual model can identify embedded formulas in documents, whether they are standard formulas in LaTeX format or formulas presented in the form of images. For formulas in the form of images, the model can automatically generate LaTeX representations of the formulas, ensuring that the formulas maintain the integrity of their mathematical expressions during subsequent editing and use.

[0039]

[0040] in, Represents the generated LaTeX format formula, Represents a formula in pictorial form.

[0041] Through the advanced content extraction of the visual big model, it is possible to deeply analyze documents with complex structures and provide important support for subsequent text segmentation and manual compensation.

[0042] S103: Split the document content of each type of document based on the target splitting strategy to determine the split content of the document of this type.

[0043] In this embodiment, the target splitting strategy is a set of pre-planned specific rules, methods and ideas, which aims to reasonably split the parsed document content according to specific standards or dimensions, so that it becomes smaller units that are more convenient for subsequent operations (such as analysis, organization, archiving, etc.). Different types of documents have different target splitting strategies.

[0044] The text segmentation phase (i.e., target segmentation strategy) can be based on large language models (such as GPT, QWEN, etc.) and combined with prompt word engineering to achieve semantic understanding and intelligent segmentation of long texts. The text segmentation process is as follows: Pagination iterative processing: In order to solve the problem that the document length exceeds the model processing capacity, the present invention uses pagination iterative technology to process a certain amount of text content each time to ensure that the number of tokens is within a reasonable range.

[0045]

[0046] in, Represents a set of text blocks divided by pages. Represents a single token in each text block, Indicates the maximum value of a token. This formula indicates that the token value of the text block set after division cannot exceed the maximum value.

[0047] Semantic analysis and hierarchical division: The large language model automatically detects hierarchical information such as chapters, paragraphs, and titles in the document through semantic analysis of the document content, ensuring that each text block is logically independent and coherent.

[0048] Semantic chunking: The model analyzes the semantic coherence between different paragraphs to ensure that the split text chunks remain complete and accurate in meaning, facilitating retrieval and processing in subsequent retrieval-augmented generation (RAG) scenarios.

[0049]

[0050] in, represents a collection of semantic blocks, Indicates A separate semantic block, Indicates that the semantic block is coherent, .

[0051] Through large-model text splitting, the present invention can efficiently divide large-scale documents into multiple semantically independent text blocks while retaining their logical and structural integrity, significantly improving the document's usability and processing efficiency.

[0052] S104: Structurally output the split contents of each type of document.

[0053] In this embodiment, the split contents are output according to a predetermined format (Markdown) to facilitate the subsequent integration and use of the RAG system or other knowledge management platforms.

[0054] Before the structured output of the split content of each type of document, it also includes: If the split content of each type of document meets the first condition, structured output is performed; If the split contents of each type of document do not meet the first condition, the split contents of the document of this type are compensated, the optimized split contents of the document of this type are determined, and then the structured output is performed; The first condition is the accuracy of the split content of each type of document.

[0055] When the accuracy of the split contents of each type of document obtained in steps S101-S103 is greater than or equal to the first preset accuracy, the structured output can be directly performed; When the accuracy of the split content of each type of document obtained in steps S101-S103 is less than the first preset accuracy, it is necessary to deal with the error in automatic splitting, and a compensation mechanism (for example, a simple and efficient manual compensation mechanism) can be used for adjustment and optimization. After the compensation is completed, the structured output is performed.

[0056] Users can manually adjust and optimize the following aspects: (1) Visual repair page: The system provides users with an intuitive visual interface where they can view the text blocks after the document is split and make manual adjustments. Through drag-and-drop operations, users can freely split or merge unreasonable text blocks to ensure that the split results meet expectations.

[0057] (2) Manual image insertion function: The system allows users to manually adjust the position of images and associated text. If the image and text do not match or the image is lost during the automatic parsing process, the user can manually insert or adjust the matching relationship between the image and the corresponding text.

[0058] (3) Formula writing auxiliary tools: Since the formula format may not be unified, some formulas may exist in the form of pictures or old version formula components. In order to ensure the correctness and standardization of the formula, the system provides auxiliary tools with Mathpix as the core, which support users to use auxiliary tools to assist in generating LaTeX syntax when manually writing or correcting formulas.

[0059] The splitting method disclosed in this disclosure provides more accurate input for RAG technology. In RAG application scenarios, accurate text blocks can quickly locate relevant information through the retrieval module and provide clear and context-consistent content support in the generation phase, thereby improving the overall retrieval and generation quality, which is particularly suitable for scenarios with long text or complex semantic documents.

[0060] The split document blocks are output in a standardized format (such as JSON, XML, etc.), which is easy to integrate into the existing RAG system, knowledge management platform or other document processing framework, providing convenient conditions for subsequent data retrieval, analysis and generation. This method has good scalability, is suitable for seamless integration with existing systems, and supports further optimization and expansion.

[0061] The present disclosure reduces the reliance on human intervention in the traditional document processing process through an automated document splitting method, which can significantly reduce labor costs and improve the work efficiency of enterprises or institutions, especially when processing a large number of documents or complex semantic content.

[0062] It can be concluded from the above that, on the one hand, the present disclosure can enhance the accuracy of document splitting. Based on the deep semantic understanding of the large model, the accurate grasp of the semantic structure of the document is achieved, and the text blocks can be reasonably divided according to the contextual relationship, avoiding the errors based on fixed templates or format splitting. Each split block has strong semantic integrity and independence, providing accurate input for subsequent text retrieval and generation, ensuring that information will not be lost or ambiguous due to unreasonable splitting.

[0063] On the other hand, the present disclosure can improve document processing efficiency. The present disclosure uses the semantic understanding ability of a large-scale pre-trained model to automatically and efficiently split long documents, reducing the need to rely on complex rules or manual annotation in traditional document processing methods. Compared with the existing technology, the processing time is significantly shortened, especially in scenarios where large quantities of documents need to be processed.

[0064] In one embodiment of the present disclosure, a document splitting method includes: Determine the target parsing strategy corresponding to each type of document based on the format information of the document; A target splitting strategy corresponding to each type of document is determined based on the format information of the document.

[0065] In one embodiment of the present disclosure, determining a target parsing strategy corresponding to each type of document based on the format information of the document includes: In response to the format information being a first type of document, using the first parsing strategy as a target parsing strategy; In response to the format information being a second type of document, using the second parsing strategy as a target parsing strategy; Among them, the first parsing strategy and the second parsing strategy are processed differently; Based on the format information of each type of document, the target splitting strategy corresponding to the document type is determined, including: In response to the format information being a first type of document, using the first splitting strategy as a target splitting strategy; In response to the format information being a second type of document, using the second splitting strategy as the target splitting strategy; The first splitting strategy and the second splitting strategy are processed in different ways.

[0066] In this embodiment, the present disclosure supports parsing and splitting of multiple document formats, including but not limited to common file types such as docx, pdf, etc. For each document format, a special parsing and processing method is used to ensure the accuracy and effectiveness of the splitting effect.

[0067] Format information refers to the relevant identifiers or feature descriptions that can characterize the document format type. Different document formats (such as docx, pdf, txt, etc.) have their own unique format information. This information can be recognized by computer systems, software, etc., so as to distinguish which format the document belongs to. For example, the file extension (.docx represents the word document format) is a common embodiment of format information.

[0068] First-category documents and second-category documents are just abstract names for documents of different formats. They are used to distinguish documents of different categories and formats, so as to illustrate the different strategies adopted for documents of different formats. They can represent any two documents of different formats in reality. For example, first-category documents can be docx format documents, and second-category documents can be pdf format documents, etc.

[0069] The first parsing strategy and the second parsing strategy correspond to the parsing methods of the first and second types of documents respectively. Since the internal structures and storage methods of documents in different formats are different, the parsing methods (processing methods) are different. For example, the first parsing strategy can be a method for extracting content from docx format documents by parsing the XML structure using a specific software library. The second parsing strategy can be a method for first determining the type of pdf format documents and then performing corresponding text extraction and chart parsing operations.

[0070] The first splitting strategy and the second splitting strategy are content splitting methods for the first and second types of documents respectively. Also, due to the different content characteristics and application scenarios of documents in different formats, the splitting methods (processing methods) will also be different. For example, the first splitting strategy can be to split the content according to the chapter titles in the docx format document, and the second splitting strategy can be to split the content according to the page sections in the pdf format document, etc.

[0071] It can be concluded from the above that the method used in this embodiment is not limited to specific types of documents, but can be flexibly applied to structured and unstructured documents, and is applicable to multiple fields, such as document management, text analysis, legal document processing, knowledge base construction, etc. Through intelligent splitting, the automation level of document processing has been effectively improved, which is suitable for document management and processing needs in various business scenarios.

[0072] In one embodiment of the present disclosure, the first type of document is an editable document; The target parsing strategy is the first parsing strategy; Parse each type of document based on the target parsing strategy to determine the document content of this type of document, including: Loading a first category of documents; wherein the first category of documents is one of each category of documents; Parsing the first type of document to generate first document content, where the first document content is at least one of a text paragraph, a formula, a table, and a picture; If the first document content includes a formula, converting the formula into a typesetting format; If the first document content includes a table, converting the table into a hypertext markup language format; If the first document content includes a picture, the picture is annotated based on the first model to generate annotation information.

[0073] In this embodiment, the first type of document is an editable document, that is, a document in docx format, a typesetting format in LaTeX format, and a hypertext markup language format in html format.

[0074] Specifically, the target parsing strategy is the first parsing strategy, and the steps of the first parsing strategy can be as follows (refer to Figure 2 ): First, load the docx library.

[0075] Then, for docx format documents, a dedicated parser is used to automatically extract the text paragraphs, formulas, tables, pictures and other contents in the document. For example, the document structure is parsed using an open source docx parsing library to extract all paragraphs and titles, keeping the logical structure of the document intact.

[0076] Secondly, for the formula part, after using pandoc to convert the document into markdown, you can directly parse the markdown to identify the embedded formula objects and extract them into LaTeX format for subsequent use or display.

[0077] After that, for the table part, you can convert the table into HTML format.

[0078] Finally, for the pictures embedded in the document, after parsing the pictures, the picture content is extracted through the visual big model (the first model, such as QWENVL) prompt word engineering, and the corresponding annotation information is generated, such as picture description, location, etc.

[0079] From the above, it can be concluded that for editable docx format documents, a specific first parsing strategy is used to achieve efficient and accurate parsing. Loading the docx library and the dedicated parser can completely extract multiple elements, retain the logical structure of the document, and provide rich materials for subsequent processing. Formulas are converted to LaTex format, which enhances the versatility and compatibility of formulas in different platforms and applications, and facilitates editing and display. Tables are converted to html format, which is conducive to flexible presentation and processing on the web page. With the help of visual large models to annotate pictures, pictures are given more searchable and understandable information, which greatly improves the usability and processing depth of document content.

[0080] In one embodiment of the present disclosure, the target splitting strategy is a first splitting strategy; The document content of each type of document is split based on the target splitting strategy to determine the split content of the document of this type, including: Define the number of pages for each processing of the first type of documents; The content of each processed page is used as the current range document content; the current range document content is part of the first document content; Process the document content in the current range and determine the splitting prompt words; Input the splitting cue words into the second model to obtain splitting segments; When the sum of the number of pages processed each time is less than the total number of pages of the first category of documents, the number of pages processed each time for defining the first category of documents continues to be executed; When the sum of the number of pages processed each time is greater than or equal to the total number of pages of the first category of documents, the execution of defining the number of pages processed each time for the first category of documents is stopped; Aggregate multiple split fragments to obtain split contents of the first category of documents.

[0081] In this embodiment, the first splitting strategy corresponds to an editable document, that is, a document in docx format.

[0082] Specifically, the document content of each type of document is split based on the first splitting strategy, and the process of determining the split content of the document of this type is as follows (refer to Figure 2 ): First, you need to define the number of pages to be processed each time and determine the content of the current scope document.

[0083] Secondly, the content of the current scope document is processed to determine the splitting prompt words.

[0084] Then, the splitting prompt words are processed based on the large model (the second model) and the split segments are output.

[0085] Afterwards, since the pages of the first category of documents have not been processed yet, it is necessary to iterate from the first step (determine the content of the current range of documents) until all pages are processed.

[0086] Finally, the results are aggregated to generate the split content of the first category of documents.

[0087] Based on the large language model (such as GPT, QWEN, etc.), the prompt word project performs semantic analysis on the extracted text content in pages and iterations. The number of pages for each iteration can be customized to control the length of the token, and the document is intelligently split according to the hierarchical structure of the document (such as chapters, paragraphs, titles, etc.). This step divides the docx document into multiple semantically independent text blocks, ensuring that each text block is semantically complete and clear, which is convenient for subsequent retrieval and generation operations.

[0088] From the above, it can be concluded that this embodiment can process the first type of documents in orderly segments by defining the number of pages to be processed each time, thereby avoiding the resource pressure and inefficiency caused by processing large-scale data at one time. Determining the splitting prompt words can accurately guide the splitting according to the characteristics of the document itself, thereby improving the accuracy and pertinence of the splitting. With the help of the second model, the split fragments are generated, and the model's powerful semantic understanding and information processing capabilities are used to make the splitting results more reasonable and logical. Iterate the process until all pages are completed to ensure that the document is completely split and the fragments are closely connected. Finally, the split fragments are aggregated to obtain the complete split content, which is convenient for the subsequent classification, retrieval, and analysis of the document content, and significantly improves the efficiency and quality of document management and utilization.

[0089] In one embodiment of the present disclosure, the second type of document is a non-editable document; The target parsing strategy is the second parsing strategy; Parse each type of document based on the target parsing strategy to determine the document content of this type of document, including: Converting a second type of document into a second image; wherein the second type of document is one type of each type of document; Parse the second image based on the third model to obtain second document content, where the second document content is at least one of a text block, a formula block, a table block, and a picture block; If the second document content includes a text block, converting the text block into text based on the first recognition model; If the second document content includes a formula block, saving a screenshot of the formula block; If the second document content includes a table block, saving a screenshot of the table block; If the second document content includes a picture block, a screenshot of the picture block is saved.

[0090] In this embodiment, the second type of document is a non-editable document, that is, it can be a document in PDF format.

[0091] Specifically, the target parsing strategy is the second parsing strategy, and the steps of the second parsing strategy can be as follows (refer to Figure 3 ): First, convert the second type of documents (pdf format documents) into images and then perform subsequent processing.

[0092] Preprocess the PDF document and use the optimized visual inspection model to identify and parse the PDF document content. Through PDF to Image technology, each page of the PDF document is converted into a high-quality image for subsequent visual inspection model recognition.

[0093] Second, use the visual detection model (third model) to annotate the content in the image, marking the text blocks, images, tables, and possible formulas of each page to ensure the accuracy of content parsing.

[0094] From the above, we can conclude that converting non-editable documents into images for processing effectively solves the problem that the format is not easy to parse directly, so that the document content can be recognized by the system. Using the optimized visual detection model to mark various blocks ensures the accuracy of parsing and can accurately locate different elements.

[0095] In one embodiment of the present disclosure, the target splitting strategy is the second splitting strategy; The document content of each type of document is split based on the target splitting strategy to determine the split content of the document of this type, including: Define the number of pages for each processing of the second type of documents; The content of each processed page is used as the current range document content; the current range document content is part of the second document content; Perform layout analysis on the current scope document content to determine the page structure of the current scope document content; Extract the page structure and determine the extracted content of the current range of document content; When the sum of the number of pages processed each time is less than the total number of pages of the second category of documents, the number of pages processed each time for defining the second category of documents continues to be executed; When the sum of the number of pages processed each time is greater than or equal to the total number of pages of the first category of documents, the execution of defining the number of pages processed each time for the second category of documents is stopped; Aggregating the multiple extracted contents to determine the extracted contents of the second category of documents; The extracted content of the second category of documents is split based on the fourth model to obtain the split content of the second category of documents.

[0096] In this embodiment, layout analysis and text block extraction (reference Figure 3), the page structure of the PDF document is identified through the layout analysis algorithm, and the content in the same layout area is aggregated into independent areas. For example, the header, footer, and body, and the OCR model is used to perform text recognition and extraction in these areas.

[0097] For the extraction of tables and formulas, the system uses a large visual model combined with prompt word engineering to identify and extract their structure.

[0098] The formula content is identified and converted into LaTeX format to ensure the subsequent editable and accurate display of mathematical formulas.

[0099] The table content is automatically parsed and converted into HTML format to maintain the integrity and editability of the table structure.

[0100] Based on the extracted content and page layout analysis, the text is further semantically split, and the semantic coherence between different paragraphs or blocks is analyzed in combination with the large model to generate multiple semantically independent text blocks. Similarly, in order to control the token length, iterative paging processing is supported.

[0101] Finally, the results are aggregated to generate the extracted content of the second category of documents, which are then split based on the large model (the fourth model) to determine the split content of the second category of documents.

[0102] It can be concluded from the above that this embodiment avoids the complexity of large-scale data processing and improves processing efficiency and stability by defining the number of processing pages for block processing. The layout analysis of the page structure helps to deeply understand the distribution of document content and accurately extract key information, so that the extracted content is more in line with the document logic. At the same time, the content is first extracted according to the page structure, and then split with the help of the fourth model to further refine the content and improve the accuracy and meticulousness of the split.

[0103] Layout analysis and text block extraction can distinguish headers, footers and other areas and extract text to make the document structure clearer. Formulas and tables are converted to specific formats, such as formulas to LaTeX format and tables to HTML format, which not only retains their structural integrity, but also enhances subsequent editability and display effects, greatly improving the content availability of non-editable documents.

[0104] Corresponding to the document splitting method of the above embodiment, Figure 4 This is a structural block diagram of a document splitting system provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 4 The document splitting system 20 includes: a document acquisition module 21, a parsing module 22, a splitting module 23 and a structured output module 24.

[0105] The document acquisition module 21 is used to acquire multiple types of documents, wherein the multiple types of documents have different formats; A parsing module 22, used to parse each type of document based on a target parsing strategy to determine the document content of the document; A splitting module 23 is used to split the document content of each type of document based on the target splitting strategy and determine the split content of the document of this type; The structured output module 24 is used to structure the output of the split contents of each type of document.

[0106] In one embodiment of the present disclosure, the document splitting system 20 further includes: a policy determination module; A strategy determination module, used to determine the target parsing strategy corresponding to each type of document based on the format information of the document; A target splitting strategy corresponding to each type of document is determined based on the format information of the document.

[0107] In one embodiment of the present disclosure, the policy determination module is further configured to: In response to the format information being a first type of document, using the first parsing strategy as a target parsing strategy; In response to the format information being a second type of document, using the second parsing strategy as a target parsing strategy; Among them, the first parsing strategy and the second parsing strategy are processed differently; Based on the format information of each type of document, the target splitting strategy corresponding to the document type is determined, including: In response to the format information being a first type of document, using the first splitting strategy as a target splitting strategy; In response to the format information being a second type of document, using the second splitting strategy as the target splitting strategy; The first splitting strategy and the second splitting strategy are processed in different ways.

[0108] In one embodiment of the present disclosure, the first type of document is an editable document; The target parsing strategy is the first parsing strategy; The parsing module 22 is further specifically used for: Loading a first category of documents; wherein the first category of documents is one of each category of documents; Parsing the first type of document to generate first document content, where the first document content is at least one of a text paragraph, a formula, a table, and a picture; If the first document content includes a formula, converting the formula into a typesetting format; If the first document content includes a table, converting the table into a hypertext markup language format; If the first document content includes a picture, the picture is annotated based on the first model to generate annotation information.

[0109] In one embodiment of the present disclosure, the target splitting strategy is a first splitting strategy; The splitting module 23 is further used for: Define the number of pages for each processing of the first type of documents; The content of each processed page is used as the current range document content; the current range document content is part of the first document content; Process the document content in the current range and determine the splitting prompt words; Input the splitting cue words into the second model to obtain splitting segments; When the sum of the number of pages processed each time is less than the total number of pages of the first category of documents, the number of pages processed each time for defining the first category of documents continues to be executed; When the sum of the number of pages processed each time is greater than or equal to the total number of pages of the first category of documents, the execution of defining the number of pages processed each time for the first category of documents is stopped; Aggregate multiple split fragments to obtain split contents of the first category of documents.

[0110] In one embodiment of the present disclosure, the second type of document is a non-editable document; The target parsing strategy is the second parsing strategy; The parsing module 22 is further used for: Converting a second type of document into a second image; wherein the second type of document is one type of each type of document; Parse the second image based on the third model to obtain second document content, where the second document content is at least one of a text block, a formula block, a table block, and a picture block; If the second document content includes a text block, converting the text block into text based on the first recognition model; If the second document content includes a formula block, saving a screenshot of the formula block; If the second document content includes a table block, saving a screenshot of the table block; If the second document content includes a picture block, a screenshot of the picture block is saved.

[0111] In one embodiment of the present disclosure, the target splitting strategy is the second splitting strategy; The splitting module 23 is further used for: Define the number of pages for each processing of the second type of documents; The content of each processed page is used as the current range document content; the current range document content is part of the second document content; Perform layout analysis on the current scope document content to determine the page structure of the current scope document content; Extract the page structure and determine the extracted content of the current range of document content; When the sum of the number of pages processed each time is less than the total number of pages of the second category of documents, the number of pages processed each time for defining the second category of documents continues to be executed; When the sum of the number of pages processed each time is greater than or equal to the total number of pages of the first category of documents, the execution of defining the number of pages processed each time for the second category of documents is stopped; Aggregating the multiple extracted contents to determine the extracted contents of the second category of documents; The extracted content of the second category of documents is split based on the fourth model to obtain the split content of the second category of documents.

[0112] See also Figure 5 , Figure 5 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 5 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of each module / unit in the above-mentioned system embodiments, such as Figure 4 The functions of modules 21 to 24 are shown.

[0113] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0114] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0115] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0116] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of the document splitting method provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device described in the embodiments of the present disclosure, which will not be repeated here.

[0117] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0118] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0119] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0120] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0121] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0122] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0123] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0124] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A document splitting method, characterized in that: include: Obtain multiple types of documents, wherein the formats of the multiple types of documents are different; Parse each type of document based on the target parsing strategy to determine the document content of the document; Splitting the document content of each type of document based on the target splitting strategy, and determining the split content of the document of this type; Structured output of the split content of each type of document.

2. The document splitting method according to claim 1, characterized in that: Also includes: Determine the target parsing strategy corresponding to each type of document based on the format information of the document; A target splitting strategy corresponding to each type of document is determined based on the format information of the document.

3. The document splitting method according to claim 2, characterized in that: The step of determining a target parsing strategy corresponding to each type of document based on the format information of the document includes: In response to the format information being a first type of document, using the first parsing strategy as a target parsing strategy; In response to the format information being a second type of document, using the second parsing strategy as a target parsing strategy; The first parsing strategy and the second parsing strategy have different processing methods; The step of determining the target splitting strategy corresponding to each type of document based on the format information of the document includes: In response to the format information being a first type of document, using the first splitting strategy as a target splitting strategy; In response to the format information being a second type of document, using the second splitting strategy as a target splitting strategy; The first splitting strategy and the second splitting strategy are processed in different ways.

4. The document splitting method according to claim 3, characterized in that: The first type of document is an editable document; The target parsing strategy is the first parsing strategy; The step of parsing each type of document based on the target parsing strategy to determine the document content of the document includes: Loading the first category of documents; wherein the first category of documents is one of each category of documents; Parsing the first type of document to generate first document content, where the first document content is at least one of a text paragraph, a formula, a table, and a picture; If the first document content includes the formula, converting the formula into a typesetting format; If the first document content includes the table, converting the table into a hypertext markup language format; If the first document content includes the picture, the picture is annotated based on the first model to generate annotation information.

5. The document splitting method according to claim 4, characterized in that: The target splitting strategy is the first splitting strategy; The step of splitting the document content of each type of document based on the target splitting strategy to determine the split content of the document of this type includes: defining the number of pages of the first category of documents to be processed each time; The content of the number of pages processed each time is used as the current range document content; the current range document content is part of the first document content; Processing the document content in the current range to determine the splitting prompt words; Inputting the splitting prompt words into a second model to obtain splitting segments; When the sum of the number of pages processed each time is less than the total number of pages of the first category of documents, the step of defining the number of pages processed each time for the first category of documents continues to be executed; When the sum of the number of pages processed each time is greater than or equal to the total number of pages of the first category of documents, the step of defining the number of pages processed each time for the first category of documents is stopped; Aggregate the multiple split fragments to obtain the split contents of the first category of documents.

6. The document splitting method according to claim 3, characterized in that: The second type of documents are non-editable documents; The target parsing strategy is the second parsing strategy; The step of parsing each type of document based on the target parsing strategy to determine the document content of the document includes: Converting the second-category document into a second picture; wherein the second-category document is one of each category of documents; Parsing the second image based on the third model to obtain second document content, where the second document content is at least one of a text block, a formula block, a table block, and a picture block; If the second document content includes the text block, converting the text block into text based on the first recognition model; If the second document content includes the formula block, saving a screenshot of the formula block; If the second document content includes the table block, saving a screenshot of the table block; If the second document content includes the picture block, a screenshot of the picture block is saved.

7. The document splitting method according to claim 6, characterized in that: The target splitting strategy is the second splitting strategy; The step of splitting the document content of each type of document based on the target splitting strategy to determine the split content of the document of this type includes: defining the number of pages of the second type of documents to be processed each time; The content of the number of pages processed each time is used as the current range document content; the current range document content is part of the second document content; Performing layout analysis on the current range document content to determine the page structure of the current range document content; Extracting the page structure to determine the extracted content of the document content in the current range; When the sum of the number of pages processed each time is less than the total number of pages of the second category of documents, the step of defining the number of pages processed each time for the second category of documents continues to be executed; When the sum of the number of pages processed each time is greater than or equal to the total number of pages of the first category of documents, the step of defining the number of pages processed each time for the second category of documents is stopped; Aggregating the multiple extracted contents to determine the extracted contents of the second category of documents; The extracted content of the second category of documents is split based on the fourth model to obtain the split content of the second category of documents.

8. A document splitting system, characterized in that: include: A document acquisition module is used to acquire multiple types of documents, wherein the multiple types of documents have different formats; A parsing module, used to parse each type of document based on a target parsing strategy to determine the document content of the document; A splitting module is used to split the document content of each type of document based on the target splitting strategy and determine the split content of the document of this type; The structured output module is used to structure the output of the split contents of each type of document.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images