Document segmentation method and device and electronic equipment

Abstract

The invention provides a document segmentation method and device and electronic equipment, relates to the technical field of artificial intelligence such as natural language processing, large language models, intelligent search and knowledge maps, and can be applied to scenes such as knowledge base construction, intelligent question and answer systems and document management systems. According to the specific implementation scheme, document analysis is conducted on a to-be-processed document, an analysis result of the to-be-processed document is obtained, and the analysis result comprises layout information and hierarchy information of a title; a title tree is constructed based on the hierarchical information of the titles, and the titles associated with nodes in the title tree are multi-level composite titles; and based on the layout information and the title tree, splitting and recombining the document region of the to-be-processed document according to a given maximum slice length to obtain document slices of the to-be-processed document. According to the method and the device, the document slicing precision and flexibility can be remarkably improved, and the document readability and the user experience are enhanced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence technology such as natural language processing, large language models, intelligent search, knowledge graphs, and more specifically to a document segmentation method, device, and electronic device that can be applied to knowledge base construction, intelligent question-and-answer systems, document management systems, and other scenarios. Background Art

[0002] Although the Large Language Model (LLM) performs well in natural language processing tasks, it is still necessary to slice documents in some cases. In related technologies, document segmentation methods usually only segment documents based on character length thresholds. However, the quality and relevance of document slices or fragments obtained in this way are not high, which affects the readability and information integrity of the document. Summary of the invention

[0003] The present disclosure provides a document segmentation method, device, electronic device and storage medium.

[0004] According to a first aspect of the present disclosure, a document segmentation method is provided, comprising:

[0005] Parsing the document to be processed to obtain a parsing result of the document to be processed, wherein the parsing result includes layout information and hierarchical information of the title;

[0006] Based on the hierarchical information of the title, a title tree is constructed, wherein the titles associated with the nodes in the title tree are multi-level composite titles;

[0007] Based on the layout information and the title tree, the document area of ​​the to-be-processed document is split and reorganized according to a given maximum slice length to obtain document slices of the to-be-processed document.

[0008] According to a second aspect of the present disclosure, there is provided a document segmentation device, comprising:

[0009] A parsing module, used to parse the document to be processed to obtain a parsing result of the document to be processed, wherein the parsing result includes layout information and hierarchical information of the title;

[0010] A construction module, used to construct a title tree based on the hierarchical information of the title, wherein the titles associated with the nodes in the title tree are multi-level composite titles;

[0011] The splitting and reorganizing module is used to split and reorganize the document area of ​​the document to be processed according to the given maximum slice length based on the layout information and the title tree to obtain the document slices of the document to be processed.

[0012] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0016] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0017] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0018] The technology disclosed in the present invention can significantly improve the accuracy and flexibility of document slicing, enhance the readability and user experience of documents, and has broad application prospects and commercial value.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 is a flowchart of a document segmentation method provided by an embodiment of the present disclosure;

[0022] Figure 2 is a flowchart of a document segmentation method provided by an embodiment of the present disclosure;

[0023] Figure 3 is a flowchart of a document segmentation method provided by an embodiment of the present disclosure;

[0024] Figure 4 is a flowchart of a document segmentation method provided by an embodiment of the present disclosure;

[0025] Figure 5A This is an example diagram of the first page of the product manual of a certain "XXXX Insurance" provided in an embodiment of the present disclosure;

[0026] Figure 5B This is an example diagram of the second page of the product manual of a certain "XXXX Insurance" provided in an embodiment of the present disclosure;

[0027] Figure 6 is a block diagram of a document segmentation device provided by an embodiment of the present disclosure;

[0028] Figure 7 is a block diagram of a document segmentation device provided by an embodiment of the present disclosure;

[0029] Figure 8 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] The terms used in one or more embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present disclosure. The singular forms of "a", "said" and "the" used in one or more embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present disclosure refers to and includes any or all possible combinations of one or more associated listed items.

[0032] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present disclosure, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present disclosure, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0033] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0034] It is worth noting that in the embodiments of the present disclosure, certain software, components, models, etc. that already exist in the industry may be mentioned. They should be considered as exemplary and their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0035] The embodiments of the present disclosure relate to artificial intelligence technology fields such as natural language processing, large language models, intelligent search, and knowledge graphs.

[0036] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0037] Natural Language Processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between humans and computers using natural language. It is a discipline that uses computer technology to analyze, understand and process natural language, that is, it uses computers as a powerful tool for language research, conducts quantitative research on language information with the support of computers, and provides language descriptions that can be used by humans and computers.

[0038] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important path to artificial intelligence.

[0039] Intelligent Retrieval is based on the relevance of documents and search terms, comprehensively examines indicators such as the importance of documents, and sorts the search results to provide higher search efficiency. The result sorting of intelligent search takes both relevance and importance into consideration. Relevance uses a weighted mixed index of each field, and the relevance analysis is more accurate. Importance refers to the evaluation of document quality through analysis of the authority of the document source and the citation relationship. This makes the result sorting more accurate and can put the documents most relevant to the user's wishes at the front, improving search efficiency.

[0040] Knowledge graph is a semantic network used to describe the relationship between entities. It is a representation method of semi-structured data, used to describe entities, attributes and the relationship between entities. Knowledge graphs are often used to build artificial intelligence applications such as intelligent search engines, recommendation systems, and question-answering systems. The core idea of ​​knowledge graphs is to convert information in the real world into graphs, where nodes represent entities and edges represent the relationship between entities. Knowledge graphs are not just a graphical knowledge base, they also contain semantic descriptions of entities and relationships that can be understood and processed by computers. The construction of knowledge graphs requires the extraction, fusion and reasoning of information, which is usually achieved using technologies such as natural language processing, image recognition, and machine learning.

[0041] Portable Document Format-Parser (PDF-Parser) refers to a software tool or library used to parse the content of PDF files. This tool can extract text, images, tables and other elements from PDF files and convert this information into other formats for further processing or analysis.

[0042] Optical Character Recognition (OCR): is a technology that converts text in an image into editable and searchable text. OCR technology can be applied to scans, photos, books, newspapers and other images containing printed or handwritten text.

[0043] Document Chunking: It is a text processing technique that divides a long document into a series of smaller, more manageable and processable fragments or chunks. Document chunking is applicable to natural language processing, information retrieval, text mining, and large-scale document processing.

[0044] The backtracking algorithm is actually a search and attempt process similar to enumeration. It mainly looks for solutions to problems during the search and attempt process. When it is found that the solution conditions are no longer met, it "backtracks" and returns to try other paths. The backtracking method is a selection search method that searches forward according to the selection conditions to achieve the goal. However, when exploring a certain step, it is found that the original choice is not optimal or cannot achieve the goal, so it goes back one step and chooses again. This technique of going back and forth if it fails is called the backtracking method, and the point of a certain state that meets the backtracking conditions is called the "backtracking point". The basic idea of ​​the backtracking algorithm is: go forward from one road, go forward if you can, go back if you can't, and try another road again.

[0045] Depth First Search is abbreviated as DFS, which stands for Depth First Search. In brief, the process is to search every possible branch path until it cannot go any deeper, and each node can only be visited once, with the goal of reaching the leaf node of the searched structure.

[0046] Preorder traversal is a method of traversing a binary tree, also known as pre-root traversal. In preorder traversal, the root node is visited first, then the left subtree is traversed, and finally the right subtree is traversed. When traversing the left and right subtrees, the preorder traversal method is still used.

[0047] Although Large Language Models (LLMs) perform well in natural language processing tasks, in some cases it is still necessary to slice documents and build a private knowledge base for the following reasons:

[0048] Performance and efficiency: LLM may encounter performance bottlenecks when processing long documents, especially very long documents. Slicing documents can improve the processing speed and response efficiency of the model.

[0049] Context Limitation: Many LLMs have a context length limit. Documents exceeding this limit cannot be processed at once, so the documents need to be sliced ​​to fit the model's input limit.

[0050] Accuracy: Document slicing can help the model locate and retrieve relevant information more accurately. For complex and detailed documents, slicing helps the model find more accurate answers in a specific context.

[0051] Resource Utilization: Building a private knowledge base can optimize resource utilization, especially when a large number of documents are involved or frequent queries are required. This can reduce the frequency of calls to the LLM and reduce costs.

[0052] Customization: Private knowledge bases allow for customization of knowledge within a specific domain or enterprise. This can better meet specific needs rather than relying on knowledge from a general model.

[0053] Controllability: Building a private knowledge base can better control the updating and maintenance of knowledge, ensuring the timeliness and accuracy of information.

[0054] In the related art, document slicing methods mainly include: plain text slicing method, regular expression-based slicing method and block processing method.

[0055] Plain text slicing method: mainly slices the document content into fixed lengths or natural paragraphs, without considering the structure and layout of the document. However, this method is simple and crude, ignoring the structural information of the document, such as titles, paragraphs, tables, etc., resulting in the loss of the context and logical relationship of the content, affecting the readability of the document and the integrity of the information.

[0056] Slicing method based on regular expressions: This method mainly slices documents by matching specific tags in documents (such as titles, paragraphs, tables, etc.) through regular expressions. However, regular expressions are difficult to handle complex document structures and changing formats, and are prone to matching errors or omissions, resulting in inaccurate slicing results. At the same time, regular expressions have high requirements on the content format of documents and have poor flexibility.

[0057] Block processing method: The document is divided into blocks by page or chapter, and each block is further sliced. However, although this method takes into account a certain document structure, the content after block division may still be too large to be effectively processed in the model. In addition, the content association between different blocks is difficult to handle, which affects the coherence and integrity of the content.

[0058] Based on this, the embodiment of the present disclosure provides a document segmentation method that can improve the accuracy of document parsing and the completeness of information. In addition, through a reasonable splitting and merging mechanism, the problem of too fragmented slices or lengthy content is avoided, the readability and coherence of the document are enhanced, and the user experience is optimized. Through layout analysis and title hierarchy construction, the output results are made more versatile and scalable, which can meet a variety of practical application scenarios, thereby enhancing the adaptability and flexibility of the product.

[0059] The document segmentation method, device and electronic device according to the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0060] It should be noted that the document segmentation method of the embodiment of the present disclosure may be executed by a document slicing device, which may be implemented by software and / or hardware, and may be configured in an electronic device. Exemplarily, the electronic device may include but is not limited to a terminal, a server, etc.

[0061] Figure 1 is a flow chart of the document segmentation method provided by the embodiment of the present disclosure. Figure 1 As shown, the document segmentation method may include but is not limited to the following steps.

[0062] In step 101, the document to be processed is parsed to obtain a parsing result of the document to be processed, wherein the parsing result includes layout information and hierarchical information of the title.

[0063] In some embodiments, the document to be processed may be, but is not limited to, a word document, a PDF document, a TXT document, a PPT document, a PNG document, etc.

[0064] In some embodiments, the document to be processed may be parsed by a document parsing module to obtain a parsing result of the document to be processed, wherein the parsing result may include but is not limited to layout information and hierarchical information of the title.

[0065] In some embodiments, the document parsing module may include a layout analysis model and an OCR recognition model. Exemplarily, the layout analysis model may retrieve and classify each block in the document to be processed, and the layout type may include but is not limited to one or more of the following: title (such as "title"); paragraph (such as "text"); picture title (such as "image_caption"); image (such as "image"); table title (such as "table_caption"); table (such as "table"); header (such as "header"); footer (such as "footer"); formula (such as "equation"); notes, reference sources (such as "reference"). The OCR recognition model can perform text recognition on blocks with a layout type of image. The OCR recognition model can also perform text recognition on documents such as PDF scans and images.

[0066] For example, the document to be processed can be divided into blocks first. For example, the document to be processed can be divided into blocks based on paragraphs. The blocks in the document to be processed can be retrieved and classified by the layout analysis model to obtain the layout information of each block and the hierarchical information of the title in the document to be processed. The OCR recognition model can be used to perform text recognition on the blocks with the layout category of image in the document to be processed. If the document to be processed is a PDF or image document, the OCR recognition model can be used to perform text recognition on the document to be processed to obtain the document content of the document to be processed.

[0067] In step 102, a title tree is constructed based on the hierarchical information of the title, and the titles associated with the nodes in the title tree may be multi-level composite titles.

[0068] In some embodiments, all titles in the document to be processed can be organized into a title tree according to the hierarchical relationship based on the hierarchical information of the title. In a possible implementation, the titles in the document to be processed can be organized into a title tree according to the hierarchical relationship using a depth-first search method based on the hierarchical information of the title; wherein the titles associated with the nodes in the title tree can be multi-level composite titles.

[0069] Exemplarily, all titles in the document to be processed are organized according to the hierarchical relationship using the depth-first search method to form a title tree. The main logic is as follows: (1) Compare the hierarchical relationship between the current node type (i.e., the level of the title) and the existing node types in the tree; (2) Determine the insertion point based on the comparison result to form the tree hierarchy; (3) Dynamically update the current parent node and depth to ensure the correct hierarchical relationship; (4) Recursively repeat the above judgment, traverse and insert all nodes, and thus obtain the title tree of the document to be processed.

[0070] In some embodiments, for each title in the document to be processed, the number, font size, whether it is bold, and the relative font size can be determined, so as to trace back the parent title of each title and complete the construction of the entire document title hierarchy, that is, each title is a multi-level composite title. In a possible implementation, the title tree can be pre-ordered to obtain the title hierarchy of the node in the title tree, and a backtracking algorithm is used to search for the parent node from the current node in the title tree step by step to generate the multi-level composite title of the node. Exemplarily, the title tree is pre-ordered to obtain the title hierarchy of each node in the title tree, and a backtracking algorithm is used to search for all parent nodes from the current node in the title tree step by step to generate the multi-level composite title of each node.

[0071] For example, taking the instruction document of a certain "XXXX insurance" product as an example, the title tree of the instruction manual is traversed in pre-order, and the title level of each node in the title tree is obtained as follows, where "Level" represents the level of the title. For example, "Level 0" means that the level of the title is level 0, which is the highest level, and is also the root node in the title tree. "Level 1" means that the level of the title is level 1, which is a child node of the root node "Level 0".

[0072] └──《XXXX Insurance》Product Manual (Level 0)

[0073] └──Risk Warning (Level 1)

[0074] ├──1. Basic features of the product (Level 2)

[0075] │├──(I) Coverage (Level 3)

[0076] │├──(II) Insurance period (Level 3)

[0077] │├──(III) Payment methods and payment periods (Level 3)

[0078] │├──(IV) Policy benefits and insurance liabilities (Level 3)

[0079] ││├──1.Basic insurance amount (Level 4)

[0080] ││├──2. First pension benefit payment date (Level 4)

[0081] ││└──3. Insurance liability (Level 4)

[0082] │├──(V) Exemption from liability (Level 3)

[0083] │└──(VI) Main investment strategies for participating insurance products (Level 3)

[0084] ├──2. Dividends and Dividend Distribution (Level 2)

[0085] ├──3. Cool-off period and policy cancellation (Level 2)

[0086] └──Four. Benefit Demonstration (Level 2)

[0087] Continuing with the above example, the backtracking algorithm is used to search all parent nodes from the current node in the title tree level by level to obtain the multi-level compound title of each node. For example, the multi-level compound title of each node in the above title tree is expressed as follows.

[0088] {

[0089] "0":"Product Manual of "XXXX Insurance"",

[0090] "1":"《XXXX Insurance》Product Manual\nRisk Warning",

[0091] "2":"《XXXX Insurance》Product Manual\nRisk Warning\n1. Basic Features of the Product",

[0092] "3":"《XXXX Insurance》Product Manual\nRisk Warning\nI. Basic Features of the Product\n(I) Scope of Insurance",

[0093] "4":"《XXXX Insurance》Product Manual\nRisk Warning\nI. Basic Features of the Product\n(II) Insurance Period",

[0094] "5":"《XXXX Insurance》Product Manual\nRisk Warning\nI. Basic Features of the Product\n(III) Payment Method and Payment Period",

[0095] "6":"《XXXX Insurance》Product Manual\nRisk Warning\nI. Basic Features of the Product\n(IV) Policy Benefits and Insurance Liabilities",

[0096] "7":"《XXXX Insurance》Product Manual\nRisk Warning\nI. Basic Features of the Product\n(IV) Policy Benefits and Insurance Liabilities\n1. Basic Insurance Amount",

[0097] "8":"《XXXX Insurance》Product Manual\nRisk Warning\nI. Basic Features of the Product\n(IV) Policy Benefits and Insurance Liabilities\n2. First Pension Benefit Collection Date",

[0098] "9":"《XXXX Insurance》Product Manual\nRisk Warning\nI. Basic Features of the Product\n(IV) Policy Benefits and Insurance Liability\n3. Insurance Liability",

[0099] "10":"《XXXX Insurance》Product Manual\nRisk Warning\nI. Basic Features of the Product\n(V) Liability Exemption",

[0100] "11":"《XXXX Insurance》Product Manual\nRisk Warning\nI. Basic Features of the Product\n(VI) Main Investment Strategies of Dividend-Type Insurance Products",

[0101] "12":"《XXXX Insurance》Product Manual\nRisk Warning\nII. Dividends and Dividend Distribution",

[0102] "13":"《XXXX Insurance》Product Manual\nRisk Warning\nIII. Cool-off Period and Policy Refund",

[0103] "14":"《XXXX Insurance》Product Manual\nRisk Warning\nIV. Benefit Demonstration"

[0104] }

[0105] It can be seen that the title associated with each node in the above title tree is a multi-level composite title, which means that richer title information can be obtained, changing a single title into a multi-level composite title, thereby improving the text retrieval effect.

[0106] In step 103, based on the layout information and the title tree, the document region of the document to be processed is split and reorganized according to a given maximum slice length to obtain document slices of the document to be processed.

[0107] In some embodiments, based on the layout information and the title tree, blocks with a layout type of paragraph, etc. can be split according to a given maximum slice length so that their text length does not exceed the maximum slice length, and multiple adjacent blocks with fewer words need to be merged so that the length of the merged slice does not exceed the maximum slice length, so that the content of the merged slice is as rich as possible.

[0108] In some embodiments, after the document to be processed is sliced ​​to obtain the document slices of the document to be processed, the document slices can be stored and retrieved in blocks, thereby improving the efficiency of large-scale document processing systems.

[0109] In the above embodiment, by constructing a multi-level composite title and splitting and reorganizing the document area, the structured information of the document can be retained, so that the content after slicing is more consistent with the logical hierarchy and semantic relationship of the original document, thereby improving the accuracy of document parsing and the integrity of information. Through a reasonable splitting and merging mechanism (i.e., based on the layout information and the title tree, the document area of ​​the document to be processed is split and reorganized according to a given maximum slice length), the problem of too fragmented slices or lengthy content can be avoided, the readability and coherence of the document are enhanced, and the user experience can be optimized. Considering the context length limit of the model, the maximum slice length can be customized, and long texts can be reasonably sliced ​​and optimized, so that the size of the content input each time is adapted to the processing capacity of the model, avoiding exceeding the context limit, and improving the response speed and accuracy of the model. In addition, for multi-column content and complex document structure, a flexible slicing scheme is provided to solve the defect that traditional methods cannot handle diversified content and adapt to a wider range of document types.

[0110] Figure 2 is a flow chart of the document segmentation method provided by the embodiment of the present disclosure. Figure 2 As shown, in Figure 1 On the basis shown, before splitting and reorganizing the document area of ​​the document to be processed, the document segmentation method may also include but is not limited to the following steps.

[0111] In step 201, it is determined based on layout information whether the document to be processed includes a table.

[0112] In some embodiments, before splitting and reorganizing the document area of ​​the document to be processed, it can be determined based on the layout information whether the document to be processed includes a table. That is, it is necessary to determine whether the document to be processed includes a block with a layout type of a table. If there is no block with a layout type of a table in the document to be processed, it can be determined that the document to be processed does not include a table; if there is no block with a layout type of a table in the document to be processed, it can be determined that the document to be processed includes a table.

[0113] In an embodiment of the present disclosure, it is determined based on layout information that the document to be processed includes a table. The number of tables included in the document to be processed may be one or more. For a single table, it can be determined whether the text content length of the table exceeds the maximum slice length. If the text content length of the table exceeds the maximum slice length, the following step 202 can be executed; if the text content length of the table does not exceed the maximum slice length, the following step 203 can be executed.

[0114] In step 202, if the length of the text content of the table exceeds the maximum slice length, the table is split to obtain multiple sub-tables of the table, and the cell formats of the multiple sub-tables are converted into a plain text format.

[0115] In some embodiments, when the length of the text content of a table exceeds the maximum slice length, the table can be split. Exemplarily, the table header head of the table can be determined, and the text content of the cells of the row is counted in units of rows. If it does not exceed the given maximum slice length, a row is added. For example, the text content of the cells of the first row of a table is counted, and the total length of the text content of the cells of the first row does not exceed the maximum slice length, then the text content of the cells of the second row of the table is continued to be counted. If the total length of the text content of the cells of the first row and the text content of the cells of the second row does not exceed the maximum slice length, then the text content of the cells of the third row of the table is continued to be counted until the given maximum slice length is reached, thereby obtaining multiple sub-tables of the table. The sub-tables obtained by splitting the table are formed by splicing the header head of the original table and the rows of the original table. For example, assuming that a table includes 4 rows, the total length of the cell text content of the first and second rows of the table can reach the maximum slice length, and the total length of the cell text content of the remaining rows of the table (i.e., the third and fourth rows) does not exceed the maximum slice length, then the table can be split into 2 sub-tables, one of which is composed of the table header + the first and second rows of the table, and the other sub-table is composed of the table header + the third and fourth rows of the table.

[0116] When multiple sub-tables of a table are obtained, the cell formats of the multiple sub-tables can be converted into a plain text format. Exemplarily, the cell formats of the multiple sub-tables can be converted into a plain text format such as markdown (a lightweight markup language that can be used to add formatting elements to a plain text document), html (HyperText Markup Language, Hypertext Markup Language), etc.

[0117] In step 203, if the length of the text content of the table does not exceed the maximum slice length, the cell format of the table is converted into a plain text format.

[0118] In some embodiments, when the length of the text content of the table does not exceed the maximum slice length, the cell format of the table can be converted into a plain text format such as markdown, html, etc.

[0119] In the above embodiment, the block with a layout type of table can be split so that its text is not too long, and the split table can be formatted so that the cell format of the table is converted into plain text such as markdown and html, which can be used directly in most editors, web pages and display tools. The text format is easy to transmit, avoiding the compatibility issues of special table formats and achieving cross-platform compatibility. The Markdown table syntax is simple, intuitive and clear, convenient for quick reading and direct viewing. The plain text format is easy to modify with any text editor, without relying on special tools, which is convenient for users to edit and modify the slice content on the front end, bringing higher usability, high readability and easy maintainability to the product. By representing the table in text form, lightweight archiving can be achieved, which takes up little space, is suitable for long-term storage and version management, and is easy to backup and retrieval. In addition, by standardizing the table information, it can be ensured that the output content is clear and readable, thereby further improving the quality of document parsing.

[0120] Figure 3 is a flow chart of the document segmentation method provided by the embodiment of the present disclosure. Figure 3 As shown in the above Figure 1 or Figure 2 On the basis of the illustrated embodiment, before splitting and reorganizing the document region of the document to be processed, the document segmentation method may further include but is not limited to the following steps.

[0121] In step 301, based on the layout information, the repeated information and / or additional information in the document to be processed is filtered.

[0122] In some embodiments, the repeated information and / or additional information may refer to information that is irrelevant to the actual content of the document to be processed.

[0123] In some embodiments, repeated information in the document to be processed can be filtered based on the layout information. Exemplarily, based on the layout information, the layout type of the document to be processed, such as headers and footers, contains repeated information and is filtered, wherein the repeated information may include, for example, page numbers, document titles, company logos, and other contents that are irrelevant to the actual content. Thus, by filtering the headers and footers, etc., which contain repeated information, noise data can be reduced and the focus can be placed on the core content, thereby avoiding interference from repeated information during statistics and analysis.

[0124] In some embodiments, additional information in the document to be processed can be filtered based on the layout information. Exemplarily, additional information in the document to be processed whose layout type is annotations, reference sources, etc. is filtered based on the layout information. That is to say, since additional information such as annotations, reference sources, etc. will not be the main content in most document parsing tasks, additional information in the document whose layout type is annotations, reference sources, etc. can be filtered. In this way, data conciseness can be improved, thereby avoiding useless information from occupying storage and computing resources.

[0125] In some embodiments, repeated information and additional information in the to-be-processed document can be filtered based on the layout information. For example, the to-be-processed document with layout types such as headers and footers containing repeated information and with layout types such as annotations and reference sources containing additional information can be filtered.

[0126] In the above embodiment, irrelevant content in the document to be processed can be filtered based on the layout information, which can further ensure that the output content is clear and readable, and improve the quality of document parsing.

[0127] Figure 4 is a flow chart of the document segmentation method provided by the embodiment of the present disclosure. Figure 4 As shown, the document segmentation method may include but is not limited to the following steps.

[0128] In step 401, the document to be processed is parsed to obtain a parsing result of the document to be processed, wherein the parsing result includes layout information and hierarchical information of the title.

[0129] Optionally, step 401 may be implemented in any of the implementations in the embodiments of the present disclosure, which is not limited in the embodiments of the present disclosure and will not be described in detail.

[0130] In step 402, a title tree is constructed based on the hierarchical information of the title, and the title associated with each node in the title tree is a multi-level composite title.

[0131] Optionally, step 402 may be implemented in any of the implementations in the embodiments of the present disclosure, which is not limited in the embodiments of the present disclosure and will not be described in detail.

[0132] In step 403, for the title of the leaf node in the title tree, the content under the title in the document area is aggregated with the title based on the layout information to obtain matching pair information of the title.

[0133] In some embodiments, for the title of each leaf node in the title tree, based on the layout information, the content under the title in the document area of ​​the document to be processed (including information such as text paragraphs and tables with layout types) can be aggregated under the title to obtain matching pair information of the title. The matching pair information can be represented by a "title-content" pair, for example.

[0134] Exemplarily, for the title of each leaf node in the title tree, the content under the title (layout type is text paragraph, table (converted to plain text), image (recognized as text), etc.) can be aggregated under the title in the form of a waterfall flow, so that the matching pair information of the title can be obtained.

[0135] For example, take the instruction document of a certain "XXXX insurance" product as an example. Figure 5A As shown, the first page of the instruction document is given, such as Figure 5B As shown in the figure, the second page of the instruction document is given. Figure 5A and Figure 5B By performing aggregation processing on the document area in the red box, we can get the matching pair information of the corresponding title, which can be represented by a json array as follows: [

[0137] {

[0138] "title":"3. Insurance liability",

[0139] "content":

[0140]

[0141]

[0142] It can be seen that the above "title-content" pair is the matching pair information of a title.

[0143] In some embodiments, before aggregating the content under the title in the document area with the title, it is determined that there is extra-long text content in the content under the title; and the extra-long text content under the title is split according to the maximum slice length. Exemplarily, before aggregating the content under the title in the document area with the title, the extra-long text content in the content under the title can be split. For example, for a single extra-long text block, the extra-long text block can be split according to the maximum slice length. For example, some paragraph text may exceed a given maximum slice length, and the text block needs to be split according to punctuation marks, wherein the punctuation marks may include but are not limited to ".", ", etc. For text blocks without punctuation marks, the string can be hard truncated and split according to the maximum slice length.

[0144] For example, the content under a certain title A is text paragraph 1, which is an overlong text block. According to the maximum slice length, the text paragraph 1 can be split into sub-paragraphs 1A and 1B. Then the pairing information of the title is expressed as follows:

[0145]

[0146] {

[0147] "title": Title A,

[0148] "content": [

[0150] {

[0151] "text":"Subparagraph 1B"

[0152] } ]

[0154] }

[0155] It can be seen that by splitting the extra-long text content under the title, it can be ensured that the text length in title-content does not exceed the maximum slice length.

[0156] In another implementation, when it is determined that the length of the content under a title exceeds the maximum slice length, a document summary or the like may be used to ensure that the length of the content under the title does not exceed the maximum slice length. For example, a large language model may be used to summarize the content under the title, generate a corresponding summary, thereby compressing the number of words, and replace the summary with the title, thereby ensuring that the length of the content under the title does not exceed the maximum slice length.

[0157] In step 404, the pairing information of the titles of the leaf nodes in the title tree is merged according to the maximum slice length to obtain a document slice of the document to be processed.

[0158] In some embodiments, the paired information of the titles of the leaf nodes in the title tree can be merged according to the maximum slice length, so that the length of the merged content does not exceed the maximum slice length, and the paired information of multiple titles with fewer words can be merged to make the merged slice content as rich as possible. For example, for the paired information of adjacent titles, such as adjacent "title-content" pairs, if the string lengths of the text contents of these adjacent "title-content" pairs are added together and still do not exceed the given maximum slice length, these small blocks need to be merged and finally spliced ​​into slices of plain text strings.

[0159] In some embodiments, the slice length calculation method is as follows: Method A, calculated by string length, where all characters are 1 in length. Method B, calculated by token: all text content of the slice is segmented by Chinese characters, words, numbers, and punctuation marks; Chinese characters are converted to tokens offline at a ratio of 0.625:1, and others are calculated at a ratio of 1:1.

[0160] In the above embodiment, the structured information of the document is retained through multi-level title construction and document area splitting and reorganization, so that the sliced ​​content is more consistent with the logical hierarchy and semantic relationship of the original document. Through a reasonable splitting and merging mechanism, the problem of too fragmented slices or lengthy content is avoided, and the readability and coherence of the document are enhanced.

[0161] It should be noted that, in some embodiments, the document segmentation method involved in the embodiments of the present disclosure can be applied to the knowledge base construction scenario. That is, when building an internal or public knowledge base of an enterprise, the document segmentation method involved in the embodiments of the present disclosure is used to segment the document, and the document is segmented into corresponding document fragments. The knowledge base is constructed based on these document fragments, which can improve the efficiency of document segmentation and organization, ensure the integrity, navigability and searchability of the knowledge base content, and make the content of the document segmented easier to retrieve and navigate, so that users can quickly locate the required information.

[0162] In some embodiments, the document segmentation method involved in the embodiments of the present disclosure can be applied to an intelligent question-answering system, that is, all documents in the intelligent question-answering system can be segmented using the document segmentation method. When obtaining a question input by a user, a search can be performed from the document slices (or document fragments) in the system based on the question to obtain a search fragment, and a corresponding answer can be generated based on the search fragment and the question input by the user. The processing capability of long documents in the intelligent question-answering system can be improved to ensure that the system can quickly and accurately retrieve and answer user questions.

[0163] In some embodiments, the document segmentation method involved in the embodiments of the present disclosure can be applied to a document management system. For example, after the document is segmented using the document segmentation method, visualization processing can be performed based on the obtained document fragments, such as improving the parsing and display effects of PDF documents in the document management system and improving the user's reading experience.

[0164] In some embodiments, the document segmentation method involved in the embodiments of the present disclosure can be applied to the construction of a large language model private knowledge base. For example, after the document is segmented using the document segmentation method, a large language model private knowledge base can be constructed based on the obtained document fragments. Thus, by pre-building a private knowledge base, the frequent calls to the large language model (LLM) are reduced, resource usage is optimized, and system operation costs are reduced.

[0165] Figure 6 is a block diagram of a document segmentation device provided by an embodiment of the present disclosure. Figure 6 As shown, the document segmentation device may include: a parsing module 601 , a building module 602 , and a splitting and reorganizing module 603 .

[0166] The parsing module 601 is used to parse the document to be processed to obtain the parsing result of the document to be processed, and the parsing result includes layout information and hierarchical information of the title.

[0167] The construction module 602 is used to construct a title tree based on the hierarchical information of the title, and the title associated with the node in the title tree is a multi-level composite title. In some embodiments, the construction module 602 is used to: based on the hierarchical information of the title, organize the titles in the to-be-processed document according to the hierarchical relationship using a depth-first search method to form a title tree; wherein a node in the title tree is associated with one title.

[0168] In some embodiments, the construction module 602 is used to: perform a pre-order traversal of the title tree to obtain the title levels of the nodes in the title tree; use a backtracking algorithm to search for the parent node from the current node step by step to generate a multi-level composite title for the node.

[0169] The splitting and reorganizing module 603 is used to split and reorganize the document area of ​​the document to be processed according to the given maximum slice length based on the layout information and the title tree to obtain the document slices of the document to be processed.

[0170] In some embodiments, the splitting and reorganization module 603 is used to: for the titles of leaf nodes in the title tree, aggregate the content under the title in the document area with the title based on the layout information to obtain matching pair information of the title; merge the pairing information of the titles of leaf nodes in the title tree according to the maximum slice length to obtain document slices of the document to be processed.

[0171] In some embodiments, the splitting and reassembling module 603 is further used to: when there is super-long text content under the title, split the super-long text content under the title according to the maximum slice length.

[0172] In some embodiments, Figure 7 As shown, the device may further include: a determination module 704 and a table processing module 705. The determination module 704 is used to determine that the document to be processed includes a table based on the layout information. The table processing module 705 is used to split the table when the text content length of the table exceeds the maximum slice length, obtain multiple sub-tables of the table, and convert the cell format of the multiple sub-tables into a plain text format. Alternatively, the table processing module 705 is also used to convert the cell format of the table into a plain text format when the text content length of the table does not exceed the maximum slice length.

[0173] In some embodiments, Figure 7 As shown, the device may further include: a filtering module 706. The filtering module 706 is used to filter the repeated information and / or additional information in the document to be processed based on the layout information; wherein the repeated information and / or additional information refers to information that is irrelevant to the actual content of the document to be processed. Figure 7 701 to 703 and Figure 6 601 to 603 have the same function and structure.

[0174] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0175] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0176] like Figure 8 , is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0177] like Figure 8As shown, the electronic device includes: one or more processors 801, memory 802, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 801 is taken as an example.

[0178] The memory 802 is a non-transient computer-readable storage medium provided by the present disclosure. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the document segmentation method provided by the present disclosure. The non-transient computer-readable storage medium of the present disclosure stores computer instructions, which are used to enable a computer to perform the document segmentation method provided by the present disclosure.

[0179] The memory 802 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the document segmentation method in the embodiment of the present disclosure (for example, the attached Figure 6 The processor 801 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 802, that is, the document segmentation method in the above method embodiment is implemented.

[0180] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 802 may optionally include a memory remotely arranged relative to the processor 801, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0181] The electronic device may further include: an input device 803 and an output device 804. The processor 801, the memory 802, the input device 803 and the output device 804 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0182] The input device 803 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 804 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0183] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0184] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0185] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0186] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0187] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0188] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0189] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A document segmentation method, comprising: Parsing the document to be processed to obtain a parsing result of the document to be processed, wherein the parsing result includes layout information and hierarchical information of the title; Based on the hierarchical information of the title, a title tree is constructed, wherein the titles associated with the nodes in the title tree are multi-level composite titles; Based on the layout information and the title tree, the document area of ​​the to-be-processed document is split and reorganized according to a given maximum slice length to obtain document slices of the to-be-processed document.

2. The method according to claim 1, before splitting and reorganizing the document area of ​​the to-be-processed document, the method further comprises: Determining, based on the layout information, that the document to be processed includes a table; If the length of the text content of the table exceeds the maximum slice length, the table is split to obtain multiple sub-tables of the table, and the cell formats of the multiple sub-tables are converted into plain text format; or, The text content length of the table does not exceed the maximum slice length, and the cell format of the table is converted into a plain text format.

3. The method according to claim 1, before splitting and reorganizing the document area of ​​the to-be-processed document, the method further comprises: Based on the layout information, the repeated information and / or additional information in the document to be processed is filtered; wherein the repeated information and / or additional information refers to information irrelevant to the actual content of the document to be processed.

4. The method of claim 1, wherein: The step of constructing a title tree based on the hierarchical information of the title includes: Based on the hierarchical information of the titles, the titles in the to-be-processed document are organized according to the hierarchical relationship using a depth-first search method to form the title tree; wherein a node in the title tree is associated with a title.

5. The method of claim 4, further comprising: Performing a pre-order traversal on the title tree to obtain the title levels of the nodes in the title tree; A backtracking algorithm is used to search for parent nodes from the current node level by level to generate a multi-level composite title for the node.

6. The method according to any one of claims 1 to 5, wherein: The step of splitting and reorganizing the document region of the to-be-processed document according to a given maximum slice length based on the layout information and the title tree to obtain document slices of the to-be-processed document includes: For the title of the leaf node in the title tree, based on the layout information, the content under the title in the document area is aggregated with the title to obtain matching pair information of the title; According to the maximum slice length, the pairing information of the titles of the leaf nodes in the title tree is merged to obtain the document slice of the document to be processed.

7. The method according to claim 6, before aggregating the content under the title in the document area with the title, the method further comprises: Determine that there is excessively long text content under the title; The extra-long text content under the title is split according to the maximum slice length.

8. A document segmentation device, comprising: A parsing module, used to parse the document to be processed to obtain a parsing result of the document to be processed, wherein the parsing result includes layout information and hierarchical information of the title; A construction module, used to construct a title tree based on the hierarchical information of the title, wherein the titles associated with the nodes in the title tree are multi-level composite titles; The splitting and reorganizing module is used to split and reorganize the document area of ​​the document to be processed according to the given maximum slice length based on the layout information and the title tree to obtain the document slices of the document to be processed.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, wherein: The computer program implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

Images