Paragraph restoration method and system for unstructured document

Abstract

The invention relates to a paragraph reduction method and system for an unstructured document. The method comprises the steps that the unstructured document to be processed is acquired, a text line information sequence is extracted, and a state maintaining mechanism is established; traversing a text line in a current processing page in the unstructured document, and executing paragraph judgment logic on the current text line to obtain a judgment result; setting of paragraph judgment logic is realized based on row head information, row tail information, row content information, row type information and context information; marking the current text line according to a judgment result, updating state information to a state maintenance mechanism, and circularly executing paragraph judgment logic and page processing loop logic until all text lines of the unstructured document are subjected to paragraph judgment to obtain a paragraph restoration result. According to the method, the non-structured document paragraph segmentation is realized by extracting the text line information sequence in the non-structured document and combining the text line multi-dimensional features and the context information, so that the problem of poor accuracy of a single geometric rule technology is effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and system for restoring paragraphs of unstructured documents. Background Art

[0002] With the rapid development of informatization, electronic documents have become the primary medium for information storage and transmission, and a large portion of these documents are unstructured. For unstructured documents, accurately restoring the original paragraph structure is difficult, whether through parsing tools or intelligent recognition methods. This can easily lead to problems such as misaligned or merged paragraphs in the parsed text. Accurate paragraph structure is crucial for subsequent natural language processing tasks and improving the user reading experience.

[0003] Currently, common paragraph restoration methods are primarily based on fixed geometric rules. By calculating the average line spacing of all text lines on a page and comparing this average spacing with the vertical spacing between the current and previous lines, the method determines whether the current line starts a new paragraph. For example, when traversing text lines, if the vertical spacing between the current and previous lines is significantly greater than the average line spacing (e.g., more than 1.5 times), the current line is considered to start a new paragraph. Alternatively, if the starting x-coordinate of the current line is significantly greater than the starting x-coordinate of the previous line, meaning there is a significant first-line indent, the current line is considered to start a new paragraph.

[0004] However, the above method, which relies solely on vertical spacing to determine paragraphs, suffers from inaccuracy. It's difficult to apply to documents with irregular line spacing or those with formulas or icons that cause localized spacing variations. Therefore, improving the accuracy of paragraph restoration in unstructured documents and improving its adaptability to document layouts is a pressing issue. Summary of the Invention

[0005] In view of this, the main purpose of the present invention is to provide a method and system for paragraph restoration of unstructured documents, which realizes paragraph segmentation of unstructured documents by extracting text line information sequences in unstructured documents and combining the multi-dimensional features and contextual information of text lines, thereby effectively solving the problem of poor accuracy of paragraph restoration of unstructured documents by single geometric rule technology.

[0006] In a first aspect, an embodiment of the present application provides a method for restoring paragraphs of an unstructured document, comprising the following steps: Obtaining an unstructured document to be processed, extracting a text line information sequence from the unstructured document, and establishing a corresponding state maintenance mechanism; Traversing text lines in a currently processed page of the unstructured document, and executing a preset paragraph judgment logic on the current text line to obtain a judgment result; wherein the paragraph judgment logic is set based on line start information, line end information, line content information, line type information, and context information; The current text line is marked according to the judgment result, and the corresponding state information is updated to the state maintenance mechanism. The paragraph judgment logic and page processing loop logic are executed cyclically until all text lines in the unstructured document have passed paragraph judgment and a paragraph restoration result is obtained.

[0007] In some embodiments, the state maintenance mechanism records the state information of the previous line of the current text line.

[0008] In some embodiments, before traversing the text lines in the current processing page of the unstructured document and executing preset paragraph judgment logic on the current text line to obtain a judgment result, the method further includes: A right page boundary feature of a currently processed page in the unstructured document is determined.

[0009] In some embodiments, the page right boundary feature is generated by adaptively analyzing text lines in the currently processed page based on a clustering algorithm.

[0010] In some embodiments, traversing text lines in a currently processed page in the unstructured document and executing preset paragraph judgment logic on the current text line to obtain a judgment result includes: Determining whether the current text line is the start of a new paragraph based on whether the current text line meets any one of the basic judgment conditions; Determine whether the current text line is the start of a new paragraph by checking whether there is a predefined mark at the beginning of the current text line; Based on the relationship between the current text line and the previous line, determining whether the current text line is the beginning of a new paragraph; When processing the first line of the currently processed page, it is determined whether the first line is the start of a new paragraph based on the position of the last line of the previous page.

[0011] In some embodiments, the basic judgment conditions include: the current text line is the first line of the unstructured document, the type of the current text line is different from the type of the previous line, and the current text line has a title tag.

[0012] In some embodiments, determining whether the current text line is the start of a new paragraph based on the relationship between the current text line and the previous text line includes: Obtaining state information of the previous row from the state maintenance mechanism; Determine whether the previous line reaches the right boundary according to the status information of the previous line; If the previous line does not reach the right boundary, then the current text line is determined to be the beginning of a new paragraph; If the previous line reaches the right boundary, whether the current text line is the start of a new paragraph is determined by the indentation relationship between the current text line and the start position of the previous line.

[0013] In some embodiments, when determining whether the current text line is the start of a new paragraph based on the indentation relationship between the current text line and the start position of the previous line, the method includes: If the indentation relationship is a significant right indentation, and there is no special symbol in the previous line, then the current text line is determined to be the beginning of a new paragraph; If the indentation relationship is a significant right indentation, and there is a special symbol in the previous line, then it is determined that the current text line and the previous line are the same paragraph; If the indentation relationship is a significant left indentation, determining that the current text line and the previous text line are in the same paragraph; If the indentation relationship is extreme left indentation, determining that the current text line is the beginning of a new paragraph; If the indentation relationship is basically aligned, it is determined that the current text line and the previous text line are in the same paragraph.

[0014] In some embodiments, when the paragraph judgment logic and page processing loop logic are cyclically executed, the following steps are included: If there is a next line for the current text line, executing the paragraph judgment logic for the next line; If there is no next line for the current text line, determining whether there is a next page for the current processing page; If the current processing page has a next page, then return to the step of determining the right boundary feature of the page; If the current processing page does not have a next page, the paragraph restoration result is output.

[0015] In a second aspect, an embodiment of the present application provides a paragraph restoration system for unstructured documents, comprising: a preprocessing module configured to obtain an unstructured document to be processed, extract a text line information sequence from the unstructured document, and establish a corresponding state maintenance mechanism; a judgment module configured to traverse text lines in a currently processed page of the unstructured document and execute a preset paragraph judgment logic on the current text line to obtain a judgment result; wherein the paragraph judgment logic is set based on line start information, line end information, line content information, line type information, and context information; The result generation module is configured to mark the current text line according to the judgment result, update the corresponding status information to the state maintenance mechanism, and cyclically execute the paragraph judgment logic and page processing loop logic until all text lines in the unstructured document have undergone paragraph judgment to obtain a paragraph restoration result.

[0016] Technical effects of the present invention: This application extracts text line information sequences from unstructured documents and combines the multi-dimensional features and contextual information of the text lines to achieve paragraph segmentation of unstructured documents. Specifically, the application includes: obtaining an unstructured document to be processed, extracting a text line information sequence from the unstructured document, and establishing a corresponding state maintenance mechanism; traversing the text lines in the current processing page of the unstructured document and executing a preset paragraph judgment logic on the current text line to obtain a judgment result; the paragraph judgment logic is set based on line start information, line end information, line content information, line type information, and contextual information; marking the current text line according to the judgment result, and updating the corresponding state information to the state maintenance mechanism, cyclically executing the paragraph judgment logic and page processing loop logic until all text lines in the unstructured document have undergone paragraph judgment to obtain a paragraph restoration result. Compared with the method that relies solely on a single combination rule, the application can effectively improve the accuracy of paragraph restoration. Moreover, compared with the method that uses a fixed threshold judgment method, the application has better processing capabilities and stronger robustness for unstructured documents of different formats.

[0017] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0019] Figure 1 A flow chart showing a method for restoring paragraphs of an unstructured document according to an embodiment of the present application is shown; Figure 2 A schematic flow chart illustrating a method for restoring paragraphs of an unstructured document according to another embodiment of the present application is shown; Figure 3 The figure shows a schematic block diagram of the structure of a paragraph restoration system for unstructured documents according to one embodiment of the present application. DETAILED DESCRIPTION

[0020] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0021] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0022] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0023] Unstructured documents refer to documents that lack clear, predefined structural information, such as scanned PDF documents. Limited by the above characteristics, the recognition and parsing of paragraph structures in unstructured documents usually have accuracy issues. Common paragraph restoration methods based on fixed geometric rules calculate the vertical spacing of text lines, which only have good effects in unstructured documents with regular line spacing. For documents with irregular line spacing or with formulas or icons that cause local spacing changes, judgment based solely on vertical spacing is prone to errors and has poor adaptability to changes in document layout. Therefore, providing a method that can improve the accuracy of paragraph restoration in unstructured documents and robustly restore the paragraph structure of unstructured documents is a key research issue in the industry.

[0024] like Figure 1 As shown, in a first aspect, an embodiment of the present application provides a method for restoring paragraphs of an unstructured document, comprising the following steps: S100, obtaining an unstructured document to be processed, extracting a text line information sequence from the unstructured document, and establishing a corresponding state maintenance mechanism; In this step, for the unstructured document to be processed, a text line information sequence is extracted from the document, wherein the text line information includes text content, boundary coordinates, line type, and an identifier of whether it is a title. It should be noted that paragraph restoration of the unstructured document is achieved by processing the text line information sequence extracted from the document. Specifically, after the text line information is extracted, the context state is initialized and a state maintenance mechanism is established to record and transmit key state information of the text line during the processing process. After the paragraph judgment is subsequently performed on the text line, the marked text line information is promptly updated to the state maintenance mechanism for subsequent text lines to serve as a reference for judgment.

[0025] S200, traversing text lines in a currently processed page of the unstructured document, and executing preset paragraph judgment logic on the current text line to obtain a judgment result; wherein the paragraph judgment logic is set based on line start information, line end information, line content information, line type information, and context information; In this step, each text line on the current page of the unstructured document is traversed and the preset paragraph judgment logic is executed on the current text line to determine whether the current text line should start a new paragraph. It should be noted that the paragraph judgment logic in this step combines the multi-dimensional characteristics of the text line and contextual information to simulate the logic of human judgment of paragraph segmentation when reading, thereby improving the accuracy of paragraph restoration in unstructured documents.

[0026] Specifically, paragraph segmentation is performed by comprehensively considering the line start, end, content, and type information of a text line, combined with contextual information. Line start information specifically refers to the relative position of the line start, including comparisons of various indent relationships; line end information includes features of the right page margin; line content information includes special list markers; and line type information includes text, images, tables, and more. This multi-dimensional feature set, combined with text line contextual information, can significantly improve the accuracy of paragraph segmentation in unstructured documents compared to single geometric rules.

[0027] S300. Mark the current text line according to the judgment result, and update the corresponding state information to the state maintenance mechanism. Circularly execute the paragraph judgment logic and page processing loop logic until all text lines in the unstructured document have passed the paragraph judgment and obtained the paragraph restoration result.

[0028] In this step, the text line currently being processed is marked according to the judgment result of the current text line obtained in step S200. If the judgment result shows that the current text line is a new paragraph, the current text line is marked as a new paragraph; if the judgment result shows that the current text line and the previous line still belong to the same paragraph, the current text line is marked as a non-new paragraph. At the same time, the information of the current line is updated to the state maintenance mechanism, and the paragraph judgment logic is executed cyclically, that is, the processing of the next line is continued, so that when the paragraph judgment processing is performed on the next line, the paragraph judgment can be realized based on the updated information. It should be noted that in addition to cyclically executing the paragraph judgment logic for the text lines in the page, this step also cyclically executes the page processing loop logic, thereby ensuring that all text lines in the unstructured document can be processed, thereby obtaining the final paragraph restoration result.

[0029] It can be seen from this that this embodiment extracts the text line information sequence in the unstructured document and combines the multi-dimensional features and contextual information of the text line to achieve paragraph segmentation for the unstructured document, that is, automatically determines which text lines should start a new paragraph, thereby restoring the original paragraph structure of the document. Compared with the method of relying solely on a single combination rule, it can effectively improve the accuracy of paragraph restoration. Moreover, compared with the method of using a fixed threshold judgment, it has better processing capabilities and stronger robustness for unstructured documents of different formats.

[0030] In some embodiments, the state maintenance mechanism records the state information of the previous line of the current text line.

[0031] In this embodiment, the state maintenance mechanism records and transmits key contextual information about the previous text line during processing, including a flag indicating whether the previous line reached the right edge of the page and whether the previous line contained special start-of-line markers, such as list symbols. By utilizing a state machine to store and transmit key status information about the previous line, this embodiment enables context-based judgment of the current line, thereby improving the accuracy of paragraph segmentation.

[0032] In some embodiments, before traversing text lines in a currently processed page in an unstructured document and executing preset paragraph judgment logic on the current text line and obtaining a judgment result, the method further includes: Determine the right page boundary characteristics of a currently processed page in an unstructured document.

[0033] In this embodiment, before executing the paragraph judgment logic for the text line to be processed, it is necessary to first generate a "reaching the right boundary" flag for each line in the page currently being processed, so that in subsequent processing, by judging whether the previous line of the currently being processed line has reached the right boundary, it can be judged whether the currently being processed line is to be determined as the start of a new paragraph. For example, if the previous line has not reached the right boundary, then the current line is the starting line of a new paragraph. Even if there is no obvious blank line separating the paragraphs, the start of the new paragraph can be judged based on the right boundary feature. That is, before the line processing and paragraph judgment loop, it is necessary to first determine the page right boundary feature, and use the judgment result of whether the previous line has reached the right boundary as the basis for judging the currently being processed text line. This does not rely on the text features of a specific language, and is therefore applicable to documents in various languages.

[0034] In some embodiments, the page right boundary feature is generated by adaptively analyzing text lines in the currently processed page based on a clustering algorithm.

[0035] In this embodiment, a clustering algorithm is used to adaptively analyze text lines on the currently processed page to determine whether they can be functionally considered to have reached the right boundary of the main text area of ​​the page. A right boundary reaching marker is generated for each line on the page. The clustering algorithm is preferably DBSCAN, which does not require a specific number of clusters and adapts to changes in text density to identify the right boundary that truly represents the main text area, providing a basis for subsequent paragraph recognition.

[0036] In some embodiments, traversing text lines in a currently processed page of an unstructured document and executing a preset paragraph judgment logic on the current text line to obtain a judgment result includes: Determine whether the current text line is the start of a new paragraph based on whether the current text line meets any of the basic judgment conditions; By checking whether there is a predefined marker at the beginning of the current text line, it is determined whether the current text line is the beginning of a new paragraph; Based on the relationship between the current text line and the previous line, determine whether the current text line is the beginning of a new paragraph; When processing the first line of the current page, determine whether the first line is the start of a new paragraph based on the position of the last line of the previous page.

[0037] In this embodiment, when performing the line processing and paragraph determination loop, each text line in the current page is traversed, and the following logic is executed on the currently processed text line to determine whether it should start a new paragraph.

[0038] Specifically, the current text line is first judged based on basic conditions, where the basic judgment conditions are associated with the essential characteristics of the document. When a text line meets any of the basic conditions, the text line can be determined to be the starting line of a new paragraph. In other words, the basic judgment conditions can quickly and universally identify the functional attributes of a text line. Secondly, if the current text line does not meet the basic judgment conditions, a further judgment is made by checking the content characteristics of the beginning of the current text line. Specifically, the beginning of the text content of the current text line is checked to see whether it contains predefined special marks, such as list items or special introductory symbols such as "a)", "1.", "——". If so, the current text line is determined to be the starting line of a new paragraph. Furthermore, if the content characteristics of the beginning of the line do not contain such list items or special introductory symbols, a comprehensive judgment based on the context is continued. Specifically, the judgment is made based on the state information of the previous line obtained from the state maintenance mechanism and the relative indentation relationship between the current line and the previous line. That is, when executing the paragraph judgment logic, the text line is judged in the order of basic judgment conditions, line beginning content characteristics, and comprehensive context judgment, thereby improving the accuracy of paragraph segmentation. Furthermore, by explicitly detecting special indentation patterns like "a)" and "1)" and integrating them with the indentation relationships of subsequent lines, this method can better handle list items and their continuation lines, avoiding incorrect segmentation. Compared to existing methods that use simple first-line indentation rules to segment paragraphs in unstructured documents, this embodiment prevents incorrect paragraph segmentation or list item merging when continuation lines within a paragraph (such as hanging indents) or when the subsequent content of the list item "a)" may be left-indented or aligned. It also offers better handling of complex situations such as different layouts, different indentation rules, and different list structures.

[0039] In addition, when processing the first line of text on a page, the first line of the current page is determined to be the start of a new paragraph by considering whether the last line of the previous page is positioned relatively high. If the last line of the previous page is positioned relatively high, the first line of the current page is determined to be the start of a new paragraph. In other words, the paragraph determination logic of this embodiment also takes into account cross-page logic, enabling it to handle paragraphs that cross page boundaries, enhancing robustness.

[0040] In some embodiments, the basic judgment conditions include: the current text line is the first line of the unstructured document, the type of the current text line is different from the type of the previous line, and the current text line has a title tag.

[0041] In this embodiment, a basic conditional judgment is first performed on the currently processed text line, specifically including whether the line is the first line of the entire unstructured document, the type of the line is different from the type of the previous line, and the line itself is clearly marked as a title. In other words, based on the starting point, changes, and display markup of the document structure, text lines are universally identified. When any of the above conditions is met, the current text line is determined to be the starting line of a new paragraph. As can be seen from this, when setting basic judgment conditions, the basic logic covering the document structure is used to quickly judge the text line, thereby improving analysis and processing efficiency.

[0042] In some embodiments, determining whether the current text line is the start of a new paragraph based on the relationship between the current text line and the previous text line includes: Get the status information of the previous line from the state maintenance mechanism; Determine whether the previous line has reached the right boundary based on the status information of the previous line; If the previous line does not reach the right boundary, the current text line is determined to be the beginning of a new paragraph; If the previous line reaches the right boundary, the indentation relationship between the current text line and the starting position of the previous line is used to determine whether the current text line is the beginning of a new paragraph.

[0043] In this embodiment, when making a comprehensive judgment based on context information, it is first necessary to obtain the state information of the previous line from the state maintenance mechanism, and make a judgment based on the relative indentation relationship between the current line and the previous line. Specifically, it is necessary to first determine whether the previous line has reached the right boundary. If the previous line of the currently processed text line has not reached the right boundary, the currently processed text line is directly determined to be the beginning of a new paragraph. If the previous line of the currently processed text line has reached the right boundary, it is necessary to further judge based on the relative indentation relationship between the current line and the previous line. It should be noted that when making a comprehensive judgment on the text line based on context information, using a clustering algorithm to judge the right boundary is more adaptable than a fixed threshold. By combining context information and multi-dimensional features, the logic of people judging paragraph segmentation when reading is simulated, so that the paragraph segmentation results are more accurate.

[0044] In some embodiments, when determining whether the current text line is the start of a new paragraph based on the indentation relationship between the current text line and the start position of the previous line, the method includes: If the indentation relationship is significant right indentation and there is no special symbol in the previous line, the current text line is determined to be the beginning of a new paragraph; If the indentation relationship is significant right indentation, and there is a special symbol in the previous line, then the current text line and the previous line are determined to be the same paragraph; If the indentation relationship is significant left indentation, the current text line and the previous line are determined to be in the same paragraph; If the indentation relationship is extreme left indentation, the current text line is determined to be the beginning of a new paragraph; If the indentation relationship is basic alignment, the current text line and the previous line are determined to be in the same paragraph.

[0045] In this embodiment, when determining the current text line, if it is determined that the previous line of the current text line reaches the right boundary, it is necessary to analyze the relationship between the starting positions of the current line and the previous line, that is, to make a judgment based on the relative indentation relationship between the current line and the previous line. The relative indentation relationship includes significant right indentation, significant left indentation, extreme left indentation, and basic alignment.

[0046] Specifically, if the indentation relationship is a significant right indentation, and at the same time there is no special symbol in the previous line, that is, the previous line does not contain a special line start mark, then it can be determined that the current text line is the starting line of a new paragraph; if the indentation relationship is a significant right indentation, but there is a special symbol in the previous line, then the current line and the previous line are determined to be the same paragraph, that is, not the starting line of a new paragraph; if the indentation relationship is an extreme left indentation, it indicates that the overall indentation level of the document may change, and the current text line is determined to be the starting line of a new paragraph; if the indentation relationship is a significant left indentation, that is, not an extreme left indentation, it usually indicates that the text in the same paragraph is wrapped, and the current text line is determined to be the same paragraph as the previous line; if the indentation relationship is basic alignment, it indicates that it is a natural continuation of the same paragraph, and the current text line is determined to be the same paragraph as the previous line. That is, this embodiment comprehensively considers different rule indentations, so that this method can have better processing capabilities for complex situations and is more robust.

[0047] In some embodiments, when the paragraph judgment logic and the page processing loop logic are cyclically executed, the following steps are included: If there is a next line for the current text line, then the paragraph judgment logic is executed for the next line; If there is no next line for the current text line, determine whether there is a next page for the current processing page; If the current processing page has a next page, then return to the step of determining the right boundary feature of the page; If there is no next page for the current processing page, the paragraph restoration result will be output.

[0048] In this embodiment, in order to fully process all text lines in the unstructured document, it is necessary to cyclically execute the paragraph judgment logic and the page processing loop logic, which will be described in detail below.

[0049] Specifically, such as Figure 2 As shown in Figure 1, after extracting a sequence of text line information, the context state is initialized and a state maintenance mechanism is established to record and transmit key context information of the previous text line relative to the currently processed text line. Subsequently, a page processing loop is established to process each page in the document in turn.

[0050] Furthermore, when actually processing pages in unstructured documents, a clustering algorithm is used to adaptively analyze the features of text lines that can be functionally regarded as reaching the right boundary for the currently processed page, and a "reaching the right boundary" flag is generated for each line in the page.

[0051] Furthermore, a line processing and paragraph determination loop is executed, that is, each text line in the current page is traversed. For the currently processed text line, the following logic is executed to determine whether it should start a new paragraph, including: Basic condition judgment: The line is the first line of the entire document, the line type is different from the previous line type, and the line itself is clearly marked as a title. If any of the above three conditions are met, the line is determined to be the beginning of a new paragraph. Otherwise, the content feature judgment at the beginning of the line is continued; Line start feature judgment: Check whether the beginning of the line of text contains predefined special marks, such as list items such as "a)" or special introductory symbols. If so, the line is judged to be the beginning of a new paragraph. Otherwise, the comprehensive context judgment is continued; Comprehensive judgment based on the context: Based on the status information of the previous line obtained from the state maintenance mechanism, combined with the relative indentation relationship between the current line and the previous line, the judgment is made: If the previous line does not reach the right boundary, the line is determined to be the beginning of a new paragraph; If the previous line has reached the right boundary, the starting position relationship between the current line and the previous line is analyzed, and whether the current line should be determined as the start of a new paragraph is determined based on whether there is significant right indentation, significant left indentation, extreme left indentation, basic alignment, etc. Cross-page logic considerations: When processing the first line of a page, determine whether to consider the line as a new paragraph based on whether the last line of the previous page is located at the top.

[0052] Furthermore, based on the above determination result, the current line is marked as a new paragraph or not, and the state information of the current line is updated to the state maintenance mechanism. Then, it is determined whether there is a next line. If so, the process returns to the line processing and paragraph processing loop step and executes the paragraph determination logic again to process the next line. If not, it further determines whether there is a next page. If there is a next page, it returns to the page processing loop step. If not, the processing result including the paragraph mark is output.

[0053] In summary, this application addresses the problems of low accuracy in paragraph restoration of unstructured documents in the prior art, poor adaptability to document layout changes, inability to effectively handle lists and special indentations, and weak cross-page processing capabilities. A paragraph restoration method based on multi-feature fusion and state maintenance is proposed. By processing a sequence of text line information extracted from a document and combining text line context information and multi-dimensional features, the method simulates the logic of how people judge paragraph segmentation when reading. That is, by fusing the end-of-line position, the beginning-of-line position, content features, and contextual state, the accuracy of paragraph segmentation is significantly improved compared to a single geometric rule. Furthermore, the method has better processing capabilities for complex situations such as different layouts, different rule indentations, list structures, and cross-page paragraphs. Using a clustering algorithm to determine the right boundary is more adaptable than a fixed threshold. By explicitly detecting special indentation patterns such as "a)" and "1)" and combining the indentation relationship of subsequent lines, list items and their continuation lines can be better processed to avoid incorrect segmentation.

[0054] like Figure 3 As shown, in a second aspect, an embodiment of the present application provides a paragraph restoration system 10 for an unstructured document, which is used to execute the above method, including: The pre-processing module 100 is configured to obtain an unstructured document to be processed, extract a text line information sequence from the unstructured document, and establish a corresponding state maintenance mechanism; The judgment module 200 is configured to traverse the text lines in the current processing page of the unstructured document and execute a preset paragraph judgment logic on the current text line to obtain a judgment result; wherein the paragraph judgment logic is set based on line start information, line end information, line content information, line type information and context information; The result generation module 300 is configured to mark the current text line according to the judgment result, update the corresponding status information to the state maintenance mechanism, and cyclically execute the paragraph judgment logic and page processing loop logic until all text lines in the unstructured document have passed the paragraph judgment and obtained the paragraph restoration result.

[0055] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor, a storage medium, and a computer program. The computer program is stored in the storage medium. When the computer program is executed by the processor, it implements any of the above-mentioned methods for restoring paragraphs of unstructured documents. For specific examples, please refer to the examples described in the above-mentioned embodiments and optional implementation methods, and this embodiment will not be repeated here.

[0056] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0057] Unless otherwise defined, the technical terms or scientific terms involved in this application should have the usual meanings understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "the" and similar words involved in this application do not indicate quantity restrictions and can indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof involved in this application are intended to cover non-exclusive inclusions. The words "connect", "connected", "coupled" and similar words involved in this application are not limited to physical or mechanical connections, but include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more, and "and / or" describes the association relationship of associated objects, indicating that three relationships can exist. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. The terms "first", "second", "third" and the like involved in this application are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0058] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make several modifications or improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for restoring paragraphs of unstructured documents, characterized in that: The steps include: Obtaining an unstructured document to be processed, extracting a text line information sequence from the unstructured document, and establishing a corresponding state maintenance mechanism; Traversing text lines in a currently processed page of the unstructured document, and executing a preset paragraph judgment logic on the current text line to obtain a judgment result; wherein the paragraph judgment logic is set based on line start information, line end information, line content information, line type information, and context information; The current text line is marked according to the judgment result, and the corresponding state information is updated to the state maintenance mechanism. The paragraph judgment logic and page processing loop logic are executed cyclically until all text lines in the unstructured document have passed paragraph judgment and a paragraph restoration result is obtained.

2. The method for restoring paragraphs of unstructured documents according to claim 1, characterized in that: The state maintenance mechanism records the state information of the previous line of the current text line.

3. The method for restoring paragraphs of unstructured documents according to claim 1, characterized in that: Before traversing the text lines in the current processing page in the unstructured document and executing the preset paragraph judgment logic on the current text line to obtain the judgment result, the method further includes: A right page boundary feature of a currently processed page in the unstructured document is determined.

4. The method for restoring paragraphs of an unstructured document according to claim 3, wherein: The page right boundary feature is generated by adaptively analyzing the text lines in the currently processed page based on a clustering algorithm.

5. The method for restoring paragraphs of an unstructured document according to any one of claims 1 to 4, characterized in that: The traversing of the text lines in the current processing page in the unstructured document and executing the preset paragraph judgment logic on the current text line to obtain a judgment result includes: Determining whether the current text line is the start of a new paragraph based on whether the current text line meets any one of the basic judgment conditions; Determine whether the current text line is the start of a new paragraph by checking whether there is a predefined mark at the beginning of the current text line; Based on the relationship between the current text line and the previous line, determining whether the current text line is the beginning of a new paragraph; When processing the first line of the currently processed page, it is determined whether the first line is the start of a new paragraph based on the position of the last line of the previous page.

6. The method for restoring paragraphs of an unstructured document according to claim 5, characterized in that: The basic judgment conditions include: the current text line is the first line of the unstructured document, the type of the current text line is different from the type of the previous line, and the current text line has a title identifier.

7. The method for restoring paragraphs of an unstructured document according to claim 5, characterized in that: The determining whether the current text line is the start of a new paragraph based on the relationship between the current text line and the previous text line includes: Obtaining state information of the previous row from the state maintenance mechanism; Determine whether the previous line reaches the right boundary according to the status information of the previous line; If the previous line does not reach the right boundary, then the current text line is determined to be the beginning of a new paragraph; If the previous line reaches the right boundary, whether the current text line is the start of a new paragraph is determined by the indentation relationship between the current text line and the start position of the previous line.

8. The method for restoring paragraphs of an unstructured document according to claim 7, characterized in that: When determining whether the current text line is the start of a new paragraph based on the indentation relationship between the current text line and the start position of the previous line, the method includes: If the indentation relationship is a significant right indentation, and there is no special symbol in the previous line, then the current text line is determined to be the beginning of a new paragraph; If the indentation relationship is a significant right indentation, and there is a special symbol in the previous line, then it is determined that the current text line and the previous line are the same paragraph; If the indentation relationship is a significant left indentation, determining that the current text line and the previous text line are in the same paragraph; If the indentation relationship is extreme left indentation, determining that the current text line is the beginning of a new paragraph; If the indentation relationship is basically aligned, it is determined that the current text line and the previous text line are in the same paragraph.

9. The method for restoring paragraphs of an unstructured document according to claim 1, wherein: When the paragraph judgment logic and page processing loop logic are executed cyclically, it includes: If there is a next line for the current text line, executing the paragraph judgment logic for the next line; If there is no next line for the current text line, determining whether there is a next page for the current processing page; If the current processing page has a next page, then return to the step of determining the right boundary feature of the page; If the current processing page does not have a next page, the paragraph restoration result is output.

10. A paragraph restoration system for unstructured documents, characterized in that: include: a preprocessing module configured to obtain an unstructured document to be processed, extract a text line information sequence from the unstructured document, and establish a corresponding state maintenance mechanism; a judgment module configured to traverse text lines in a currently processed page of the unstructured document and execute a preset paragraph judgment logic on the current text line to obtain a judgment result; wherein the paragraph judgment logic is set based on line start information, line end information, line content information, line type information, and context information; The result generation module is configured to mark the current text line according to the judgment result, update the corresponding status information to the state maintenance mechanism, and cyclically execute the paragraph judgment logic and page processing loop logic until all text lines in the unstructured document have undergone paragraph judgment to obtain a paragraph restoration result.

Images