Semantic segmentation method and apparatus based on title recognition

Abstract

The application provides a semantic segmentation method and device based on title recognition, which comprises the following steps: dividing a document to be segmented into a plurality of first-level semantic blocks according to a plurality of first-level titles of the document to be segmented; determining a search starting line and a search ending line of a first-level semantic block to be processed; starting from the search starting line, searching line by line in a top-down order; performing boundary search for several rounds to obtain a lower boundary of the first-level semantic block to be processed; the boundary search comprises the following steps: if the document to be segmented is a target type document, determining whether a current text line matches an ending mode corresponding to the target type document; if the current text line matches the ending mode, determining an upper text line of the current text line as the lower boundary; if the document to be segmented is not the target type document, determining whether the following conditions are met: the length of the current text line is greater than a first threshold value, and the length of a text line between the current text line and the search starting line is less than a second threshold value; if the conditions are met, determining a next text line of the current text line as the lower boundary.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a semantic segmentation method and apparatus based on title recognition. Background Technology

[0002] With the widespread use of digital documents, automatically understanding their semantic structure has become a significant challenge. Document semantic segmentation aims to divide documents into meaningful semantic units (such as chapters and paragraphs) to support subsequent content analysis, retrieval, and knowledge extraction. However, it is difficult to fix the style and regular expressions of documents written in different formats or at different times, making it difficult to adapt to changes. Deep learning-based solutions often require tens of thousands of labeled samples and are most prone to gaps in support at hierarchical nesting. Even more problematic is the tendency to miss the main text by judging fixed lengths or special symbols at the end of the document.

[0003] It is evident that existing document semantic segmentation methods have several problems. For example, rule-based methods rely on predefined patterns, making it difficult to adapt to diverse document formats and heading styles, resulting in low segmentation accuracy. Furthermore, machine learning-based methods require large amounts of labeled data and tend to overlook hierarchical relationships between headings when processing complex document structures, leading to segmentation errors. Additionally, existing methods often rely on fixed delimiters or length thresholds to determine the end of a document, failing to dynamically adapt to changes in document content, especially when the last heading is followed by a large amount of text, easily resulting in missegmentation.

[0004] Therefore, a method is needed that can accurately identify heading levels and dynamically determine segmentation boundaries to improve the robustness and accuracy of document semantic segmentation. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a semantic segmentation method and apparatus based on title recognition to eliminate or improve one or more defects existing in the prior art.

[0006] According to the first aspect, a semantic segmentation method based on title recognition is provided, comprising: determining the various levels of headings in the document to be segmented, and dividing the document to be segmented into multiple first-level semantic blocks according to the multiple first-level headings; for the first-level semantic block to be processed among the multiple first-level semantic blocks, determining the search start line and search end line of the first-level semantic block to be processed, and performing a series of boundary searches from the search start line in a bottom-up order to obtain the lower boundary of the first-level semantic block to be processed, wherein the search start line is located below the search end line, and each round of boundary search includes the following steps: when the document to be segmented is a target type document, determining whether the current text line is a target type document. If the text line matches the ending pattern corresponding to the target document type, the previous text line is determined as the lower boundary of the first-level semantic block to be processed. If the document to be segmented is not a target document type, it is determined whether the length of the current text line is greater than a preset first threshold and the length of the text line between the current text line and the search start line is less than a preset second threshold. If the conditions are met, the next text line is determined as the lower boundary of the first-level semantic block to be processed. If the lower boundary of the first-level semantic block to be processed is not determined in the current round and the current round reaches a preset round threshold, the search start line is determined as the lower boundary of the first-level semantic block to be processed.

[0007] According to the second aspect, a semantic segmentation apparatus based on title recognition is provided, comprising: a segmentation unit, configured to determine the various levels of titles in a document to be segmented, and to divide the document to be segmented into multiple first-level semantic blocks according to the multiple first-level titles; and a boundary determination unit, configured to, for a first-level semantic block to be processed among the multiple first-level semantic blocks, determine the search start line and the search end line of the first-level semantic block to be processed, and perform a series of boundary searches from the search start line upwards to obtain the lower boundary of the first-level semantic block to be processed, wherein the search start line is located below the search end line, and each round of boundary search includes the following steps: when the document to be segmented is a target type document, determine... If the current text line matches the ending pattern of the target document type, the previous text line is defined as the lower boundary of the first-level semantic block to be processed. If the document to be segmented is not the target document type, it is determined whether the length of the current text line is greater than a preset first threshold and the length of the text line between the current text line and the search start line is less than a preset second threshold. If these conditions are met, the next text line is defined as the lower boundary of the first-level semantic block to be processed. If the lower boundary of the first-level semantic block to be processed is not determined in the current round and the current round reaches a preset round threshold, the search start line is defined as the lower boundary of the first-level semantic block to be processed.

[0008] According to a third aspect, an electronic device is provided, including a processor, a memory, and a computer program / instructions stored in the memory, the processor being configured to execute the computer program / instructions, wherein when the computer program / instructions are executed, the electronic device performs the steps of the method as described in any one of the first aspects. According to a fourth aspect, a computer-readable storage medium is provided, having stored thereon a computer program / instructions that, when executed by a processor, perform the steps of the method as described in any one of the first aspects.

[0009] The semantic segmentation method and apparatus based on title recognition provided in the embodiments of this specification can determine the titles at all levels in the document to be segmented, and divide the document to be segmented into multiple first-level semantic blocks according to multiple first-level titles. For a given first-level semantic block to be processed among multiple first-level semantic blocks, the search start line and search end line of the first-level semantic block to be processed can be determined. The search start line is located below the search end line. The search can be performed line by line from bottom to top starting from the search start line, performing several rounds of boundary search to obtain the lower boundary of the first-level semantic block to be processed. The boundary search in each round can include the following steps: If the document to be segmented is a target type document, determine whether the current text line matches the ending pattern corresponding to the target type document. If they match, determine the previous text line of the current text line as the lower boundary of the first-level semantic block to be processed; If the document to be segmented is not a target type document, determine whether the length of the current text line is greater than a preset first threshold and the length of the text line between the current text line and the search start line is less than a preset second threshold. If they are satisfied, determine the next text line of the current text line as the lower boundary of the first-level semantic block to be processed; If the lower boundary of the first-level semantic block to be processed is not determined in the current round of boundary search and the current round reaches a preset round threshold, determine the search end line as the lower boundary of the first-level semantic block to be processed. Therefore, by performing hierarchical title recognition and bottom-up line-by-line reverse boundary search on the document to be segmented, the lower boundary of the first-level semantic block is dynamically determined, thereby achieving accurate segmentation of the document to be segmented.

[0010] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0011] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0012] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0013] Figure 1 A flowchart of a semantic segmentation method based on title recognition according to one embodiment is shown; Figure 2 A schematic diagram illustrating an example of determining the start and end rows of a search is shown; Figure 3 This diagram illustrates an example of determining the lower boundary when the document to be segmented is a target type document; Figure 4 A schematic block diagram of a semantic segmentation apparatus based on title recognition according to one embodiment is shown. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0015] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0016] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0017] It is understood that the ordinal numbers such as "first" and "second" mentioned in this specification are only used to distinguish multiple objects of the same or different categories (such as components, steps, parameters, etc.), and do not indicate the priority, importance or order relationship between objects, nor do they constitute a limitation on the technical features.

[0018] As mentioned earlier, document semantic segmentation aims to divide a document into meaningful semantic units (such as chapters and paragraphs) to support subsequent content analysis, retrieval, and knowledge extraction. However, some documents may contain non-textual content at the end of chapters or the end of the document itself. For example, the end of some chapters or articles may include translator information, author information, copy information, etc. Clearly, this content does not belong to the main text, but it is easily segmented into semantic blocks as part of the main text, leading to document segmentation errors.

[0019] Therefore, a method is needed that can accurately identify heading levels and dynamically determine segmentation boundaries to improve the robustness and accuracy of document semantic segmentation.

[0020] Therefore, the embodiments of this specification provide a semantic segmentation method based on title recognition. By performing hierarchical title recognition and bottom-up line-by-line reverse boundary search on the document to be segmented, the lower boundary of the first-level semantic block can be dynamically determined, thereby achieving accurate segmentation of the document to be segmented.

[0021] Please see Figure 1 , Figure 1 A flowchart of a semantic segmentation method based on title recognition according to one embodiment is shown. It will be understood that this method can be executed by any device, apparatus, platform, or cluster of devices with computing and processing capabilities. Figure 1 As shown, this semantic segmentation method based on title recognition may include the following steps 101 to 102, specifically: Step 101: Determine the headings at all levels in the document to be segmented, and divide the document into multiple first-level semantic blocks based on the multiple first-level headings.

[0022] In this embodiment, when an input document is received, it can be preprocessed as a document to be segmented, extracting text lines and their positions, etc. For example, the appropriate processing path can be automatically selected based on the document's format characteristics. For instance, for DOCX format documents, a professional document parsing library (such as python-docx, Apache POI, etc.) can be called to parse the document structure and extract the text content and format attributes of each paragraph. For example, the format attributes of a paragraph may include, but are not limited to, metadata information such as font style and paragraph indentation. As another example, for PDF (Portable Document Format) format documents, a processing method combining OCR (Optical Character Recognition) technology with a Markdown conversion engine can be used. First, the text content of the PDF document can be recognized by OCR. Then, the text content is converted into a structured Markdown format, preserving the original hierarchical structure and format characteristics, to obtain the text content and format attributes of each paragraph. For example, for JSON (JavaScript Object Notation) format documents used to store text content, various traversal algorithms (e.g., depth-first traversal, breadth-first traversal) can be used to traverse the JSON tree structure, extract text nodes and their hierarchical relationships, reconstruct the semantic structure of the document, and obtain the text content and formatting attributes of each paragraph. It can be understood that in the preprocessing stage, a text cleaning process can also be performed, including removing irrelevant control characters, removing noise characters, filtering invalid patterns such as advertising text, unifying character encoding formats, and standardizing the text (e.g., full-width / half-width conversion, unification of simplified and traditional Chinese characters, standardization of spaces, etc.) to ensure that the input data for subsequent processing flows has a high degree of consistency and accuracy.

[0023] In this example, the headings at each level of the document to be segmented can also be identified based on the text of each paragraph. For instance, the text content and format of each paragraph can be matched with predefined regular expressions for headings at each level to identify the headings, that is, to identify text lines that match the characteristics of each heading level. As an example, the identified headings can include first-level headings, second-level headings, third-level headings, etc. Then, based on the positional information of multiple first-level headings, the document to be segmented can be divided into multiple first-level semantic blocks, ensuring that the boundary of each first-level semantic block starts from the current first-level heading and extends to the beginning of the next first-level heading, forming a complete semantic unit. For example, assuming a document includes first-level heading 1, first-level heading 2, and first-level heading 3 in top-to-bottom order, the text line containing first-level heading 1 and the text line between first-level heading 1 and first-level heading 2 can be divided into the same first-level semantic block; the text line containing first-level heading 2 and the text line between first-level heading 2 and first-level heading 3 can be divided into the same first-level semantic block; and the text line containing first-level heading 3 and the text lines following it can be divided into the same first-level semantic block. In practice, first-level semantic blocks can be further divided into second-level and third-level semantic blocks based on second-level and third-level headings. Therefore, through hierarchical heading recognition (first-level, second-level, third-level, etc.), fine-grained segmentation of document semantics can be achieved, improving segmentation accuracy and interpretability.

[0024] In some examples, determining the headings at each level in the document to be segmented in step 101 may include the following steps (1) to (2), specifically: Step (1): Obtain the text of each paragraph in the document to be segmented.

[0025] Step (2): For the text of each paragraph, perform the preset title recognition step.

[0026] In this example, the text of each paragraph in the document to be segmented can be obtained, and a preset title recognition step can be performed on the text of each paragraph. The title recognition step may include the following steps (21) and (22), specifically: Step (21) matches the text of each paragraph with the heading patterns in the pre-set heading pattern library, and obtains the candidate heading lines of each level of heading in each paragraph based on the matching results.

[0027] In this example, the title pattern library may include structural patterns and constraint rules. Among them, the structural patterns may include patterns for matching headings at all levels, and the constraint rules may include rules for determining that a text line is not a candidate heading line. Here, a pattern can refer to a template for describing the structure or features of a string. In practice, regular expressions can be used to construct patterns. In this example, the structural patterns may include models for matching headings at all levels such as first-level headings, second-level headings, third-level headings, etc. For example, it may include a regular expression for matching first-level headings starting with "1.": r'^1[、.].+', and the structural patterns may also include a general pattern for matching a sequence of numerical numberings (such as "1.", "(2)"). The constraint rules can be semantic and format constraint rules, and may include rules for determining which text lines are not candidate heading lines. For example, it may include a rule for determining that a text line ending with a full stop is not a candidate heading line, and may also include a rule for determining that a text line containing specific words such as "Figure", "Table", etc. is not a candidate heading line, and may also include a rule for filtering candidate headings based on a preset length interval (title_total_range, subtitle_range), etc. Through the constraint rules, text lines that are formally similar to headings but semantically inconsistent can be filtered out. In some implementation manners, text lines matching the template can be selected as primary candidate heading lines based on the structural patterns, and then, the constraint rules are used to exclude candidate heading lines that do not conform to the constraint rules among the primary candidate heading lines.

[0028] Step (22), based on the candidate heading lines of headings at all levels, determine the heading lines at all levels.

[0029] In this example, based on the candidate heading lines of headings at all levels determined in step (21), the heading lines at all levels can be determined. For example, the candidate heading lines of headings at all levels can be directly used as the heading lines. In practice, in order to improve the accuracy of heading line recognition, a machine learning model can also be used to further screen the candidate heading lines.

[0030] In some examples, in the above step (22), based on the candidate heading lines of headings at all levels, determining the heading lines at all levels may specifically include the following steps (221) and step (222). Specifically: Step (221), extract multi-dimensional features of each candidate heading line, and input the multi-dimensional features of each candidate heading line into a pre-trained heading classification model, and the heading classification model outputs a classification result of whether each candidate heading line is a heading line.

[0031] In this example, a title classification model can be pre-trained, such as a classifier based on random forests or shallow neural networks. This model can be trained using supervised training on a pre-defined training sample set. Based on the multi-dimensional features of the input text lines, the model outputs a classification result indicating whether a text line is a title line. In this example, the extracted multi-dimensional features of the candidate title lines can include formatting features and contextual features. Formatting features can include visual features such as font size, bolding, center alignment, paragraph indentation, and line spacing. Contextual features can include the position of the candidate title line in the document, the formatting features of the preceding text line, and the formatting features of the following text line. In this example, the multi-dimensional features of each candidate title line can be input into the title classification model, which then outputs a classification result indicating whether each candidate title line is a title line.

[0032] Step (222): Determine the title rows at each level based on the classification results output by the title classification model.

[0033] In this example, the classification result output by the heading classification model can include whether a line is a heading or not. The heading lines at each level can be determined based on the classification result. For example, if the classification result for a candidate heading line is "heading line," then that candidate heading line can be determined as a heading line. For instance, taking a heading with three levels as an example, a complete hierarchical semantic structure of the document can be constructed based on the identified heading levels, forming a tree-like structure from first-level headings to third-level headings, fully presenting the semantic organization architecture of the document.

[0034] In this example, a hybrid strategy combining initial screening using a title pattern library with precise judgment by a machine learning model is employed for title line identification. First, the title pattern library is used to quickly locate all possible candidate title lines, ensuring high recall. Then, multi-dimensional features are extracted from each candidate title line, and these features are input into the title classification model. The model then comprehensively evaluates these features to determine whether a candidate title line is a true title line, thus achieving high precision while maintaining high recall.

[0035] In some examples, step (222) above may include steps a) and b), specifically: Step a) Determine multiple title rows based on the classification results output by the title classification model.

[0036] Step b) If two header rows are adjacent among multiple header rows, determine whether the two adjacent header rows meet the preset merging conditions. If they do, merge the two adjacent header rows into one header row.

[0037] In this example, adjacent two-line headings can be further identified. Specifically, when two heading lines are adjacent among multiple heading lines identified by the heading classification model, it can be determined whether these two adjacent heading lines constitute a single unit and whether they need to be merged. That is, it is determined whether the two adjacent heading lines meet the preset merging conditions; if so, the two adjacent heading lines are merged into one heading line. Through positional correlation analysis, it is possible to effectively avoid misjudging short paragraphs adjacent to headings as end boundaries, ensuring the accuracy of segmentation.

[0038] In some examples, the merging criteria may include at least one of the following: the format of two adjacent header lines matches (e.g., the same or similar) and does not match (is different) the format of the body lines; the total length of two adjacent header lines is less than a preset third threshold; or the two adjacent header lines contain preset characters.

[0039] In this example, a format continuity check can be performed on two adjacent heading lines to examine whether their font, size, and other formatting are consistent and significantly different from the body text format. If the two heading lines are consistent in format and significantly different from the body text, then the formatting is considered consistent. Content relevance calculations can also be performed on two adjacent heading lines to assess the reasonableness of the combined length and analyze semantic hierarchy and complementarity based on preset rules. Furthermore, semantic association analysis can be performed on adjacent heading lines, applying key heuristic rules. For example, if the next line is shorter than the previous line, or begins with a preset character such as a dash "—" to explain the previous line, then the two adjacent heading lines are highly likely to form a two-line structure of "main title - subtitle". Through the above comprehensive judgment, two-line headings can be accurately identified and merged.

[0040] This example demonstrates how to divide a document into multiple main semantic blocks based on the distribution of first-level headings. Each semantic block can represent a complete logical chapter. Further refinement can then be performed, dividing each first-level semantic block into finer-grained semantic units based on the identified second- and third-level heading structures. A top-down segmentation strategy is employed to progressively decompose semantic units, forming a hierarchical semantic block network structure.

[0041] Step 102: For the first-level semantic blocks to be processed among multiple first-level semantic blocks, determine the search start line and search end line of the first-level semantic block to be processed, and perform a search line by line from bottom to top starting from the search start line, and perform several rounds of boundary search to obtain the lower boundary of the first-level semantic block to be processed.

[0042] In this embodiment, the first-level semantic block to be processed can refer to a first-level semantic block among multiple first-level semantic blocks whose lower boundary has not been determined through boundary search. For the first-level semantic block to be processed, the search start line and search end line can first be determined from multiple text lines of the first-level semantic block to be processed. Here, the search start line can be located below the search end line. For example, the last line of the first-level semantic block to be processed can be determined as the search start line. In practice, if the end of a chapter or article contains non-text content, this non-text content is often not too much, so a specified line of the first-level semantic block to be processed (e.g., the 10th line from the bottom, or the 20th line from the bottom, etc.) can be determined as the search end line.

[0043] In some examples, step 102, determining the search start line and search end line of the first-level semantic block to be processed, may include the following steps 1) through 3), specifically: Step 1) Determine the header line corresponding to the last subheading of the first-level semantic block to be processed as the search termination line of the first-level semantic block to be processed.

[0044] Step 2), if the first-level semantic block to be processed is the last first-level semantic block, determine the last line of the document to be segmented as the search starting line.

[0045] Step 3) If the first-level semantic block to be processed is not the last first-level semantic block, determine the line before the next first-level heading of the first-level heading corresponding to the first-level semantic block to be processed as the search starting line.

[0046] In this example, the first-level semantic block to be processed may include multiple subheading lines. The heading line corresponding to the last subheading of the first-level semantic block to be processed can be determined as the search termination line for the first-level semantic block to be processed. The search start line can be dynamically determined based on the context. Specifically, if the first-level semantic block to be processed is the last first-level semantic block of the document to be segmented, the last line of the document to be segmented can be determined as the search start line to ensure complete coverage of the end content of the document. If the first-level semantic block to be processed is not the last first-level semantic block of the document to be segmented, the line preceding the next first-level heading of the first-level heading corresponding to the first-level semantic block to be processed can be determined as the search start line to ensure no overlap between first-level semantic blocks. Figure 2 As shown, Figure 2 This diagram illustrates an example of determining the start and end rows of a search. Figure 2In the example shown, the document to be segmented can be divided into multiple first-level semantic blocks based on multiple first-level headings. For each first-level semantic block, the heading line corresponding to the last subheading in that first-level semantic block can be determined as the search termination line for that first-level semantic block. For the last first-level semantic block of the document to be segmented, the last line of the document to be segmented can be determined as the search start line. For non-last first-level semantic blocks of the document to be segmented, the line preceding the next first-level heading (e.g., "VI. XXX" in the figure) of the first-level semantic block corresponding to that first-level semantic block (e.g., "VII. XXXX" in the figure) can be determined as the search start line.

[0047] In this embodiment, after determining the search start line and the search end line, a line-by-line search can be performed from the search start line towards the search end line (i.e., in bottom-up order), performing several rounds of boundary search to obtain the lower boundary (i.e., the end boundary) of the first-level semantic block to be processed. In this example, since the search start line is located below the search end line and the search proceeds line-by-line in bottom-up order, it can be called a reverse search or a backward search. Here, the boundary search in each round includes the following steps 1021 to 1023, specifically: Step 1021: If the document to be segmented is a target type document, determine whether the current text line matches the ending pattern corresponding to the target type document. If they match, determine the previous text line of the current text line as the lower boundary of the first-level semantic block to be processed.

[0048] In this embodiment, the target type document can refer to a document with a corresponding ending pattern. Here, a pattern can refer to a template used to describe the string structure or characteristics, and an ending pattern can refer to a template used to describe the string structure or characteristics of the ending line. In this example, a regular expression can be used to construct the ending pattern. In practice, some documents may have non-text content added at the end of chapters and / or the end of the document according to specified requirements. For example, certain document types such as some technical reports, academic papers, and journal articles are required to add non-text content such as "Editor: xxxx", "Translator: xxx; Affiliation: xxxxx", "Author: xxx, xxx; Affiliation: xxxxx", "CC: xxxx, xxxxx; Editing Unit: xxxxx" at the end of the article. This non-text content added according to specified requirements often has specific string or character characteristics, so a corresponding ending template can be set. That is to say, for specific document types such as technical reports, academic papers, and journal articles, the corresponding ending regular expression pattern can be pre-set (e.g., manually set). For example, if a certain type of document is required to add "(Editor: xxxx)" at the end of the article, a corresponding closing regular expression pattern can be pre-created for this type of document. This closing regular expression pattern is used to match "(Editor: xxxx)".

[0049] In this example, if the document to be segmented is a target type document, it can be determined whether the current text line matches the ending pattern corresponding to the target type document. If they match, the previous text line is determined as the lower boundary of the first-level semantic block to be processed. Figure 3 As shown, Figure 3 This diagram illustrates an example of determining the lower boundary when the document to be segmented is a target type document. Figure 3 In the example shown, the current text line is line 21, and the current text line includes the content "(Editor: Zhang San;)". The current text line matches the preset end regular expression pattern, so the previous text line (i.e., line 20) is determined as the lower boundary of the first-level semantic block to be processed.

[0050] Step 1022: If the document to be segmented is not a target type document, determine whether the current text line length is greater than a preset first threshold and the length of the text line between the current text line and the search start line is less than a preset second threshold. If the conditions are met, determine the next text line of the current text line as the lower boundary of the first-level semantic block to be processed.

[0051] In this embodiment, if the document to be segmented is not a target type document, an adaptive boundary determination method based on text length can be used to determine the lower boundary. Specifically, during boundary search, it can be determined whether the current text line satisfies the following conditions: the length of the current text line is greater than a preset first threshold, and the length of the text line between the current text line and the search start line is less than a preset second threshold. If these conditions are met, the next text line of the current text line is determined as the lower boundary of the first-level semantic block to be processed. Here, the first and second thresholds can be set according to actual needs; the first and second thresholds can be the same or different. That is, in a semantic block, when a normal text line with a length greater than or equal to the preset first threshold is found, and the length of the text lines after this text line (i.e., closer to the end of the semantic block) is less than the preset second threshold (i.e., short lines), then the next line of the current text line (i.e., the end of the main text line break) is determined to be the natural paragraph end line, i.e., the semantic block boundary. This logic is equivalent to finding the first normal text line from back to front based on the text line length, thereby incorporating the subsequent short lines (end of the main text line break) into the current semantic block.

[0052] Step 1023: If the lower boundary of the first-level semantic block to be processed is not determined in the current round and the current round reaches the preset round threshold, the search starting line is determined as the lower boundary of the first-level semantic block to be processed.

[0053] In this embodiment, research has shown that non-text content at the end of a document's chapters or document ends often doesn't occupy much space, and the lower boundary can usually be found after a limited number of boundary search rounds. To prevent excessive boundary search rounds or to handle documents without clear ending features, a `max_seek_steps` parameter (maximum number of search steps) can be set, i.e., a round threshold. During each round of boundary search, it can be determined whether the current search round has reached the maximum number of search steps. If the current boundary search round has not determined the lower boundary of the first-level semantic block to be processed, and the current boundary search round has reached the preset round threshold, the search starting line can be returned as a forced boundary. This is a safety net strategy to ensure the robustness of the algorithm and that there will always be output. In some examples, the lower boundary determined through step 1023 can be logged for subsequent monitoring and optimization reference.

[0054] In some examples, the boundary search in each round may also include Figure 1 The following step S is not shown in the text; specifically: Step S: If the lower boundary of the first-level semantic block to be processed is not determined in the current round and the current round does not reach the round threshold, the previous text line of the current text line is used as the new current text line to perform the boundary search in the next round.

[0055] In this example, if the lower boundary of the first-level semantic block to be processed is not determined in the current boundary search round, and the current boundary search round has not reached the aforementioned round threshold, then the previous text line can be used as the new current text line for the next round of boundary search. For example, suppose the current text line is line 12. If the lower boundary of the first-level semantic block to be processed is not determined through line 12, and the current boundary search round has not reached the round threshold, then since a line-by-line search is performed from bottom to top, line 11 can be used as the new current text line for the next round of boundary search.

[0056] In some examples, to ensure the reasonableness of the searched lower boundary, it can be validated. For instance, validation can include logical validity checks, content integrity checks, and so on. Logical validity checks can verify whether the determined lower boundary lies between the search start and end lines. Content integrity checks can verify whether the content block defined by the lower boundary is complete. For example, it can check whether the text line containing the lower boundary ends with a punctuation mark indicating the end, such as a period. Generally, if the text line containing the lower boundary of a content block does not end with a punctuation mark, or if it contains commas, pauses, or other punctuation marks that do not indicate the end, it indicates that the content block was segmented from a complete sentence and is incomplete.

[0057] In some examples, the semantic segmentation method based on title recognition described above may also include the following steps (1) to (3) specifically: Step 1) Determine candidate semantic blocks based on the position of each first-level heading and the lower boundary of each first-level semantic block.

[0058] In this example, for Heading 1, the lower boundary of the first-level semantic block 1 corresponding to Heading 1 can be determined through steps 101 and 102 above. Based on this, the text line where Heading 1 is located, the text line where the lower boundary of the first-level semantic block 1 is located, and the text line between the two can be divided into the same candidate semantic block.

[0059] Step 2) Based on the position of each level of heading, add the corresponding level marker to each level of heading contained in the candidate semantic block.

[0060] In this example, based on the identified positions of each level of heading, corresponding level markers can be added to the headings within the candidate semantic block. That is, level markers are added to the heading lines of each level of heading within the candidate semantic block. These level markers can be used to indicate the level of the heading corresponding to that heading line, such as level one, level two, level three, etc. In this example, a unified symbol specification and marking system ensures the clarity and consistency of the document structure.

[0061] Step 3) Validate the candidate semantic blocks after adding hierarchical markers based on the preset validation rules, and output the candidate semantic blocks that pass the validation as the final semantic blocks.

[0062] In this example, validation rules can be pre-defined. For instance, these rules could stipulate that the body text under each level of heading cannot be empty. Alternatively, they could specify that heading levels descend sequentially and cannot skip levels; for example, a level one heading cannot skip a level two heading and directly appear as a level three heading. Based on these rules, candidate semantic blocks with added hierarchy markers can be validated, and the validated candidate semantic blocks can be output as the final semantic blocks. As an example, the output can support multiple format conversions, including plain text, structured JSON, and standard Markdown, ensuring seamless integration of the segmentation results with subsequent processing workflows and providing a high-quality data foundation for applications such as document analysis, knowledge extraction, and content management.

[0063] Understandable. Figure 1 The semantic segmentation method based on title recognition shown describes the process of dividing a document to be segmented into multiple first-level semantic blocks and determining the lower boundaries of the first-level semantic blocks. In practice, the same semantic segmentation method can be used to further segment the first-level semantic blocks, which is equivalent to performing semantic segmentation on the basis of the first-level semantic blocks. Figure 1 The semantic segmentation method based on title recognition shown in the figure divides a first-level semantic block into multiple second-level semantic blocks and determines the lower boundary of each second-level semantic block. For example, in some examples, the first-level semantic block to be processed may include multiple second-level subheadings, and the first-level semantic block to be processed can be divided into multiple second-level semantic blocks based on these multiple second-level subheadings. For each second-level semantic block to be processed, the search start line and search end line of the second-level semantic block to be processed can be determined. Starting from the search start line of the second-level semantic block to be processed, a line-by-line search is performed from bottom to top, performing several rounds of boundary search to obtain the lower boundary of the second-level semantic block to be processed. The search start line of the second-level semantic block to be processed is located below the search end line of the second-level semantic block to be processed.

[0064] Here, the process of dividing a primary semantic block into multiple secondary semantic blocks and determining the lower boundary of each secondary semantic block is similar to the process of dividing the document to be segmented into multiple primary semantic blocks and determining the lower boundary of each primary semantic block, and will not be elaborated here. Similarly, the secondary semantic blocks can be further semantically segmented based on the same semantic segmentation method, which is equivalent to performing semantic segmentation on the basis of the secondary semantic blocks. Figure 1 The semantic segmentation method based on title recognition shown in the figure divides the secondary semantic block into multiple tertiary semantic blocks and determines the lower boundary of the tertiary semantic blocks.

[0065] To verify the effectiveness of the semantic segmentation method based on title recognition in this embodiment, some experiments were conducted.

[0066] Specifically, on a test set containing 1000 technical documents, the method in this embodiment, due to its use of reverse boundary search, achieved a semantic segmentation accuracy of 92.3%, a significant improvement compared to rule-based methods (81.5% accuracy) and traditional machine learning-based methods (83.7% accuracy). Particularly when processing documents with complex formats, the reverse boundary search mechanism of this invention effectively avoided 17.8% of boundary errors. Please refer to Table 1, which shows a performance comparison of different methods in document semantic segmentation.

[0067] Table 1. Performance comparison of different methods in document semantic segmentation In summary, this embodiment solves the semantic segmentation challenge of diverse documents through hierarchical title recognition and reverse boundary search mechanisms. This method is applicable to the analysis and processing of structured texts such as technical documents, academic papers, and reports, providing a reliable technical foundation for intelligent document understanding and knowledge extraction.

[0068] According to another embodiment, a semantic segmentation apparatus based on title recognition is provided. This title-recognition-based semantic segmentation apparatus can be deployed in any device, platform, or device cluster with computing and processing capabilities.

[0069] Figure 4 A schematic block diagram of a semantic segmentation apparatus based on title recognition according to one embodiment is shown. Figure 4 As shown, the semantic segmentation device 400 based on title recognition includes: The segmentation unit 401 is used to determine the headings at all levels in the document to be segmented, and to divide the document to be segmented into multiple first-level semantic blocks according to the multiple first-level headings; Boundary determination unit 402 is used to determine the search start line and search end line of the unprocessed first-level semantic block among the plurality of first-level semantic blocks, and perform boundary search line by line from bottom to top starting from the search start line, performing several rounds of boundary search to obtain the lower boundary of the unprocessed first-level semantic block, wherein the search start line is located below the search end line, and each round of boundary search includes the following steps: If the document to be segmented is a target type document, determine whether the current text line matches the ending pattern corresponding to the target type document. If they match, determine the previous text line of the current text line as the lower boundary of the first-level semantic block to be processed. If the document to be segmented is not the target type document, determine whether the current text line length is greater than a preset first threshold and the length of the text line between the current text line and the search start line is less than a preset second threshold. If the conditions are met, the next text line of the current text line is determined as the lower boundary of the first-level semantic block to be processed. If the lower boundary of the first-level semantic block to be processed is not determined in the current round and the current round reaches the preset round threshold, the search starting line is determined as the lower boundary of the first-level semantic block to be processed.

[0070] According to another embodiment, an electronic device is also provided, including a processor, a memory, and a computer program / instructions stored in the memory, characterized in that the processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the electronic device performs the steps of the method as described above.

[0071] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program / instructions are stored, characterized in that, when executed by a processor, the computer program / instructions implement the steps of the method as described above. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0072] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0073] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0074] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A semantic segmentation method based on title recognition, comprising: Identify the headings at each level in the document to be segmented, and divide the document into multiple first-level semantic blocks based on the multiple first-level headings; For the unprocessed first-level semantic block among the multiple first-level semantic blocks, the search start line and search end line of the unprocessed first-level semantic block are determined. Starting from the search start line, a search is performed line by line in a bottom-up order, and several rounds of boundary search are performed to obtain the lower boundary of the unprocessed first-level semantic block. The search start line is located below the search end line. Each round of boundary search includes the following steps: If the document to be segmented is a target type document, determine whether the current text line matches the ending pattern corresponding to the target type document. If they match, determine the previous text line of the current text line as the lower boundary of the first-level semantic block to be processed. If the document to be segmented is not the target type document, determine whether the current text line length is greater than a preset first threshold and the length of the text line between the current text line and the search start line is less than a preset second threshold. If the conditions are met, the next text line of the current text line is determined as the lower boundary of the first-level semantic block to be processed. If the lower boundary of the first-level semantic block to be processed is not determined in the current round and the current round reaches the preset round threshold, the search starting line is determined as the lower boundary of the first-level semantic block to be processed.

2. The method according to claim 1, characterized in that, The boundary search in each round also includes the following steps: If the lower boundary of the first-level semantic block to be processed is not determined in the current round and the current round does not reach the round threshold, the previous text line of the current text line is used as the new current text line to perform the boundary search in the next round.

3. The method according to claim 1, characterized in that, Determining the search start line and search end line of the first-level semantic block to be processed includes: The header line corresponding to the last subheading of the first-level semantic block to be processed is determined as the search termination line of the first-level semantic block to be processed; If the first-level semantic block to be processed is the last first-level semantic block, the last line of the document to be segmented is determined as the search starting line; If the first-level semantic block to be processed is not the last first-level semantic block, the line preceding the next first-level title of the first-level title corresponding to the first-level semantic block to be processed is determined as the search starting line.

4. The method according to claim 1, characterized in that, Determining the headings at each level in the document to be segmented includes: Extract the text of each paragraph in the document to be segmented; For each paragraph's text, perform the following heading recognition steps: The text of each paragraph is matched with the heading patterns in a pre-defined heading pattern library. Based on the matching results, candidate heading lines for each level of heading in each paragraph are obtained. The heading pattern library includes structural patterns and constraint rules. The structural patterns include patterns for matching headings at each level, and the constraint rules include rules for determining that a text line is not a candidate heading line. Based on the candidate rows of headings at each level, determine the heading rows at each level.

5. The method according to claim 4, characterized in that, The determination of header rows based on candidate header rows at each level includes: Extract multi-dimensional features of each candidate title line and input the multi-dimensional features of each candidate title line into a pre-trained title classification model. The title classification model outputs the classification result of whether each candidate title line is a title line. The multi-dimensional features include format features and context features. Based on the classification results output by the title classification model, determine the title rows at each level.

6. The method according to claim 5, characterized in that, The step of determining the title rows at each level based on the classification results output by the title classification model includes: Based on the classification results output by the title classification model, multiple title rows are determined; If two header rows are adjacent among the plurality of header rows, it is determined whether the two adjacent header rows meet a preset merging condition. If they do, the two adjacent header rows are merged into one header row. The merging condition includes at least one of the following: The format of two adjacent header lines matches, but the format of the body text lines does not match; The total length of two adjacent header lines is less than the preset third threshold; Two adjacent header lines contain preset characters.

7. The method according to claim 1, characterized in that, The primary semantic block to be processed includes multiple secondary subheadings, and the method further includes: The primary semantic block to be processed is divided into multiple secondary semantic blocks according to the multiple secondary subheadings; For the unprocessed secondary semantic block among the multiple secondary semantic blocks, the search start line and search end line of the unprocessed secondary semantic block are determined. Starting from the search start line of the unprocessed secondary semantic block, the search is performed line by line in a bottom-up order, and several rounds of boundary search are performed to obtain the lower boundary of the unprocessed secondary semantic block. The search start line of the unprocessed secondary semantic block is located below the search end line of the unprocessed secondary semantic block.

8. The method according to claim 1, characterized in that, The method further includes: Candidate semantic blocks are determined based on the position of each first-level heading and the lower boundary of each first-level semantic block; Based on the position of each level of heading, add the corresponding level marker to each level of heading contained in the candidate semantic block; The candidate semantic blocks with added hierarchical markers are validated based on the preset validation rules, and the validated candidate semantic blocks are output as the final semantic blocks.

9. An electronic device comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the electronic device implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 8.

Images