Multi-modal document-oriented intelligent segmentation and core information extraction system

Abstract

The invention discloses a multi-modal document-oriented intelligent segmentation and core information extraction system, which relates to the field of text information processing, and comprises a title screening module, a multi-strategy segmentation module, a multi-modal fusion module and a core information extraction module, the title screening module is used for obtaining downstream task requirements and multi-modal documents, screening multi-level titles through a title matching template, obtaining associated titles related to downstream tasks, and positioning corresponding associated text contents; the multi-strategy segmentation module selects an adaptive self-adaptive segmentation method to intelligently segment associated text contents to obtain text blocks with complete semantics; the multi-modal fusion module calls the multi-modal large model to process non-text elements to obtain a multi-modal fusion block; the core information extraction module extracts high-frequency keywords and core information in the multi-modal fusion block through cooperative processing of a statistical path and a semantic path, and intelligent segmentation and core information extraction of a multi-modal document are achieved.

Description

Technical Field

[0001] This invention relates to the field of text information processing, specifically to an intelligent segmentation and core information extraction system for multimodal documents. Background Technology

[0002] In natural language processing, intelligent question answering, knowledge base construction, and intelligent information retrieval, document preprocessing is a crucial link between raw document data and downstream tasks. Its core objective is to provide downstream tasks with high-quality data input that eliminates invalid data and is semantically complete through key operations such as data filtering, intelligent text segmentation, and core information extraction. With the surge in document data volume across industries and the continuous expansion of document content from plain text to multimodal formats such as mixed text and images and tables, traditional document processing methods are no longer sufficient to meet the demands of downstream tasks.

[0003] Traditional document processing methods have significant shortcomings: First, document segmentation generally relies on fixed token segmentation methods, failing to combine document hierarchical structure and semantic coherence for adaptive and semantically complete intelligent segmentation, easily leading to semantic fragmentation or cross-segment semantic loss. Second, traditional methods are mostly designed for plain text documents and cannot effectively process multimodal documents containing non-text elements such as images and formulas, failing to achieve unified alignment and utilization of cross-modal content. Third, existing core information extraction methods mostly rely on simple word frequency statistics or simple summarization methods, insufficiently utilizing domain knowledge, semantic context, and task objectives, making it difficult to simultaneously ensure the accuracy of key content extraction and domain adaptability. Therefore, how to improve the ability of intelligent document segmentation and core information extraction has become a key problem that needs to be solved in current text information processing. Summary of the Invention

[0004] The purpose of this invention is to provide an intelligent segmentation and core information extraction system for multimodal documents. Through a title filtering module to accurately locate related text content, multi-strategy segmentation to ensure the semantic integrity of the text, multimodal information fusion to achieve cross-modal content alignment, and dual-path core information extraction to improve extraction accuracy, this system can process multimodal documents and provide high-quality data support with complete semantics for downstream tasks.

[0005] The technical solution to achieve the objective of this invention is as follows:

[0006] An intelligent segmentation and core information extraction system for multimodal documents includes a title filtering module, a multi-strategy segmentation module, a multimodal fusion module, and a core information extraction module;

[0007] The title filtering module obtains downstream task requirements and multimodal documents, filters multi-level titles in the multimodal documents using title matching templates, obtains related titles related to downstream tasks, and locates the corresponding related text content.

[0008] The multi-strategy segmentation module selects an appropriate adaptive segmentation method to segment the associated text content and generate semantically complete text blocks. The adaptive segmentation method includes title structure segmentation, rule aggregation segmentation, and semantic aggregation segmentation.

[0009] The multimodal fusion module calls the multimodal large model to process the non-text elements corresponding to the associated title, generates non-text semantic descriptions, and binds them with the corresponding text blocks to obtain the multimodal fusion block;

[0010] The core information extraction module uses a combination of statistical and semantic paths. The statistical path extracts high-frequency keywords from the multimodal fusion block based on word segmentation and word frequency statistics, while the semantic path calls a large language model based on task prompts to generate accurate summaries or key sentences.

[0011] Furthermore, the system receives the downstream task requirement description and the original multimodal document input by the user, preprocesses the original multimodal document, unifies the document input format, standardizes the format, and removes noise such as watermark text or garbled characters from the document to obtain the preprocessed multimodal document.

[0012] Furthermore, the title filtering module includes a document parsing unit and an associated title filtering unit;

[0013] Specifically, the document parsing unit uses a document parsing tool to perform full parsing of the multimodal document, extracts the document's structural information, and constructs a multi-level heading tree structure based on the document's structural information, including the following steps:

[0014] A unique identifier ID for the title is generated according to the title level and the order of appearance in the document. The title information is then standardized to generate title node attributes. The title node attributes include title ID, level identifier, title text, parent title ID, title position index in the document, the text boundary corresponding to the title, non-text elements in the text and their position indexes, and associated task initialization tag.

[0015] The text boundary corresponding to the title includes the start position and the end position of the text.

[0016] Based on the heading level and parent heading ID, a hierarchical dependency relationship is established. A multi-level heading tree structure is constructed from top to bottom through hierarchical iteration. First, the heading node with a level of 1 and an empty parent heading ID is selected as the root node.

[0017] For each root node, iterate through all second-level heading nodes whose parent heading ID is the root node, insert them as child nodes into the root node's child node list, and so on, until all heading nodes have been included in the tree structure, thus obtaining a multi-level heading tree structure.

[0018] Specifically, the associated title filtering unit, based on the requirements of downstream tasks, matches multi-level titles from top to bottom in the multi-level title tree structure according to a preset title matching template, sorts the multi-level titles by digital weight according to preset rules, filters associated titles related to downstream tasks, and locates the corresponding associated text content.

[0019] Preferably, when a title node has been included in the associated title set, its subtitle nodes no longer perform rule matching, in order to reduce redundant calculations and maintain the semantic inheritance relationship of the title tree structure.

[0020] Furthermore, the associated text content set is input into the multi-strategy segmentation module, which selects an appropriate adaptive segmentation method to segment it. The multi-strategy segmentation module includes a title structure segmentation unit, a rule aggregation segmentation unit, and a semantic aggregation segmentation unit.

[0021] Specifically, the title structure segmentation unit performs structured segmentation of related text content based on a tree structure of multi-level titles. First, it extracts the hierarchical identifiers and text boundary information of related title nodes, determines the segmentation granularity (such as chapter, section, subsection) based on the hierarchical identifiers, reads the text boundary in the attributes of each related title node, and, according to the title hierarchy order and the start and end positions of the text, adopts a top-down recursive segmentation method for title nodes with nested relationships to divide the related text content set into several independent segments, ensuring that the segmentation results correspond to the original document hierarchy.

[0022] Preferably, the related text content after the title structure is segmented is split into short sentences according to punctuation marks. The short sentences are then cleaned to remove redundant symbols, spaces, and meaningless characters, resulting in a set of short sentences that serves as the input basis for subsequent aggregation and segmentation.

[0023] Specifically, the rule aggregation and segmentation unit is based on a set of short sentences. It uses a preset rule base to structurally merge adjacent short sentences to generate structured text blocks with rule aggregation, ensuring the continuity of text context semantics and logical integrity.

[0024] The preset rule base is used to identify structural or semantic continuity relationships between short sentences. The rules include, but are not limited to, grammatical structure rules, format structure rules, semantic continuity rules, and regular expression matching rules. The preset rule base can be customized and loaded or dynamically expanded according to the type of input document.

[0025] Preferably, the structured merging of adjacent short phrases based on a preset rule base, while preserving tail overlap, includes the following steps:

[0026] Iterate through the set of short sentences sequentially and perform rule matching judgment on adjacent short sentences;

[0027] If any rule or regular expression matching condition is met, the current short sentence is merged with the previous short sentence to form a temporary text block;

[0028] If the rule conditions are not met, it is determined to be a semantic breakpoint, the current aggregation ends, and a new text block is generated;

[0029] When a repetition or high similarity segment between the end of the first sentence and the beginning of the second sentence is detected between adjacent short sentences, the overlap interval between the two is calculated, and the overlap range is determined by the longest common substring or edit distance method. The overlapping segments are preserved during merging.

[0030] After each merge operation, the merged result is used as the new current text block, and rule matching continues until all short sentences are traversed, and a set of structured text blocks is output.

[0031] Preferably, the preset regular expression can be customized and loaded by the user according to the text type or called through a template library, thereby realizing an extensible rule aggregation and segmentation method.

[0032] Specifically, the semantic aggregation and segmentation unit dynamically aggregates and generates semantically complete text blocks by calculating the semantic relationships between adjacent short sentences, including the following steps:

[0033] For each short sentence, a dense semantic vector and a sparse semantic vector are generated to form the semantic vector representation of the short sentence, thus obtaining the short sentence-semantic vector mapping table;

[0034] The sparse semantic vector is a high-dimensional semantic representation based on word frequency statistical features, and the dense vector is a low-dimensional semantic representation generated based on a semantic coding model. The lightweight semantic coding model Sentence-BERT is preferred to perform semantic coding on each short sentence to generate a dense semantic vector with uniform dimensions.

[0035] The semantic vector of the first short sentence is used as the initial paragraph cluster center, and the short sentence is added to the list of short sentences to be merged;

[0036] The subsequent short sentences are traversed sequentially, and the semantic similarity between the current short sentence and the cluster center is calculated based on the weighted cosine similarity of sparse semantic vectors and dense semantic vectors.

[0037] If the semantic similarity is greater than or equal to the semantic similarity threshold, add it to the list of short sentences to be merged, and update the cluster center by calculating the average of the vectors of all short sentences in the current list;

[0038] If the semantic similarity is less than the similarity threshold, it is judged as a semantic break, the current aggregation ends, the list of short sentences to be merged is encapsulated into a semantic aggregation text block, and the cluster center is re-initialized with the semantic vector of the current short sentence to start a new round of aggregation;

[0039] Repeat the above steps until all short sentences have been traversed, generating a semantic aggregated text block set.

[0040] Preferably, the lightweight semantic coding model Sentence-BERT is pre-trained on a large-scale general corpus to obtain pre-trained weights, and these pre-trained weights are loaded as the initial weights of the model.

[0041] Furthermore, all non-text elements and their position indices corresponding to the associated title nodes are extracted, and the original non-text elements are extracted using a document parsing tool; preprocessing is performed according to the element type, and they are uniformly converted into a standard data description format; a multimodal large model is called to perform semantic parsing and description generation on different types of non-text elements to obtain the corresponding semantic descriptions of non-text elements; the non-text elements include, but are not limited to: images, formulas, tables, pseudo-algorithms, etc.

[0042] Based on the position index, each non-textual semantic description is bound to the text block processed by the multi-strategy segmentation module to achieve cross-modal alignment and generate a multimodal fusion block.

[0043] Optionally, the multimodal large model is a large-scale language model with visual understanding and text generation capabilities, such as GPT-4V, BLIP-2, or Florence-2. It outputs corresponding non-textual semantic descriptions by inputting non-textual element content and its contextual information.

[0044] Furthermore, the core information extraction module extracts high-frequency keywords and core information from the multimodal fusion block through the collaborative processing of statistical and semantic paths, including statistical information units and semantic information units.

[0045] Specifically, the statistical information unit extracts high-frequency keywords that can characterize the core theme of the text block based on word frequency statistics and domain feature weighting, including the following steps:

[0046] Build a domain-specific dictionary, a general stop word list, and a domain-specific stop word list;

[0047] The text block is segmented using a Chinese word segmentation tool, and meaningless and non-informational words are filtered out to obtain a set of effective words.

[0048] Perform word frequency statistics on the set of valid words, record the occurrence frequency of each valid word, and generate an initial word frequency table;

[0049] A domain weight coefficient is introduced to adjust word frequency, assigning high weight to core effective words in the domain and low weight to non-core high-frequency effective words, in order to optimize word frequency statistics results;

[0050] A preset high-frequency word threshold is used to filter valid words whose frequency is greater than or equal to the preset high-frequency word threshold. These words are then sorted in descending order of frequency value, and the top few are selected. Use 10 effective words to generate a list of high-frequency keywords.

[0051] Specifically, the semantic information unit selects appropriate task prompt words based on downstream task requirements and document content to guide the large language model to generate accurate summaries or key sentences that meet task requirements. The accurate summary is a brief description of the multimodal text block, and the key sentence is composed of multiple keywords in the multimodal text block.

[0052] Optionally, the open-source Qwen2-VL-7B-Instruct model can be used as the large language model. The Qwen2-VL-7B-Instruct model supports long text understanding.

[0053] Furthermore, the list of high-frequency keywords generated by the statistical path and the accurate summary or key sentences generated by the semantic path are associated according to the multimodal fusion block to generate structured processing results.

[0054] Compared with the prior art, the significant advantages of this invention are:

[0055] 1. Construct a semantically aware multi-strategy segmentation module, abandoning the traditional fixed token segmentation logic, and designing three adaptive segmentation methods for different document contents: title structure segmentation, semantic aggregation segmentation, and rule aggregation segmentation, to avoid text semantic fragmentation and ensure the semantic coherence and integrity of text blocks.

[0056] 2. A multimodal fusion module is introduced to effectively process multimodal documents containing non-text elements such as images, formulas, and tables. By calling a large multimodal model to perform semantic parsing and description generation on non-text elements, cross-modal alignment and fusion are achieved.

[0057] 3. By constructing a collaborative processing mechanism of statistical and semantic paths to extract core information from text blocks, and by introducing domain-feature-weighted word frequency statistics and a large language model based on task prompt words, key information related to downstream tasks can be extracted more accurately, thereby improving the efficiency and quality of information extraction. Attached Figure Description

[0058] Figure 1 A flowchart of an intelligent segmentation and core information extraction system for multimodal documents;

[0059] Figure 2 This is a flowchart of the semantic aggregation and segmentation process in this invention;

[0060] Figure 3 This is a flowchart illustrating the process of generating a list of high-frequency keywords in this invention. Detailed Implementation

[0061] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0062] like Figure 1 As shown, this invention discloses an intelligent segmentation and core information extraction system for multimodal documents, including a title filtering module, a multi-strategy segmentation module, a multimodal fusion module, and a core information extraction module:

[0063] The title filtering module obtains downstream task requirements and multimodal documents, filters multi-level titles in the multimodal documents using title matching templates, obtains related titles related to downstream tasks, and locates the corresponding related text content.

[0064] The multi-strategy segmentation module selects an appropriate adaptive segmentation method to segment the associated text content and generate semantically complete text blocks. The adaptive segmentation method includes title structure segmentation, rule aggregation segmentation, and semantic aggregation segmentation.

[0065] The multimodal fusion module calls the multimodal large model to process the non-text elements corresponding to the associated title, generates non-text semantic descriptions, and binds them with the corresponding text blocks to obtain the multimodal fusion block;

[0066] The core information extraction module uses a combination of statistical and semantic paths. The statistical path extracts high-frequency keywords from the multimodal fusion block based on word segmentation and word frequency statistics, while the semantic path calls a large language model based on task prompts to generate accurate summaries or key sentences.

[0067] Furthermore, the system receives a description of downstream task requirements and an original multimodal document input by the user. The multimodal document contains at least two types of information, including text, images, tables, or formulas. In this embodiment, considering that the original multimodal document may come from various sources, have inconsistent file formats and encoding methods, and may even contain noise information such as watermarked text and garbled characters, a preprocessing operation is first performed on the original multimodal document. The preprocessing includes format standardization, character encoding unification, and noise content removal to generate a preprocessed multimodal document.

[0068] Furthermore, the title filtering module includes a document parsing unit and an associated title filtering unit;

[0069] Specifically, the document parsing unit uses a document parsing tool to perform full parsing of the multimodal document, extracts the document's structural information, and constructs a multi-level heading tree structure based on the document's structural information. The full parsing involves extracting the document's structural information based on layout analysis algorithms and OCR recognition technology. This structural information includes: the text content of all headings in the document, the physical location of the headings within the document, heading level identifiers, the text boundaries corresponding to each heading, and the non-text elements contained within the text and their position indices.

[0070] Furthermore, constructing a multi-level heading tree structure based on the document's structural information includes the following steps:

[0071] A unique identifier ID for the title is generated according to the title level and the order of appearance in the document. The title information is then standardized to generate title node attributes. The title node attributes include title ID, level identifier, title text, parent title ID, title position index in the document, the text boundary corresponding to the title, non-text elements in the text and their position indexes, and associated task initialization tag.

[0072] The text boundary corresponding to the title includes the start position and the end position of the text.

[0073] Based on the heading level and parent heading ID, a hierarchical dependency relationship is established. A multi-level heading tree structure is constructed from top to bottom through hierarchical iteration. First, the heading node with a level of 1 and an empty parent heading ID is selected as the root node.

[0074] For each root node, iterate through all second-level heading nodes whose parent heading ID is the root node, insert them as child nodes into the root node's child node list, and so on, until all heading nodes have been included in the tree structure, thus obtaining a multi-level heading tree structure.

[0075] Specifically, the associated title filtering unit, based on the requirements of downstream tasks, matches multi-level titles from top to bottom in the multi-level title tree structure according to a preset title matching template, sorts the multi-level titles by digital weight according to preset rules, filters associated titles related to downstream tasks, locates the text content corresponding to each associated title based on the text boundary information in the associated title node attributes, and generates a set of associated text content.

[0076] Preferably, when a title node has been included in the associated title set, its subtitle nodes no longer perform rule matching, in order to reduce redundant calculations and maintain the semantic inheritance relationship of the title tree structure.

[0077] In this embodiment, after completing dynamic title filtering and obtaining the associated text content set, this embodiment further provides a semantically aware multi-strategy segmentation module, which is used to divide the associated text content set into structured content fragments to adapt to the output granularity of different downstream tasks. To avoid semantic fragmentation and information loss caused by traditional fixed-token-based segmentation, this embodiment comprehensively implements content segmentation through a variety of adaptive segmentation methods. Among them, the adaptive segmentation methods include, but are not limited to: title structure segmentation, rule aggregation segmentation, and semantic aggregation segmentation. The above methods are automatically selected or combined to perform segmentation based on the document structure and content features.

[0078] Specifically, heading structure segmentation is the preferred strategy of the multi-strategy segmentation module. It performs structured segmentation based on the original hierarchical identifiers of the document, ensuring that the segmentation results are consistent with the document logic, including the following steps:

[0079] Based on the multi-level heading tree structure, extract the hierarchical identifier and text boundary information of each associated heading node;

[0080] Based on the hierarchical identifier, determine the segmentation granularity (such as chapter, section, subsection), read the text boundary in the attributes of each associated title node, and divide the associated text content set into several independent segments according to the title hierarchy order and the start and end positions of the text.

[0081] For title nodes with nested relationships, a top-down recursive segmentation method is used to ensure that the segmentation results correspond to the original document hierarchy.

[0082] Preferably, the main text content after structural segmentation is split into short sentences according to punctuation marks, and the short sentences are cleaned to remove redundant symbols, spaces and meaningless characters, resulting in a set of short sentences, which serves as the input basis for subsequent aggregation and segmentation.

[0083] For text content with standardized text format and consistent logical expression, this embodiment adopts a rule-aggregated segmentation unit. Based on a set of short sentences, adjacent short sentences are structurally merged through a preset rule base to form rule-aggregated text blocks, ensuring the continuity of text context semantics and logical integrity.

[0084] Specifically, the preset rule base is used to identify structural or semantic continuity relationships between short sentences. The rules include, but are not limited to, grammatical structure rules, format structure rules, semantic continuity rules, and regular expression matching rules. The preset rule base can be customized and loaded or dynamically expanded according to the input document type.

[0085] Specifically, the structured merging of adjacent short phrases based on a preset rule base, while preserving tail overlap, includes the following steps:

[0086] Iterate through the set of short sentences sequentially and perform rule matching judgment on adjacent short sentences;

[0087] If any rule or regular expression matching condition is met, the current short sentence is merged with the previous short sentence to form a temporary text block;

[0088] If the rule conditions are not met, it is determined to be a semantic breakpoint, the current aggregation ends, and a new text block is generated;

[0089] When a repetition or highly similar segment between the end of the first sentence and the beginning of the second sentence is detected between adjacent short sentences, the overlap interval between the two is calculated, and the overlap range is determined by the longest common substring or edit distance method. The overlapping segments are preserved during merging to enhance the semantic connection of the context and the coherence of reading.

[0090] After each merge operation, the merged result is used as the new current text block, and rule matching continues until all short sentences are traversed, and a set of structured text blocks is output.

[0091] Preferably, the preset regular expression can be customized and loaded by the user according to the text type or called through a template library, thereby realizing an extensible rule aggregation and segmentation method.

[0092] For example, in the implementation of building a knowledge base in the field of AI technology, if the first sentence is "the encoder of the model adopts the Transformer architecture" and the second sentence is "the Transformer architecture includes a multi-head attention mechanism", both sentences contain the AI ​​field term "Transformer architecture", which triggers the overlap matching rule of the end of the first sentence and the beginning of the second sentence, and the two sentences need to be merged.

[0093] like Figure 2 As shown, for related text content with no fixed format and complex semantic logic, this embodiment uses a semantic aggregation segmentation unit and a semantic aggregation segmentation method to dynamically aggregate and generate semantically complete text blocks by calculating the semantic relationships between short sentences. The semantic aggregation segmentation method includes the following steps:

[0094] For each short sentence, a dense semantic vector and a sparse semantic vector are generated to form the semantic vector representation of the short sentence, thus obtaining the short sentence-semantic vector mapping table;

[0095] The sparse semantic vector is a high-dimensional semantic representation based on word frequency statistical features, and the dense vector is a low-dimensional semantic representation generated based on a semantic coding model. The lightweight semantic coding model Sentence-BERT is preferred to perform semantic coding on each short sentence to generate a dense semantic vector with uniform dimensions.

[0096] The semantic vector of the first short sentence is used as the initial paragraph cluster center, and the short sentence is added to the list of short sentences to be merged;

[0097] The subsequent short sentences are traversed sequentially, and the semantic similarity between the current short sentence and the cluster center is calculated based on the weighted cosine similarity of sparse semantic vectors and dense semantic vectors.

[0098] If the semantic similarity is greater than or equal to the semantic similarity threshold, add it to the list of short sentences to be merged, and update the cluster center by calculating the average of the vectors of all short sentences in the current list;

[0099] If the semantic similarity is less than the similarity threshold, it is judged as a semantic break, the current aggregation ends, the list of short sentences to be merged is encapsulated into a semantic aggregation text block, and the cluster center is re-initialized with the semantic vector of the current short sentence to start a new round of aggregation;

[0100] Repeat the above steps until all short sentences have been traversed, generating a semantic aggregated text block set.

[0101] Preferably, the lightweight semantic coding model Sentence-BERT is pre-trained on a large-scale general corpus to obtain pre-trained weights, and these pre-trained weights are loaded as the initial weights of the model.

[0102] Optionally, if a short sentence is detected to contain a key semantic transition word, even if its semantic similarity with the previous short sentence is greater than or equal to a preset semantic similarity threshold, a new paragraph aggregation must be forcibly initiated.

[0103] Optionally, the semantic similarity threshold is determined by semantic aggregation verification using a small sample text set, or it can be adaptively adjusted according to different text types.

[0104] Optionally, this embodiment can combine rule-based segmentation with semantic aggregation segmentation. First, structured text blocks are formed by aggregation according to rules, and then deep semantic segmentation is achieved through dense / sparse vector semantic aggregation.

[0105] Furthermore, all non-text elements and their position indices corresponding to the associated title nodes are extracted, and the original non-text elements are extracted using a document parsing tool. The elements are preprocessed according to their type and uniformly converted into a standard data description format. A multimodal large model is called to perform semantic parsing and description generation on different types of non-text elements to obtain corresponding semantic descriptions of non-text elements. The non-text elements include, but are not limited to: images, formulas, tables, pseudo-algorithms, etc.

[0106] For image elements, generate image semantic descriptions that are relevant to the document's theme;

[0107] For formula elements, a mathematical meaning description in natural language form is generated based on the recognized mathematical expression;

[0108] For table elements, extract their structural information and main data features to generate a semantic summary of the table content;

[0109] For pseudo-algorithm elements, identify the algorithm steps and logical flow, and generate corresponding algorithm semantic descriptions;

[0110] Optionally, the multimodal large model can be a large-scale language model with visual understanding and text generation capabilities, such as GPT-4V, BLIP-2, or Florence-2, which outputs corresponding non-textual semantic descriptions by inputting non-textual element content and its contextual information.

[0111] Furthermore, based on the position index, each non-textual semantic description is bound to the text block processed by the multi-strategy segmentation module to achieve cross-modal alignment and generate a multimodal fusion block.

[0112] In this embodiment, the core information extraction module extracts high-frequency keywords and core information from the multimodal fusion block through the collaborative processing of statistical paths and semantic paths, including statistical information units and semantic information units.

[0113] like Figure 3 As shown, the statistical information unit extracts high-frequency keywords that can represent the core theme of the text based on word frequency statistics and domain feature weighting, including the following steps:

[0114] Build a domain-specific dictionary, a general stop word list, and a domain-specific stop word list;

[0115] The text block is segmented using the Chinese word segmentation tool jieba. Meaningless and non-informative words are filtered out from the segmentation results to obtain a set of effective words.

[0116] Perform word frequency statistics on the set of valid words, record the occurrence frequency of each valid word, and generate an initial word frequency table;

[0117] A domain weight coefficient is introduced to adjust word frequency, assigning high weight to core effective words in the domain and low weight to non-core high-frequency effective words, in order to optimize word frequency statistics results;

[0118] A preset high-frequency word threshold is used to filter valid words whose frequency is greater than or equal to the preset high-frequency word threshold. These words are then sorted in descending order of frequency value, and the top few are selected. Use 10 effective words to generate a list of high-frequency keywords;

[0119] For example, in the implementation of building a knowledge base in the field of AI technology, a domain-customized dictionary containing AI domain terms such as "Transformer" and "loss function" is constructed, while a domain stop word list containing generalized words such as "experiment," "method," and "result" is constructed for word segmentation and filtering.

[0120] Specifically, the semantic information unit selects appropriate task prompt words based on downstream task requirements and document content to guide the large language model to generate accurate summaries or key sentences that meet task requirements. The accurate summary is a brief description of the semantic text block, and the key sentence is composed of multiple keywords in the semantic text block.

[0121] For example, the open-source Qwen2-VL-7B-Instruct model is selected as the large language model. The Qwen2-VL-7B-Instruct model supports long text understanding and can process data of multiple modalities.

[0122] Furthermore, the list of high-frequency keywords generated by the statistical path and the accurate summary or key sentences generated by the semantic path are associated according to the multimodal fusion block to generate structured processing results.

[0123] Optionally, for each keyword in the high-frequency keyword list, its core information association ID is marked, so that downstream tasks such as intelligent question answering can quickly locate the core content in the document through keywords.

[0124] Optionally, all multimodal fusion blocks can be compiled into precise summaries or key sentences according to the chapter order of the original document to generate a set of core information of the document, which can be directly imported into downstream task models, such as knowledge base systems in the legal and medical fields or RAG retrieval modules.

[0125] This embodiment discloses an intelligent segmentation and core information extraction system for multimodal documents, aiming to solve the problems of untargeted title filtering, semantic fragmentation of fixed-token segmentation, and single modality in traditional document information processing methods. By constructing a task-driven title filtering module, a semantically aware multi-strategy adaptive segmentation module, and a core information extraction module that integrates multimodal information fusion and collaborative processing of statistical and semantic paths, the system achieves efficient processing flow and core information extraction for multimodal documents, improving the semantic integrity, modality compatibility, and information retrieval accuracy of document processing. It is applicable to downstream task scenarios such as knowledge base construction, intelligent question answering, and RAG.

[0126] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A system for intelligent segmentation and core information extraction of multimodal documents, characterized in that, It includes a title filtering module, a multi-strategy segmentation module, a multi-modal fusion module, and a core information extraction module; The title filtering module obtains downstream task requirements and multimodal documents, filters multi-level titles in the multimodal documents using title matching templates, obtains related titles related to downstream tasks, and locates the corresponding related text content. The multi-strategy segmentation module selects an appropriate adaptive segmentation method to segment the associated text content and generate semantically complete text blocks. The adaptive segmentation method includes title structure segmentation, rule aggregation segmentation, and semantic aggregation segmentation. The multimodal fusion module calls the multimodal large model to process the non-text elements corresponding to the associated title, generates non-text semantic descriptions, and binds them with the corresponding text blocks to obtain the multimodal fusion block; The core information extraction module uses a combination of statistical and semantic paths. The statistical path extracts high-frequency keywords from the multimodal fusion block based on word segmentation and word frequency statistics, while the semantic path calls a large language model based on task prompts to generate accurate summaries or key sentences.

2. The intelligent segmentation and core information extraction system for multimodal documents as described in claim 1, characterized in that, The title filtering module includes a document parsing unit and an associated title filtering unit; The document parsing unit uses a document parsing tool to perform full parsing of the multimodal document, extract the document's structural information, and construct a multi-level heading tree structure based on the document's structural information; The associated title filtering unit, based on the downstream task requirements and the multi-level title tree structure, uses a preset title matching template to match the multi-level titles, sorts the multi-level titles by digital weight according to preset rules, filters associated titles related to the downstream tasks, and locates the corresponding associated text content.

3. The intelligent segmentation and core information extraction system for multimodal documents as described in claim 2, characterized in that... Constructing a multi-level heading tree structure based on the document's structured information includes the following steps: A unique identifier ID for the title is generated according to the title level and the order of appearance in the document. The title information is then standardized to generate title node attributes. The title node attributes include title ID, level identifier, title text, parent title ID, title position index in the document, the text boundary corresponding to the title, non-text elements in the text and their position indices, and associated task initialization tag. Based on the heading level and parent heading ID, a hierarchical dependency relationship is established. A multi-level heading tree structure is constructed from top to bottom through hierarchical iteration. First, the heading node with a level of 1 and an empty parent heading ID is selected as the root node. Subheading nodes are inserted under it in turn until all heading nodes have been included in the tree structure, and finally a multi-level heading tree structure is obtained.

4. The intelligent segmentation and core information extraction system for multimodal documents as described in claim 1, characterized in that, The multi-strategy segmentation module includes a title structure segmentation unit, a rule aggregation segmentation unit, and a semantic aggregation segmentation unit; The title structure segmentation unit is based on a multi-level title tree structure. It extracts the hierarchical identifier and text boundary information of each related title node, and determines the segmentation granularity based on the hierarchical identifier. For title nodes with nested relationships, a top-down recursive segmentation method is adopted to divide the related text content set into several independent text segments. The rule aggregation segmentation unit is based on a set of short sentences. It constructs a preset rule base to structurally merge adjacent short sentences and generate rule-aggregated text blocks. The preset rule base includes grammatical structure rules, format structure rules, semantic continuation rules, and regular expression matching rules. The semantic aggregation segmentation unit dynamically aggregates and generates semantically aggregated text blocks by calculating the semantic relationships between adjacent short sentences.

5. The intelligent segmentation and core information extraction system for multimodal documents as described in claim 4, characterized in that, The process of merging short sentences based on preset rules includes the following steps: Iterate through the set of short sentences sequentially and perform rule matching judgment on adjacent short sentences; If any rule or regular expression matching condition is met, the current short sentence is merged with the previous short sentence to form a temporary text block; If the rule conditions are not met, it is determined to be a semantic breakpoint, the current aggregation ends, and a new text block is generated; When a repetition or high similarity segment between the end of the first sentence and the beginning of the second sentence is detected between adjacent short sentences, the overlap interval between the two is calculated, and the overlap range is determined by the longest common substring or edit distance method. The overlapping segments are preserved during merging. After each merge operation, the merged result is used as the new current text block, and rule matching continues until all short sentences are traversed, and a set of structured text blocks is output.

6. The intelligent segmentation and core information extraction system for multimodal documents as described in claim 4, characterized in that, The generation of semantically aggregated text blocks includes the following steps: For each short sentence, calculate the dense semantic vector and the sparse semantic vector to obtain the semantic vector representation and generate a short sentence-semantic vector mapping table; The semantic vector of the first short sentence is used as the initial paragraph cluster center, and the short sentence is added to the list of short sentences to be merged; The subsequent short sentences are traversed sequentially, and the semantic similarity between the current short sentence and the cluster center is calculated based on the weighted cosine similarity of sparse semantic vectors and dense semantic vectors. If the semantic similarity is greater than or equal to the semantic similarity threshold, add it to the list of short sentences to be merged, and update the cluster center by calculating the average of the vectors of all short sentences in the current list; If the semantic similarity is less than the similarity threshold, the current aggregation ends, the list of short sentences to be merged is encapsulated into a semantic aggregation text block, and the cluster center is reinitialized with the semantic vector of the current short sentence to start a new round of aggregation; Repeat the above steps until all short sentences have been traversed, generating a semantic aggregated text block set.

7. The intelligent segmentation and core information extraction system for multimodal documents as described in claim 1, characterized in that, The resulting multimodal fusion block includes: Extract all non-text elements and their position indices corresponding to the associated title nodes, and extract the original non-text elements using a document parsing tool. Preprocess them according to element type, convert them into a standard data description format, and call a multimodal large model to perform semantic parsing and description generation on different types of non-text elements to obtain semantic descriptions of non-text elements. Based on the location index, each non-textual semantic description is bound to a text block processed by the multi-strategy segmentation module to generate a multimodal fusion block.

8. The intelligent segmentation and core information extraction system for multimodal documents as described in claim 1, characterized in that, The core information extraction module includes statistical information units and semantic information units; The statistical information unit generates a list of high-frequency keywords based on word frequency statistics and domain feature weighting; The semantic information module uses preset task prompts and calls a large language model to generate accurate summaries or key sentences for each multimodal fusion block.

9. The intelligent segmentation and core information extraction system for multimodal documents as described in claim 8, characterized in that, Generating a list of high-frequency keywords includes the following steps: Build a domain-specific dictionary, a general stop word list, and a domain-specific stop word list; The Chinese word segmentation model is called to segment the text block, and words that are meaningless or have no informational value are filtered out to obtain a set of effective words. Perform word frequency statistics on the set of valid words, record the occurrence frequency of each valid word, and generate an initial word frequency table; A domain weight coefficient is introduced to adjust word frequency, assigning different weights to core effective words and non-core high-frequency effective words in the domain. Filter valid words with a frequency greater than or equal to a preset high-frequency word threshold, sort them in descending order of word frequency value, and take the top ones. Use 10 effective words to generate a list of high-frequency keywords.

10. The intelligent segmentation and core information extraction system for multimodal documents as described in claim 8, characterized in that, Generating accurate summaries or key sentences includes: selecting appropriate task prompts based on downstream task requirements and document content to guide the large language model to generate accurate summaries or key sentences that meet task requirements. The accurate summary is a brief description of the multimodal text block, and the key sentence is composed of multiple keywords from the multimodal text block.

Images